Week 14
Router Pilots & Next Steps
Month 4 focus: we are designing and trying an agentic query router on top of QUID. The goal is not to replace diffusion expansion, but to learn when to call it, when to try HyDE-style expansion, and when to leave the query alone. Early pilots are underway; the preprint is still forthcoming.
Place in the arc
Month 4 focus: we are designing and trying an agentic query router on top of QUID. The goal is not to replace diffusion expansion, but to learn when to call it, when to try HyDE-style expansion, and when to leave the query alone. Early pilots are underway; the preprint is still forthcoming.
Objectives
These objectives define what success looked like for Week 14. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [x] Stand up agentic router pilots on NFCorpus (50 queries)
- [x] Stand up agentic router pilots on SciFact (50 queries)
- [x] Draft Month 4 narrative into the written report
- [x] Document expansion weeks (13-14)
- [x] Keep code pointed at GitHub (
Zhreyu/quid) - [ ] Scale pilots to larger BEIR slices (next)
- [ ] Learned router toward oracle labels (next)
- [ ] Preprint polish (coming soon)
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
Week 14 Progress Report - Router Pilots & Next Steps
Project: QUID - Diffusion-Based Query Expansion for Dense Information Retrieval
Student: Shreyas S
Week: 14 (Month 4 · in progress)
Focus: Early agentic router pilots, Review narrative, preprint prep
Objectives for This Week
- [x] Stand up agentic router pilots on NFCorpus (50 queries)
- [x] Stand up agentic router pilots on SciFact (50 queries)
- [x] Draft Month 4 narrative into the written report
- [x] Document expansion weeks (13-14)
- [x] Keep code pointed at GitHub (
Zhreyu/quid) - [ ] Scale pilots to larger BEIR slices (next)
- [ ] Learned router toward oracle labels (next)
- [ ] Preprint polish (coming soon)
Pilot Setup (early runs)
| Item | Value |
|---|---|
| GPU | Modal L4 (24GB) |
| Diffusion | LLaDA-8B-Instruct, 16 steps, gen_length 32 |
| Embeddings | BGE-M3 |
| HyDE backend | LLaDA document-style prompt (OpenAI quota unavailable) |
| Metrics | nDCG@10; route vs oracle accuracy |
These are pilot-scale slices (n=50), not frozen claims. We are using them to decide what to run next.
Early signals (not final)
SciFact (science claims, n=50) - selective tools look promising
| Method | nDCG@10 | Δ vs vanilla |
|---|---|---|
| always_vanilla | 0.6299 | - |
| always_QUID | 0.5999 | −0.0300 |
| agentic_single_shot | 0.6280 | −0.0019 |
| agentic_multi_step | 0.6625 | +0.0326 |
| oracle | 0.7004 | +0.0705 |
Working takeaway: blind always-expand looks brittle here; multi-step tool use looks more stable in this pilot. Larger confirmation runs are still ahead.
NFCorpus (medical, n=50)
| Method | nDCG@10 | Δ vs vanilla |
|---|---|---|
| always_vanilla | 0.3305 | - |
| always_QUID | 0.2704 | −0.0601 |
| agentic (single=multi) | 0.3053 | −0.0252 |
| oracle | 0.3577 | +0.0272 |
Early signal: selective routing recovers ground vs always-QUID on this slice. We still need fuller splits before locking numbers.
Artifacts: reports/agentic_results/*_q50.json.
Month 4 narrative (what we are building)
- Core thesis unchanged: masked diffusion (LLaDA) for query expansion; semantic anchoring.
- Agentic angle (in progress): treat expansion methods as tools; let a router decide when to call them.
- Why now: QUID is domain-dependent - always expanding is the wrong default on some query types.
- What we will do next: larger BEIR slices, stronger critique/retry, learned router toward oracle, preprint polish.
Deliverables in motion
reports/final_report.md- §9.2 Agentic query router (Month 4)reports/week_13.md,reports/week_14.mdsrc/agentic_router.py,src/run_agentic_benchmark.py- README agentic section
- Code: https://github.com/Zhreyu/quid
Weekly Status: MONTH 4 IN PROGRESS · PREPRINT COMING SOON
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
Challenges
No major blockers recorded this week.
Key learnings
See work completed for takeaways.
Plan for next week
Continue per project roadmap.