Week 07
Error Analysis & Paper Outline
Month 2 deepens the science: full baselines, ablations, mechanism analysis (semantic anchoring / drift), and the first complete paper draft.
Place in the arc
Month 2 deepens the science: full baselines, ablations, mechanism analysis (semantic anchoring / drift), and the first complete paper draft.
Objectives
These objectives define what success looked like for Week 7. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [ ] Complete error analysis
- [ ] Document all limitations
- [ ] Finalize paper outline
- [ ] Begin drafting abstract and introduction
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
1. Comprehensive Error Analysis
Analyzed 78 failure cases across all datasets:
| Error Type | Count | % | Systematic? |
|---|---|---|---|
| Factual expansion needed | 24 | 31% | Yes - inherent |
| Topic drift | 28 | 36% | Partial - can mitigate |
| Over-expansion | 14 | 18% | Yes - short query issue |
| Ambiguous query | 12 | 15% | Yes - inherent |
Example Failures:
- Factual needed: "CRISPR mechanism" → needs specific facts
- Topic drift: "stem cell" → expanded to general biology
- Over-expansion: "pain" → too many diluting terms
- Ambiguous: "apple" - fruit or company?
2. Limitations Documentation
Fundamental Limitations:
- Cannot generate factual content - vocabulary expansion only
- Requires domain-specific vocabulary gap to help
- 400-830ms latency overhead per query
Technical Limitations:
- LLaDA-8B requires ~16GB GPU memory
- Non-standard generation code complexity
- Long queries (>50 tokens) not tested
Scope Limitations:
- English only
- Four BEIR datasets
- Single GPU tested
3. Paper Outline Finalized
1. Introduction (1.5 pages)
- Query-document asymmetry
- HyDE limitations (hallucination)
- QUID: semantic anchoring
2. Related Work (1.5 pages)
- Query expansion methods
- Text diffusion models
- Dense retrieval
3. Method (1.5 pages)
- Problem formulation
- QUID architecture
- Retrieval pipeline
4. Experiments (3 pages)
- Datasets and baselines
- Main results (Table 1)
- Ablations (Table 2)
- Analysis
5. Conclusion (0.5 pages)
Total: 8 pages + references4. Abstract Draft
Dense retrieval systems suffer from vocabulary mismatch: user queries are short while relevant documents use domain-specific terminology. Current approaches like HyDE use autoregressive LLMs to generate hypothetical documents, but may hallucinate facts. We introduce QUID (Queries Unmasked by Iterative Diffusion), which frames query expansion as denoising using masked text diffusion. We find domain-dependent effectiveness: QUID improves nDCG@10 by 8.5% on medical (NFCorpus) and 7.9% on financial (FiQA) queries while performing neutrally on general scientific text. Analysis reveals QUID maintains "semantic anchoring" - staying close to query intent while expanding vocabulary.
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
1. Comprehensive Error Analysis
Analyzed 78 failure cases across all datasets:
| Error Type | Count | % | Systematic? |
|---|---|---|---|
| Factual expansion needed | 24 | 31% | Yes - inherent |
| Topic drift | 28 | 36% | Partial - can mitigate |
| Over-expansion | 14 | 18% | Yes - short query issue |
| Ambiguous query | 12 | 15% | Yes - inherent |
Example Failures:
- Factual needed: "CRISPR mechanism" → needs specific facts
- Topic drift: "stem cell" → expanded to general biology
- Over-expansion: "pain" → too many diluting terms
- Ambiguous: "apple" - fruit or company?
2. Limitations Documentation
Fundamental Limitations:
- Cannot generate factual content - vocabulary expansion only
- Requires domain-specific vocabulary gap to help
- 400-830ms latency overhead per query
Technical Limitations:
- LLaDA-8B requires ~16GB GPU memory
- Non-standard generation code complexity
- Long queries (>50 tokens) not tested
Scope Limitations:
- English only
- Four BEIR datasets
- Single GPU tested
3. Paper Outline Finalized
1. Introduction (1.5 pages)
- Query-document asymmetry
- HyDE limitations (hallucination)
- QUID: semantic anchoring
2. Related Work (1.5 pages)
- Query expansion methods
- Text diffusion models
- Dense retrieval
3. Method (1.5 pages)
- Problem formulation
- QUID architecture
- Retrieval pipeline
4. Experiments (3 pages)
- Datasets and baselines
- Main results (Table 1)
- Ablations (Table 2)
- Analysis
5. Conclusion (0.5 pages)
Total: 8 pages + references4. Abstract Draft
Dense retrieval systems suffer from vocabulary mismatch: user queries are short while relevant documents use domain-specific terminology. Current approaches like HyDE use autoregressive LLMs to generate hypothetical documents, but may hallucinate facts. We introduce QUID (Queries Unmasked by Iterative Diffusion), which frames query expansion as denoising using masked text diffusion. We find domain-dependent effectiveness: QUID improves nDCG@10 by 8.5% on medical (NFCorpus) and 7.9% on financial (FiQA) queries while performing neutrally on general scientific text. Analysis reveals QUID maintains "semantic anchoring" - staying close to query intent while expanding vocabulary.
Challenges
- Limitation framing: Need to present honestly without undermining contribution.
- Abstract length: Cut from 180 to 148 words for conference format.
Key learnings
- Most failures are in cases where QUID wasn't expected to help
- Honest limitations section builds credibility
- "Semantic anchoring" is the right framing for the contribution
Plan for next week
- Complete paper draft (all sections)
- Create publication-quality figures
- Prepare code for release
Hours logged
| Activity | Hours |
|---|---|
| Error analysis | 10 |
| Limitations documentation | 6 |
| Paper outline | 4 |
| Abstract/intro drafting | 10 |
| Documentation | 2 |
| Total | 32 |
Notes for advisor
Error analysis and limitations complete. Paper outline ready. Abstract drafted. On track to complete full draft next week.