Week 05
All Baselines Complete
Month 2 deepens the science: full baselines, ablations, mechanism analysis (semantic anchoring / drift), and the first complete paper draft.
Place in the arc
Month 2 deepens the science: full baselines, ablations, mechanism analysis (semantic anchoring / drift), and the first complete paper draft.
Objectives
These objectives define what success looked like for Week 5. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [ ] Implement Query2Doc baseline
- [ ] Implement BM25 and BM25+RM3 baselines
- [ ] Complete statistical significance testing
- [ ] Begin ablation studies
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
1. Query2Doc Implementation & Evaluation
Implemented Query2Doc with few-shot prompting:
| Dataset | HyDE | Query2Doc | QUID |
|---|---|---|---|
| NFCorpus | 0.362 | 0.368 | 0.371 |
| FiQA | 0.447 | 0.452 | 0.451 |
| SciFact | 0.724 | 0.731 | 0.679 |
| TREC-COVID | 0.628 | 0.632 | 0.634 |
Query2Doc edges out HyDE slightly but still loses to QUID on specialized domains.
2. BM25 Baselines
Implemented using Pyserini:
| Dataset | BM25 | BM25+RM3 | QUID (Dense) |
|---|---|---|---|
| NFCorpus | 0.325 | 0.338 | 0.371 |
| FiQA | 0.236 | 0.251 | 0.451 |
| SciFact | 0.665 | 0.678 | 0.679 |
| TREC-COVID | 0.656 | 0.688 | 0.634 |
BM25+RM3 beats QUID on TREC-COVID - lexical methods still strong for some domains.
3. Statistical Significance Testing
Bootstrap 95% CIs (n=1000) and paired t-tests:
NFCorpus:
- QUID vs Vanilla: p < 0.01 (significant)
- QUID vs HyDE: p < 0.05 (significant)
FiQA:
- QUID vs Vanilla: p < 0.01 (significant)
- QUID vs HyDE: p = 0.23 (not significant, but QUID is free)
SciFact:
- HyDE vs QUID: p < 0.01 (HyDE significantly better)
TREC-COVID:
- QUID vs Vanilla: p < 0.01 (significant)
4. Ablation Studies Started
Diffusion Steps Full Analysis:
| Steps | nDCG@10 | Latency | Quality/Latency |
|---|---|---|---|
| 4 | 0.351 | 105ms | 3.34 |
| 8 | 0.362 | 210ms | 1.72 |
| 16 | 0.367 | 420ms | 0.87 |
| 32 | 0.371 | 830ms | 0.45 |
| 64 | 0.373 | 1650ms | 0.23 |
Recommendation: 32 steps for offline, 16 for interactive.
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
| Dataset | Vanilla | Template | BM25 | BM25+RM3 | HyDE | Query2Doc | QUID |
|---|---|---|---|---|---|---|---|
| NFCorpus | 0.342 | 0.351 | 0.325 | 0.338 | 0.362 | 0.368 | 0.371 |
| FiQA | 0.418 | 0.429 | 0.236 | 0.251 | 0.447 | 0.452 | 0.451 |
| SciFact | 0.687 | 0.691 | 0.665 | 0.678 | 0.724 | 0.731 | 0.679 |
| TREC-COVID | 0.598 | 0.612 | 0.656 | 0.688 | 0.628 | 0.632 | 0.634 |
Challenges
- Bootstrap computation: 1000 iterations × 4 datasets × 7 methods took several hours. Parallelized.
- FiQA significance: QUID vs HyDE not significant despite numeric advantage.
Key learnings
- Statistical significance confirms domain-dependent effectiveness
- QUID's wins are real (p < 0.01 on key comparisons)
- Lexical methods (BM25+RM3) remain competitive on TREC-COVID
Plan for next week
- Complete remaining ablations (temperature, masks, embeddings)
- Vocabulary overlap analysis
- Per-query characteristic correlations
Hours logged
| Activity | Hours |
|---|---|
| Query2Doc implementation | 6 |
| BM25 baselines | 6 |
| Statistical testing | 10 |
| Ablations | 8 |
| Documentation | 2 |
| Total | 32 |
Notes for advisor
All baselines complete. Statistical significance confirmed. Publication-ready results table generated.