Week 06
Ablations Complete
Month 2 deepens the science: full baselines, ablations, mechanism analysis (semantic anchoring / drift), and the first complete paper draft.
Place in the arc
Month 2 deepens the science: full baselines, ablations, mechanism analysis (semantic anchoring / drift), and the first complete paper draft.
Objectives
These objectives define what success looked like for Week 6. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [ ] Complete ablation studies (temperature, masks, embeddings)
- [ ] Vocabulary overlap analysis
- [ ] Per-query characteristic correlations
- [ ] Mechanism analysis synthesis
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
1. Temperature Ablation
| Temperature | nDCG@10 | Expansion Diversity | Semantic Drift |
|---|---|---|---|
| 0.5 | 0.358 | Low (0.12) | 0.18 |
| 0.7 | 0.365 | Medium-Low (0.21) | 0.20 |
| 1.0 | 0.371 | Medium (0.34) | 0.23 |
| 1.3 | 0.363 | Medium-High (0.45) | 0.28 |
| 1.5 | 0.352 | High (0.58) | 0.35 |
Optimal: Temperature 1.0
2. Mask Count Ablation
| Masks | nDCG@10 | Avg Tokens Filled |
|---|---|---|
| 20 | 0.358 | 15.2 |
| 30 | 0.365 | 23.8 |
| 50 | 0.371 | 34.7 |
| 75 | 0.368 | 48.3 |
| 100 | 0.359 | 61.2 |
Optimal: 50 masks. Model self-regulates - doesn't fill all masks.
3. Embedding Model Sensitivity
| Embedding | NFCorpus Δ | FiQA Δ | SciFact Δ | TREC-COVID Δ |
|---|---|---|---|---|
| BGE-M3 | +8.5% | +7.9% | -1.2% | +6.0% |
| E5-large | +7.6% | +7.2% | -0.7% | +5.4% |
| GTR-T5 | +6.1% | +5.9% | -1.8% | +4.2% |
| MiniLM | +6.3% | +5.2% | -0.4% | +4.8% |
Pattern consistent across all embedding models!
4. Vocabulary Overlap Analysis
| Method | NFCorpus | FiQA | SciFact | TREC-COVID |
|---|---|---|---|---|
| QUID | 0.67 | 0.71 | 0.58 | 0.64 |
| HyDE | 0.61 | 0.65 | 0.72 | 0.59 |
QUID generates higher vocabulary overlap on specialized domains.
5. Per-Query Characteristic Correlations
| Query Feature | Correlation with QUID Δ |
|---|---|
| Domain terminology density | r = 0.41 (strongest) |
| Query specificity | r = 0.34 |
| Low baseline performance | r = -0.28 |
| Query length | r = 0.18 |
Key insight: QUID helps most when queries contain domain-specific terminology.
6. Mechanism Summary
Three-part mechanism:
- Vocabulary expansion: Adds domain-appropriate terms
- Semantic anchoring: Stays closer to query than autoregressive methods
- Conservative generation: Fewer but more precise terms
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
1. Temperature Ablation
| Temperature | nDCG@10 | Expansion Diversity | Semantic Drift |
|---|---|---|---|
| 0.5 | 0.358 | Low (0.12) | 0.18 |
| 0.7 | 0.365 | Medium-Low (0.21) | 0.20 |
| 1.0 | 0.371 | Medium (0.34) | 0.23 |
| 1.3 | 0.363 | Medium-High (0.45) | 0.28 |
| 1.5 | 0.352 | High (0.58) | 0.35 |
Optimal: Temperature 1.0
2. Mask Count Ablation
| Masks | nDCG@10 | Avg Tokens Filled |
|---|---|---|
| 20 | 0.358 | 15.2 |
| 30 | 0.365 | 23.8 |
| 50 | 0.371 | 34.7 |
| 75 | 0.368 | 48.3 |
| 100 | 0.359 | 61.2 |
Optimal: 50 masks. Model self-regulates - doesn't fill all masks.
3. Embedding Model Sensitivity
| Embedding | NFCorpus Δ | FiQA Δ | SciFact Δ | TREC-COVID Δ |
|---|---|---|---|---|
| BGE-M3 | +8.5% | +7.9% | -1.2% | +6.0% |
| E5-large | +7.6% | +7.2% | -0.7% | +5.4% |
| GTR-T5 | +6.1% | +5.9% | -1.8% | +4.2% |
| MiniLM | +6.3% | +5.2% | -0.4% | +4.8% |
Pattern consistent across all embedding models!
4. Vocabulary Overlap Analysis
| Method | NFCorpus | FiQA | SciFact | TREC-COVID |
|---|---|---|---|---|
| QUID | 0.67 | 0.71 | 0.58 | 0.64 |
| HyDE | 0.61 | 0.65 | 0.72 | 0.59 |
QUID generates higher vocabulary overlap on specialized domains.
5. Per-Query Characteristic Correlations
| Query Feature | Correlation with QUID Δ |
|---|---|
| Domain terminology density | r = 0.41 (strongest) |
| Query specificity | r = 0.34 |
| Low baseline performance | r = -0.28 |
| Query length | r = 0.18 |
Key insight: QUID helps most when queries contain domain-specific terminology.
6. Mechanism Summary
Three-part mechanism:
- Vocabulary expansion: Adds domain-appropriate terms
- Semantic anchoring: Stays closer to query than autoregressive methods
- Conservative generation: Fewer but more precise terms
Challenges
- Embedding model compute: Testing 4 models × 4 datasets required sampling for initial analysis.
- Correlation interpretation: Many features correlated with each other; causal interpretation difficult.
Key learnings
- QUID's benefit is robust to hyperparameters and embedding choice
- Domain terminology density is the strongest predictor of benefit
- The mechanism analysis provides publishable insights
Plan for next week
- Complete error analysis (78 failure cases)
- Document limitations
- Begin paper outline
Hours logged
| Activity | Hours |
|---|---|
| Temperature/mask ablations | 8 |
| Embedding sensitivity | 8 |
| Vocabulary analysis | 6 |
| Per-query analysis | 8 |
| Documentation | 2 |
| Total | 32 |
Notes for advisor
Ablations complete. QUID is robust. Mechanism analysis provides clear story: "semantic anchoring" differentiates QUID from autoregressive methods.