Week 03
Core Experiments
This week sits in Month 1 - building the substrate for QUID: environment, LLaDA porting, BEIR wiring, and the first signals of domain-dependent retrieval gains.
Place in the arc
This week sits in Month 1 - building the substrate for QUID: environment, LLaDA porting, BEIR wiring, and the first signals of domain-dependent retrieval gains.
Objectives
These objectives define what success looked like for Week 3. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [ ] Implement QUID pipeline
- [ ] Run initial experiments on NFCorpus and FiQA
- [ ] Hyperparameter tuning
- [ ] Begin failure case analysis
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
1. QUID Pipeline Implementation
Built the core QUID pipeline:
class QUID:
def __init__(self, diffusion_model, encoder):
self.diffusion = diffusion_model # LLaDA-8B
self.encoder = encoder # BGE-M3
def expand_query(self, query: str, num_masks: int = 50) -> str:
# 1. Tokenize query
# 2. Append [MASK] tokens
# 3. Run diffusion for N steps
# 4. Return expanded text2. NFCorpus Experiments
Initial Results (32 steps, 50 masks, temp=1.0):
| Method | nDCG@10 | MRR@10 | Recall@100 |
|---|---|---|---|
| Vanilla | 0.342 | 0.401 | 0.287 |
| Template | 0.351 | 0.412 | 0.294 |
| QUID | 0.371 | 0.438 | 0.312 |
+8.5% nDCG@10 improvement over vanilla!
3. FiQA Experiments
| Method | nDCG@10 | MRR@10 | Recall@100 |
|---|---|---|---|
| Vanilla | 0.418 | 0.512 | 0.623 |
| Template | 0.429 | 0.528 | 0.641 |
| QUID | 0.451 | 0.553 | 0.668 |
+7.9% nDCG@10 improvement!
4. Hyperparameter Tuning
Diffusion Steps (NFCorpus):
| Steps | nDCG@10 | Latency |
|---|---|---|
| 8 | 0.358 | 210ms |
| 16 | 0.367 | 420ms |
| 32 | 0.371 | 830ms |
| 64 | 0.373 | 1650ms |
Optimal: 32 steps (64 gives marginal gain for 2x latency)
Mask Count: 50 optimal (30-75 all acceptable)
Temperature: 1.0 optimal (0.7-1.3 acceptable)
5. Initial Failure Analysis
Identified 41 queries (12.7%) where QUID underperformed:
- Topic drift: 18 cases
- Over-expansion: 12 cases
- Wrong domain terms: 7 cases
- Ambiguous query: 4 cases
Most failures on very short queries (≤3 words).
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
1. QUID Pipeline Implementation
Built the core QUID pipeline:
class QUID:
def __init__(self, diffusion_model, encoder):
self.diffusion = diffusion_model # LLaDA-8B
self.encoder = encoder # BGE-M3
def expand_query(self, query: str, num_masks: int = 50) -> str:
# 1. Tokenize query
# 2. Append [MASK] tokens
# 3. Run diffusion for N steps
# 4. Return expanded text2. NFCorpus Experiments
Initial Results (32 steps, 50 masks, temp=1.0):
| Method | nDCG@10 | MRR@10 | Recall@100 |
|---|---|---|---|
| Vanilla | 0.342 | 0.401 | 0.287 |
| Template | 0.351 | 0.412 | 0.294 |
| QUID | 0.371 | 0.438 | 0.312 |
+8.5% nDCG@10 improvement over vanilla!
3. FiQA Experiments
| Method | nDCG@10 | MRR@10 | Recall@100 |
|---|---|---|---|
| Vanilla | 0.418 | 0.512 | 0.623 |
| Template | 0.429 | 0.528 | 0.641 |
| QUID | 0.451 | 0.553 | 0.668 |
+7.9% nDCG@10 improvement!
4. Hyperparameter Tuning
Diffusion Steps (NFCorpus):
| Steps | nDCG@10 | Latency |
|---|---|---|
| 8 | 0.358 | 210ms |
| 16 | 0.367 | 420ms |
| 32 | 0.371 | 830ms |
| 64 | 0.373 | 1650ms |
Optimal: 32 steps (64 gives marginal gain for 2x latency)
Mask Count: 50 optimal (30-75 all acceptable)
Temperature: 1.0 optimal (0.7-1.3 acceptable)
5. Initial Failure Analysis
Identified 41 queries (12.7%) where QUID underperformed:
- Topic drift: 18 cases
- Over-expansion: 12 cases
- Wrong domain terms: 7 cases
- Ambiguous query: 4 cases
Most failures on very short queries (≤3 words).
Challenges
- Latency: 830ms per query may be slow for production. 16-step variant available.
- Short query problem: Queries ≤3 words have insufficient context. Implemented adaptive strategy.
Key learnings
- QUID shows strong results on specialized domains (+8.5% medical, +7.9% financial)
- Hyperparameters are robust - no per-domain tuning needed
- Short queries are the primary failure mode
Plan for next week
- Run SciFact and TREC-COVID experiments
- Implement HyDE baseline
- Cross-dataset analysis
Hours logged
| Activity | Hours |
|---|---|
| QUID implementation | 8 |
| NFCorpus experiments | 8 |
| FiQA experiments | 6 |
| Hyperparameter tuning | 8 |
| Failure analysis | 4 |
| Total | 34 |
Notes for advisor
Excellent initial results. Domain-dependent hypothesis looking promising with strong gains on NFCorpus and FiQA. Will test on SciFact next week to verify the hypothesis about general scientific domains.