Week 02
Infrastructure Complete
This week sits in Month 1 - building the substrate for QUID: environment, LLaDA porting, BEIR wiring, and the first signals of domain-dependent retrieval gains.
Place in the arc
This week sits in Month 1 - building the substrate for QUID: environment, LLaDA porting, BEIR wiring, and the first signals of domain-dependent retrieval gains.
Objectives
These objectives define what success looked like for Week 2. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [ ] Complete LLaDA generation code port
- [ ] Finish BEIR evaluation infrastructure
- [ ] Download and preprocess all datasets
- [ ] Implement vanilla and template baselines
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
1. LLaDA Generation Port (Complete)
Successfully ported the core LLaDA generation logic:
def generate_diffusion(
model,
input_ids, # Query tokens + [MASK] tokens
num_steps=32, # Diffusion steps
temperature=1.0,
top_p=0.95
) -> torch.Tensor:Key implementation details:
- Confidence-based unmasking implemented
- Variable-length generation supported
- All unit tests passing on Modal A10G GPU
- Generation speed: ~0.8s for 32 steps
2. BEIR Evaluation Infrastructure (Complete)
class BEIREvaluator:
def __init__(self, dataset_name: str):
self.corpus, self.queries, self.qrels = load_beir_dataset(dataset_name)
self.encoder = SentenceTransformer('BAAI/bge-m3')
def evaluate(self, query_expansion_fn) -> dict:
# Returns nDCG@10, MRR@10, Recall@KFeatures: corpus embedding caching (FAISS), bootstrap CIs, paired significance tests.
3. Dataset Preparation (Complete)
| Dataset | Queries | Documents | Corpus Size | Status |
|---|---|---|---|---|
| NFCorpus | 323 | 3,633 | 12 MB | ✅ Indexed |
| FiQA | 648 | 57,638 | 89 MB | ✅ Indexed |
| SciFact | 300 | 5,183 | 18 MB | ✅ Indexed |
| TREC-COVID | 50 | 171,332 | 1.2 GB | ✅ Indexed |
4. Baselines Implemented
Vanilla BGE-M3 Results:
| Dataset | nDCG@10 | MRR@10 | Recall@100 |
|---|---|---|---|
| NFCorpus | 0.342 | 0.401 | 0.287 |
| FiQA | 0.418 | 0.512 | 0.623 |
| SciFact | 0.687 | 0.721 | 0.892 |
| TREC-COVID | 0.598 | 0.712 | 0.445 |
Template Expansion Results:
| Dataset | nDCG@10 | Δ vs Vanilla |
|---|---|---|
| NFCorpus | 0.351 | +2.6% |
| FiQA | 0.429 | +2.6% |
| SciFact | 0.691 | +0.6% |
| TREC-COVID | 0.612 | +2.3% |
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
1. LLaDA Generation Port (Complete)
Successfully ported the core LLaDA generation logic:
def generate_diffusion(
model,
input_ids, # Query tokens + [MASK] tokens
num_steps=32, # Diffusion steps
temperature=1.0,
top_p=0.95
) -> torch.Tensor:Key implementation details:
- Confidence-based unmasking implemented
- Variable-length generation supported
- All unit tests passing on Modal A10G GPU
- Generation speed: ~0.8s for 32 steps
2. BEIR Evaluation Infrastructure (Complete)
class BEIREvaluator:
def __init__(self, dataset_name: str):
self.corpus, self.queries, self.qrels = load_beir_dataset(dataset_name)
self.encoder = SentenceTransformer('BAAI/bge-m3')
def evaluate(self, query_expansion_fn) -> dict:
# Returns nDCG@10, MRR@10, Recall@KFeatures: corpus embedding caching (FAISS), bootstrap CIs, paired significance tests.
3. Dataset Preparation (Complete)
| Dataset | Queries | Documents | Corpus Size | Status |
|---|---|---|---|---|
| NFCorpus | 323 | 3,633 | 12 MB | ✅ Indexed |
| FiQA | 648 | 57,638 | 89 MB | ✅ Indexed |
| SciFact | 300 | 5,183 | 18 MB | ✅ Indexed |
| TREC-COVID | 50 | 171,332 | 1.2 GB | ✅ Indexed |
4. Baselines Implemented
Vanilla BGE-M3 Results:
| Dataset | nDCG@10 | MRR@10 | Recall@100 |
|---|---|---|---|
| NFCorpus | 0.342 | 0.401 | 0.287 |
| FiQA | 0.418 | 0.512 | 0.623 |
| SciFact | 0.687 | 0.721 | 0.892 |
| TREC-COVID | 0.598 | 0.712 | 0.445 |
Template Expansion Results:
| Dataset | nDCG@10 | Δ vs Vanilla |
|---|---|---|
| NFCorpus | 0.351 | +2.6% |
| FiQA | 0.429 | +2.6% |
| SciFact | 0.691 | +0.6% |
| TREC-COVID | 0.612 | +2.3% |
Challenges
- Numerical precision: Initial port had slight differences due to float32 vs bfloat16. Resolved.
- TREC-COVID size: Required batched embedding to avoid OOM. Implemented successfully.
Key learnings
- 32 diffusion steps is the sweet spot for quality/speed
- NFCorpus has notably lower baseline - good candidate for expansion
- Infrastructure phase complete ahead of schedule
Plan for next week
- Implement QUID pipeline
- Run initial experiments on NFCorpus
- Begin hyperparameter tuning
Hours logged
| Activity | Hours |
|---|---|
| LLaDA porting | 10 |
| BEIR infrastructure | 8 |
| Dataset preprocessing | 6 |
| Baseline implementation | 6 |
| Documentation | 2 |
| Total | 32 |
Notes for advisor
Major milestone: all infrastructure complete in 2 weeks. LLaDA ported, BEIR ready, baselines established. Ready to begin core experiments.