Week 09
Revision & Reproducibility
Month 3 focuses on polish - reproducibility, advisor feedback, code release, and presentation readiness - while locking the core thesis for Review 2.
Place in the arc
Month 3 focuses on polish - reproducibility, advisor feedback, code release, and presentation readiness - while locking the core thesis for Review 2.
Objectives
These objectives define what success looked like for Week 9. Meeting them either unlocked the next experiment, closed a risk, or produced evidence for the thesis claim that diffusion expansion is domain-dependent.
- [ ] Self-review and revise paper draft
- [ ] Complete code documentation
- [ ] Begin reproducibility verification
- [ ] Polish figures and tables
What we built and measured
Below is the detailed record for the week - methods, implementation notes, and experimental setup - expanded from the team log so a reader can follow the technical path without the repository open.
1. Paper Revision - First Pass
Issues Identified and Fixed:
| Section | Issue | Fix |
|---|---|---|
| Abstract | Too long (160 words) | Cut to 148 words |
| Introduction | Weak motivation | Added clinician search example |
| Method | Missing algorithm details | Added pseudocode box |
| Results | Inconsistent decimal places | Standardized to 3 decimals |
| Analysis | Drift definition unclear | Added formal equation |
2. Code Documentation Complete
README.md sections:
- Installation instructions (pip, conda)
- Quick start example (5 lines)
- Full API documentation
- Hyperparameter guide
- Citation information
Docstrings added to:
- All public functions
- Class definitions
- Module headers
3. Reproducibility Verification Started
Testing from fresh clone:
| Experiment | Paper Result | Fresh Run | Status |
|---|---|---|---|
| NFCorpus nDCG@10 | 0.371 | 0.370 | ✅ Match |
| FiQA nDCG@10 | 0.451 | Running... | 🔄 |
4. Figure Polish
- Increased font sizes for readability
- Standardized color scheme across figures
- Added error bars to bar charts
- Improved axis labels
Numbers and what they mean
Metrics are only useful with interpretation. Where tables appear, read the deltas as claims about when QUID helps (vocabulary gap) versus when HyDE or vanilla wins (knowledge / claim gap). Agentic weeks emphasize tool-call policies over blind expansion.
1. Paper Revision - First Pass
Issues Identified and Fixed:
| Section | Issue | Fix |
|---|---|---|
| Abstract | Too long (160 words) | Cut to 148 words |
| Introduction | Weak motivation | Added clinician search example |
| Method | Missing algorithm details | Added pseudocode box |
| Results | Inconsistent decimal places | Standardized to 3 decimals |
| Analysis | Drift definition unclear | Added formal equation |
2. Code Documentation Complete
README.md sections:
- Installation instructions (pip, conda)
- Quick start example (5 lines)
- Full API documentation
- Hyperparameter guide
- Citation information
Docstrings added to:
- All public functions
- Class definitions
- Module headers
3. Reproducibility Verification Started
Testing from fresh clone:
| Experiment | Paper Result | Fresh Run | Status |
|---|---|---|---|
| NFCorpus nDCG@10 | 0.371 | 0.370 | ✅ Match |
| FiQA nDCG@10 | 0.451 | Running... | 🔄 |
4. Figure Polish
- Increased font sizes for readability
- Standardized color scheme across figures
- Added error bars to bar charts
- Improved axis labels
Challenges
- Reproducibility variance: Minor differences in 4th decimal due to GPU non-determinism. Acceptable.
- Documentation scope: Balancing thoroughness vs. time. Focused on user-facing docs.
Key learnings
- Self-review catches many issues before external review
- Reproducibility verification builds confidence in results
- Good documentation takes significant effort
Plan for next week
- Complete reproducibility verification
- Send paper to advisor for review
- Prepare presentation outline
Hours logged
| Activity | Hours |
|---|---|
| Paper revision | 12 |
| Code documentation | 10 |
| Reproducibility testing | 6 |
| Figure polish | 4 |
| Total | 32 |
Notes for advisor
Paper draft revised and ready for your review. Will send PDF by end of day. Key feedback requested on:
- Is the "semantic anchoring" framing clear?
- Are the comparisons fair?
- Is the negative result (SciFact) well-framed?