I headed to MIT CODE 2025 (Conference on Digital Experimentation), not just to listen, but to stress-test my own reality.
As a Data Scientist at Safeway (part of Albertsons Companies), I often work at the intersection of theory and messy, real-world application. I walked into MIT with hypotheses about my work and a few "radical" ideas I was trying to push through.
I walked out with those ideas validated, a notebook full of industry secrets, and the confidence to execute immediately. Here is my complete experience.
✅ The "Aha!" Moment: Validating 2 Key Ideas
The biggest win of the conference wasn't a lecture—it was the confirmation that I was on the right track. I had been wrestling with two major technical initiatives at Albertsons, and through hallway conversations and session takeaways, I found the "permission" I needed to move forward.
1. Fixing Flawed Randomization
I had long suspected our legacy "pre-split" randomization method—where engineers deterministically assigned users based on a sorted list—was flawed. It felt like a silent killer of data integrity.
- ✦The Validation: Networking with experts confirmed that without true randomization, our downstream metrics were compromised.
- ✦The Result: Empowered by this consensus, I returned to work and successfully corrected this method, replacing it with a robust, randomized assignment process.
2. Implementing CUPED for Variance Reduction
I was also pushing to implement CUPED (Controlled-Experiment Using Pre-Experiment Data) to speed up our tests.
- ✦The Validation: Seeing data scientists from tech giants discuss their reliance on similar variance reduction techniques gave me the ammunition I needed.
- ✦The Result: I didn't just propose it; I was able to directly complete the implementation of CUPED (or "Cupid" as we jokingly call it in the hallway tracks) into our pipeline. This has been a game-changer for our experiment velocity.
🧠 What I Learned: The "Productivity Paradox" & Market Mechanics
Beyond my own projects, the sessions were a goldmine of forward-thinking concepts.
The Productivity Paradox
There was a fascinating discussion during the Fireside Chat about why AI helps us code faster but doesn't necessarily make us feel more productive.
Lane 1 vs. Lane 2: We are currently in "Lane 1" (task automation). The real breakthrough will be "Lane 2"—redesigning entire workflows. The bottleneck has shifted from "doing" to "sense-making".
Interference in Marketplaces
Ramesh Johari from Stanford gave a plenary talk that changed how I view two-sided markets. He explained that "interference"—where one user's treatment affects another—is a massive source of bias.
The Fix: His advice was practical: Randomize on the short side of the market (the supply-constrained side) to minimize this bias.
Welfare > P-Values
Tim Sodejono presented a framework that hit home: "Welfare Maximization." Instead of obsessing over p-values (p < 0.05), we should focus on the economic return of an intervention minus its costs. It's a compound decision problem, not just a statistics test.
🎤 Fun Facts & The Human Touch
It wasn't all algorithms and p-values. The human side of MIT CODE was just as impactful.
My "AI" Sidekick
I walked around with my Plot AI device, recording [permitted] conversations to generate transcripts/summaries. Great conversation starter that kept me present.
Networking with Giants
Incredible informal chats with Minal (Amazon Web Lab) and Chetan (founder of a platform acquired by Datadog).
The "Low-Tech" Realization
I called Albertsons "low tech", but they validated that getting foundations right (randomization, power) is infinitely more valuable than fancy garbage-producing platforms.
🚀 The Bottom Line
MIT CODE 2025 was a reminder that you don't need a massive tech stack to do world-class data science. You need curiosity, rigor, and the courage to question the "standard" way of doing things.
I came back to Dallas not just with "learnings," but with completed upgrades to our experimentation stack. Randomization is fixed. CUPED is live. And I'm just getting started.
Curious about Data Science, Yoga, or Lifestyle Design? Connect with me on LinkedIn or explore more here at JustSuyash.com.