No Rain, No Gain: SRM Tears and Build vs Buy (+Marketing Analytics Summit discount code!)
TLC Connector | Teach. Learn. Converse.
Welcome back, TLCers!
*Make sure you scroll down for the exclusive 20% TLC discount on the Marketing Analytics Summit!
Everyone is familiar with that age-old saying, "April Showers bring May Flowers." This age-old description of spring rings true (funnily enough, it's raining as we write this), but It's also a nice metaphor for learning from our mistakes. In this month’s Chatter, we highlight some “rain” in the oft-discussed Sample Ratio Mismatch (SRM). For some of y’all, we realize these words might conjure storm sirens instead of gentle spring rains, but we at the TLC are nothing if not optimistic!
Rainstorms bring growth, and SRMs bring challenges and important decisions we must embrace in order to grow. One such pivotal decision is whether to build or buy our experimentation platform. Che Sharma, Founder & CEO of Eppo, provides a thoughtful framework.
A Simple "Build vs. Buy" Decision Framework for Experimentation
Brought to you by Che Sharma, Founder & CEO of
When you're committed to A/B testing, the question of whether you should build or buy an experimentation platform is inevitable. Our team has seen both sides of the "build vs buy" coin, and we’ve put together a framework to help identify factors around the success or failure of an internal platform and determine which approach is right for your company. Read on to dive deeper into the "build vs buy" debate.
TLC Chatter
Many of you had… colorful opinions… on the subject of SRM. Here was one of our favorite conversations. Will we be censored?! 🤪 We will not.
Erin Weigel: “Views” just seem like a vanity metric to report on anyway. That tells me nothing about the randomization split or the actual conversion metric, which I imagine is submissions.
Laura Stude: The test was on a welcome modal. They’re now saying there was “a bug in the implementation,” which seems like a coverup for “big ass bug in our system” if you ask me.
Amrdeep Singh Athwal: Yep sounds like a cover-your-ass statement.
Erin Weigel: Yeah. That’s some bullshit right there. 👍 They were splitting on the wrong thing then. Was it welcome modal vs. no welcome modal? Or two different welcome modals against one another? Maybe we dub 2024 “the year of the SRM” and we all go on an educational campaign explaining this concept to people. 😄 I’m in. @Shiva Manjunath @Lukas Vermeer you both on board too? 😂 We can make it a branded campaign.
Lukas Vermeer: @Erin Weigel: I think every year since 2019 has pretty much been “year of the SRM” for me, so sure I’m in for 2024 too. 😇😂 FWIW my explanation of SRM is here.
Do you have more to add to this conversation?
We think you should also check out Lukas’ SRM taxonomy of causes. After all those tears, maybe we should get some flowers? Okay, no? Just maybe a little bit of learning, then.
Learn about All-Team Experimentation with
Coming Soon: TLC Conversations & Industry Events
May 17 -TLC Conversation: Hypotheses are Overrated—What You’re Missing in Your Process with Shiva Manjunath
May 23 - Experimentation Culture Awards
Jun 4-7 - Marketing Analytics Summit Phoenix, AZ. Join my workshop on June 5! Use TLC20 to save 20% when registering*
June 6 - Convert X Women in Experimentation Annual Summit (TLC Sponsored!)
June 11 - TLC Mentorship Summit at #ExElite. This TLC Live Event is at the Hilton Metropole Birmingham National Exhibition Center (NEC), 13:45-17:00.
June 11-12 - Experimentation Elite 2024 at the Hilton Metropole Birmingham NEC (See you there!)
July 12 - TLC Conversation: Hidden Hazards: when client-side experimentation issues go unnoticed with Joey Kant
*Don’t forget that discount code for the Marketing Analytics Summit June 4-7, friends. Use TLC20 to get:
20% off when registering
The opportunity to attend a live broadcast of the Analytics Power Hour podcast with some of our favorites - Val Kroll, Julie Hoyer, Michael Helbling & Tim Wilson.
Fascinating talks about everything from data science & AI to connecting offline purchases to online behavior.
 | A guest post by
|