Fresh Starts: First Tests, Experiment Plans, & Career (Re)Starts

TLC Connector | Teach. Learn. Converse.

, and

Aug 28, 2024

Hi TLCers!

Locally, kids shuffled back to school this week, putting us in mind of the motivational power of fresh starts (see Katy Milkman’s work). We have fresh news to announce (!), but we’d also like to recognize our community members who support fresh starts in our industry.

Kevin Anderson and Tom van den Berg come to mind as start-artists. Check out their helpful resources:

Experimentation Jobs - Need a fresh start? This is the #1 job board for experimentation roles. Here’s a handy link to the evergreenly-latest recent job postings.
CRO Weekly - Tom’s fantastic newsletter wherein experts share stories and career interviews about their fresh starts in the industry, first tests, etc.
Experimental Mind - Kevin’s digital space for sharing what’s new and next in experimentation.

As for fresh starts in the TLC: We are thrilled to welcome two new newsletter sponsors—ABsmartly and our longtime friend and very first partner ever, Conductrics! Look for great content from each of these partners in the coming months.

We’ve also posted updates about new event (ABsmartly & GrowthBook - joining Conductrics) and community (Creative CX) partners for Experimentation island (#Ei2025), and we’ve updated our (fabulous🎶) performer details. Check out the updated website for more details, and claim your tickets before they’re all gone!

Plan to Connect: TLC Event Guide

Our partner Eppo also brings us content this month about beginning a test and all the work that goes into creating your experimentation plan.

Let’s begin!

Your Comprehensive Guide To Creating an Experiment Plan by Kristi Angel of

Check Out Eppo

Are shortcuts in planning and execution sabotaging your experimentation efforts? Experimentation is a complex process, and a lack of thorough and thoughtful planning can lead to unreliable results, doubled runtimes, and wasted resources. This guide highlights the importance of writing and sharing detailed experiment plans to avoid pitfalls and enhance experiment quality.

Why not launch trustworthy experiments on the first try? Read on to learn how to create effective experiment plans.

Why We Chose to Sponsor the TLC by Matt Gershoff of

Check Out Conductrics

The TLC is about community, inclusion, and shared learning—the success of which requires approaching knowledge with a generosity of spirit rather than as a way to one-up others. Ultimately, it is about being of service to others.

The idea of service to others and customer empathy underpin the work we do at Conductrics. We believe that experimentation is neither a magic input/output machine nor a procedure for achieving 'scale.' Rather, we believe that the true value of experimentation lies not in the technology or in any particular statistical method but in providing unparalleled service: We use a principled process to help encourage and support organizations to be both intentional and empathetic when making decisions.

Ultimately, we can only be effective if we are honest with others as well as ourselves. This means putting in the effort to be educated about what experimentation and statistical inference can and cannot answer in each context for each use case.

This is what the TLC is for and is why we are such strong supporters of it and Kelly.

(Awe, shucks! Thanks, Matt and the whole Conductrics crew. Thank you for supporting the TLC community and for always answering our questions—no matter how late at night… I’m so excited to get some of your thought leadership here finally!)

Join the TLC

TLC Chatter

This month, we’ve been talking about beginnings in many forms; fittingly enough, beginnings were also a topic of discussion on the TLC Slack! Shiva Manjunath once again prompted the discussion: “Got another Q for the community: What was your actual first A/B test?” Here’s a small sample of what you had to say about your first A/B tests:

Lucia van den Brink: “Control: formal written text vs changing whole shop (omg!) into a more simple and playful written text at a Telco. And no, didn't move the needle, but was a hell of a test. Built with a custom-built testing platform from an agency - and this is how I got hooked to Experimentation.”
Florent Buisson: “Adding a progress bar at the top of the screen on a multistep form, plus a few minor tweaks. This was so wildly successful that for a little while, my team was known internally as the progress bar people.”
Dylan Lewis: “Changing the try for free length of time from 30 days to 7 days (in 2004)”

Mariana shared her first test - and it that it was still running!

Marina Ostrowski: “Adding a booking preview on the rates page of the hotel I work with! Still don't have results, because it's in progress, but high expectations on this one — win or lose, it'll always be my first test 🔥

This exchange between Erin, Els, and Paula was just brilliant!

Erin Weigel (#Ei2025 speaker): “I’m pretty sure it was making an email marketing campaign more accessible and usable by removing text in images and making it 16px plain text. I realized many people were likely missing the content because images didn’t always automatically download, so the message and coupon was basically lost. It was a good one. 😄
- Els Aerts: “@Erin Weigel, Accessibility and usability fundamentals always (okay maybe not always always) win. What a great first test!
- Erin Weigel: @Els research told me that was what I needed to spend my time on. 😄
- Lucia van den Brink: @Erin Weigel doing ab testing right from the start
- Els: @Erin Weigel, It’s Friday night. I’m having (a) Negroni(s). Do you want to make me cry?!
- Erin Weigel: @Lucia van den Brink, I had no idea what I was doing. 😂 Also, I annoyed the heck out of the data analyst with my incessant and ignorant questions. 😝
- Paula Sappington (#Ei2025 speaker): Preach, folks - preach.

Join us at #Ei2025!

Then, perhaps the most entertaining share thanks to both image AND sound. I promise - you don’t want to miss this story. SQUEEZE THE WOOKIE!

Natalie Thomas: “ When we were still a five person agency, we had a Wookie plushie that made a sound when you squeezed it. We squeezed it every time we launched a test. It was a right of passage to squeeze it the first time and the excessively long sound really helped circumvent the stress of launch. It now lives in a glass case on our awards shelf, in case you’re wondering.
- “[click to] enjoy the excruciating duration of the Wookie sound”

Why We Chose to Sponsor the TLC by

Check Out ABsmartly

Our top 10 reasons for why we chose to partner with TLC.

We believe in Kelly as a community builder and support her fully
We support cool humans doing cool shit
We believe in the power of community, and its power to advance experimentation
We are an experimentation platform built by experimenters, and we enjoy collaborating with other experimenters
We are an underdog in this space. We are a relatively small yet hugely passionate and ambitious team and visibility is important for our brand
We believe in creating and fostering safe spaces for people to discuss, learn and innovate
We are aligned in our values; Experimentation, Empathy, Energy, Excellence
We believe this is one of the most thriving and active communities for experimentation and if we can play just a small part it would be an honour
We are on a mission to build one of the best platforms out there for product teams to experiment. To do that, we must stay abreast as to what experimenters need, where they get stuck, how they experiment.
Did we mention we love Kelly?

(Dammit - y’all are really making me blush! Love you, too! Thank you for everything you do—for the TLC, for the community—and for the industry.)