Tricks & Treats: Statistical Ghosts, Bayesian Angels and Frequentist Demons

TLC Connector: Halloween Edition | Teach. Learn. Converse.

and

Oct 30, 2024

Have you seen anything lately that isn’t there? Perhaps a statistical ghost? Some creepy sample mismatch ratio? ‘Tis the season to give our fears a face, turn them loose in the neighborhood, and feed them candy. We hope your October has been more treat than trick. Here are some full-sized goodies from our community to put in your pillowcase and eventually lord over your siblings.

- Shiva provided a Halloween CRO-themed costume:

- Aleksander Fabijan’s memorable scary pumpkin post about the statistical flywheel and how to find/address SMR. As a presenter at the inaugural Experimentation island in Feb. ‘25, Aleks will further illuminate how to grow experimentation, what it takes for a flywheel to fly. Also, atmosphere-shattering news: Josh Silverbauer has made Aleks’ intro song, “Learn Control from Fabijan.” Take your protein pills and put your helmets on because this fantastic oddity will surely be a universal hit!

Two CRO themed jack-o-lanterns — Credit: Aleks Fabijan

- Lizzie Eardley’s TLC talk about statistical ghosts, which are false positive data phantoms that materialize in our analysis and make us think we’re seeing an impact that doesn’t really exist. She gives ghostbusting tips to avoid wasting time and money on four of the most pernicious ghouls.

Bayesian Angels and Frequentist Demons

by Sven Schmit of

Check out Eppo

Angels and demons. Gods and monsters. When we can’t keep pace with mortality (or statistics), we invent myths that succeed where we fail. Fear works to keep us at attention, and this chilling post unburies the still-beating heart of our statistical fears: What’s random? What’s probable? What dangerous phantoms lurk in what we observe? What, ever, can we trust? In “Bayesian Angels and Frequentist Demons,” Sven frames our deepest fears and original conflicts. Truth, confidence, sensibility, cataclysm: these are the stakes when we unveil the horror of what we know and what we never will.

Click here to accept the stakes. . . if you dare.

Thanks to our sponsor

Check out ABsmartly

TLC Chatter

Here are the nightmares that keep the community up at night.

Anonymous: Zombie Metrics....you know, the ones that everyone puts into an experiment, and when they come alive, they drag you into a pit of death because teams either want to point at them and say, see, I was right, or WTF is that about? If only they knew about the zombie hunter named Bonferonni that would whip them into shape. Unfortunately, it comes at a cost because he also might kill your ability to see a signal in your treasured primary metric heroes!
Ezequiel Boehler: To have nightmares you need to be asleep and last time I checked people that run or help run experimentation programs don't have time for that 😛. Joking aside, I would say the phrase "This test hasn't reached stat sig yet so let's keep it running one more week" and all the alternatives it's what sends a chill down my spine because I think that once you hear that, you don’t want to open the dark rusty old closet at the end of the room with all the other stuff that might be happening on that program.
Iqbal Ali pointed out, “Zombie Metrics needs to be a movie. Am I the only person who would watch the shit out of that thing? On a similar note: phantom metrics. You know, when a product owner sees a result that isn’t there. Perhaps reading a metric that only they can see… Wouldn’t make as good a movie, though.”
Dylan Lewis gave us: “Statistical Vampires - Where they suck the life out of the room by having discussions on the (small) differences between Bayes and Frequentist until @Matt Gershoff (vampire hunter) enters the room. He shows them a mirror to verify their silliness and then uses practicality (garlic) to keep them away from the hero.
Merritt Aho dove in with a plethora of creatively named examples:
- The Chimera - when the false positive risk has not been managed properly
- The Siren - when stakeholders want to continue running a test because results are positive but insignificant
- The Troll - an underpowered test
- The Golem - primary KPI is awesome, but other key metrics are bad
- The Hell Hound - when it's a win overall, but segments produce opposing results
- The Leprechaun - when the key metric is unimpressively flat, but some side benefit has been created with very compelling results
This led Dylan Lewis to come up with another, “Frankenstein - what your site looks like if you only let AB testing make every decision for you”
- Merritt replied, ‘just one big red button that says "click here!"’
Tyler Buffington and Michael Trang gave examples of nightmare job posts with ridiculous expectations and low pay.
Georgiana Hunter-Cozens pointed out that beyond the nightmare of companies seeking ridiculous job expectations with insanely low pay, there is the terror that occurs when companies don’t “add salary to job ads, and then being confused why you’re applying for a job because you’re too junior/senior”