One of the first things I did when I joined Ro a few months ago? Implement an experimentation process. I adapted much of this process from my time working with Brian Baflour at HubSpot (more on his mad scientist approach to growth here).
Below is the email I sent to our growth marketing and product teams, and the exact resources we're using.
I've been spending the past few weeks thinking about our experimentation process, from our culture around experimentation to our tooling/infra to support it. Here are my thoughts and a proposal for us moving forward.
Blue line = a company that's slower to learn and slower to apply learnings Green line = a company that's faster to learn and faster to apply learnings
Green line co has higher batting averages because they know more about their users, products, and channels. They also capitalize on new opportunities faster.
Blue line co runs experiments and skips analyses. They leave learnings in the heads of individuals and fail to share them until team members leave or forget. Halfway through experiments, they realize that there are conflicting experiments or campaigns being run by other teams. They often have to stop and re-start experiments.
While I'm not labeling us the blue line co, I've noticed a few opportunities to improve our current approach to experimentation:
Bring more consistency to experimental design, measurement, and hygiene by codifying an experimentation process that follows best practices, resources, and templates
Prevent silo'ing of learnings within teams and individuals by creating a monthly growth learnings meeting across growth, product growth, & other stakeholders running experiments.
That said, I have a few asks of you...
Experimentation consistency:
Most of you are already tracking experiments in your own google sheets. I kindly request that you move neighborhoods. Please add your experiments to this shared experiment tracker that will enable us to gain better visibility of all the ongoing (and historical) experiments run across teams. Feel free to structure your team's sheet however you'd like, the key is we keep all our experiments in a central location. No need to add all previous experiments, but please use this for all new and upcoming experiments!
Use the experiment doc template if you don't use a spec. Link out to these docs in the experiment tracker.
Share learnings:
When an experiment is about to go live, please email the growth product and growth marketing email aliases with the subject line [Experiment Launch] {Your Experiment Name}. You can just copy and paste the experiment doc (from objective to experiment design section) and link out to the doc. When your experiment has concluded, please do the same (copying and pasting the results section) with the subject line [Experiment Concluded] {Your Experiment Name}.
Come and contribute to our first monthly growth learnings meeting in May (invite coming soon). Before each meeting, I'll ask that everyone contributes their learnings to the deck.
Ultimately, success from this process comes from having the diligence to stick to the process. It might seem like it's slowing you down at first, but it actually speeds us up as a collective. It means we're sharing learnings amongst each other faster, we don't have to re-instrument or throw experiments out as often. It's hard and it can be boring, but we have the opportunity to decide whether we want to be Green line co or Blue line co. I say we choose green :)
p.s. not sure how to design your experiment? whether you should run a multivariate or not? where to find more resources? Check out our new Experimentation Best Practices.