From Lab to Real World: Why Experiments Matter for Startups
And How to Run Them to Achieve High Growth
In my last article, I talked about the dangers of scope creep and compared my current home renovation to software development when it comes to this challenge. Our manageable kitchen update has exploded đŁđ„ into a massive home renovation that has even expanded to the outside of the house.Â
This has hit our pocketbooks hard and is keeping us entirely out of the house for almost a year. đ« Itâs somewhat natural and timely to compare this to countless years of software development. Iâve seen many projects go off the rails because someone wanted to add âjust one more featureââlike an outdoor kitchen on the patio. đ€đ
Continuing on with my analogy, Iâm exploring a related topic: real-world testing. When it comes to both home renovations and software development, itâs one thing to see plans on paper...but itâs a whole different ball game to experience them in the real world.
In the case of our home, everything sounded wonderful with the new design, and the back-and-forth discussions created an enormous level of excitement. The drawings then showed up from builder, and again it all looked fantastic ⊠on paper. Even browsing through the Internet looking for ideas created a âbuzzâ on the renovation.
But my gut told me we shouldnât approve it before we âlived itâ. So, we walked through our ripped apart kitchen and patio. We measured out on the subfloor where everything was supposed to go. The build even laid out roughed-in wood to show where islands, cabinets, lights and appliances would go. Â
During this process, âwe the usersâ realized that the space between the counter and island was too narrow to pass through easily and that the lights werenât lined up with where weâd be working. Maybe that sounds trivial, but we want a kitchen thatâs functional (not frustrating) and we wanted to get it right before it was fully built.
By experimenting within the space, we were able to provide feedback to the builders and sign off on the changes with confidence. We made a lot of changes to our renovation including eliminating entire cabinets, changing the entire layout of the island, planning gas lines and outlets for things we wanted down the road. Now the builders can execute the plan with far more confidence on the end result than if we didnât experiment. As âusersâ weâre also happy that we did the experiments.
When software development is done right, it follows this same process. Prototyping and storyboarding isnât enough. You need users to get in there, play around, and experience it. You need to do real-world tests and experiments to learn whatâs working and whatâs not. And you need these to happen as early on in the process as possible so you can be confident with the plans going forward.
Need further convincing? OpenViewâs 2020 SAAS Product Benchmarks survey shows that running experiments correlates strongly with high growth rates amongst startups. In fact, about 60% of high growth startups report running 1-3 experiments per month.Â
How does your startup stack up?
If youâre looking to run more experiments, here are four tried and true tips:
Make someone responsible for running experiments. At Charli, this is our growth team, but it could also be a group of people assembled from different teams who have the expertise to contribute. Clear accountability will increase your chances of success. The OpenView survey found that companies with at least one growth employee are 2x as likely to run experiments on a regular basis.
Tie your experiments to your overall goals. These could be your corporate or product-related goals. Here, at Charli, our experiments are always tied to user activation, retention and growth -- so we use experimentation to test different ways of delighting users, deepening their engagement and creating +1 new users for each acquisition. Testing isn't just about throwing stuff at a wall and seeing what sticks.
Create an approach or framework for how you test and measure experiments. It is important to understand what you are trying to achieve with the experiment and how youâll gauge itâs success or failure. There has to be a methodology behind the experiment whether that is formally or informally put together. You will be learning and evaluating the results and you will even learn a great deal from experiments that fail. Create a plan and stick to it, while leaving some flexibility to pivot as you learn. Involve key stakeholders from the outset so everyone is onboard and working from the same playbook.
Take your experiments out of the lab. Sure, you can test-run your product internally, but the real magic comes when you introduce real users in their own environments. Donât be afraid to do this early, even if your product isnât the gold-plated version. At Charli, we started right out of the gate with a group of early beta testers. Then, in October 2020, we expanded to a larger public beta group. Yes, it was scary showing both groups our product before it was perfected, but it was a critical step and our learnings were immeasurable. Psst - If youâre still not part of our public beta group, itâs not too late to sign up here.
Whether youâre managing a home renovation, a software development team or even an entire startup, experimentation is critical. Donât wait too long before you test drive your product in the real world, or you might just end up with a product that doesnât take off or a kitchen that makes cooking a chore.