Despite the demand for platforms that let developers experiment with different versions of apps, the infrastructure required remains relatively complex to build. Beyond data pipelines and statistical methods, and experimentation infrastructure relies on analytical workflows often sourced from difficult-to-configure cloud environments.
Plenty of startups have emerged in recent years to abstract away the app experimentation infrastructure, including Split, Statsig, and Optimizely. A more recent arrival is Eppo, which today emerged from stealth with $19.5 million including a $16 Series A Menlo Ventures and a $3.5 million seed round led by Amplify Partners.
According to CEO Che Sharma, Eppo was inspired by his experiences building experimentation platforms as an early data scientist at Airbnb and Webflow, a website builder. “Nothing in the commercial landscape provided the power of experimentation systems like Airbnb, which meant building the same system over and over,” he told TechCrunch via email. “I built Eppo to leverage the modern data stack and the latest in causal inference literature, allowing companies to tie product team efforts to business metrics like revenue, with boosted statistical power.”
Sharma concedes that the app experimentation space is becoming congested, if not saturated, with competitors. But he says that Eppo is differentiated by its analysis tools, which use confidence intervals to make it ostensibly easier to understand and interpret the results of a randomized experiment. Eppo also supports experimentation with AI and machine learning models, leveraging techniques to perform live experiments that show whether one model is outperforming another.
Sharm claims that Eppo is one of the first commercial platforms to include CUPED variance reduction, an approach that tries to remove variance in a metric that can be accounted for by pre-experiment information. For example, say a property-booking company runs an experiment aiming to increase the number of daily bookings that they receive. The number of bookings per property per day can range from zero to thousands. But the average bookings-per-day for each property can often be determined before the experiment; through CUPED, this knowledge can be used to test whether properties start to receive more, less, or about the same number of bookings-per-day after the experiment compared to before it.
Image Credits: Eppo
“Of all products in the modern data stack, experimentation has one of the clearest relationships to revenue return on investment because it injects C-suite- and board-level metrics into every decision a product team makes,” Sharma said. “Especially in tough recession markets, the C-suite needs their product teams to probably drive business metrics like revenue. Without experimentation, product teams are in a constant cycle of shipping, pointing at engagement- and click-level vanity metrics, but never having confidence that the business’ financial outlook has improved from their work.”
Sharma also asserts that Eppo is more privacy-preserving that most experimentation platforms because it performs all of its data computation in the cloud, on Snowflake. As opposed to collecting clicks, engagements, and other personally identifiable information, the Eppo platform only stores aggregated, anonymized experiment results.
“We are major evangelists of a new way of building analytics products that is much more privacy-focused,” he said. “Other experimentation platforms require sending the universe of data to them, essentially storing replicas of each customer’s own data ecosystem.”
Of course, even the best experimentation software isn’t helpful if employees don’t use it. Buy-in can be tough to achieve, in part because experimentation can expose the true, sometimes lower-than-anticipated success rate of a product development. Even at tech giants like Google and Bing, the vast majority — about 80% to 90% — of experiments fail to generate positive results.
But Sharma, while declining to answer a question about revenue, says that uptake remains strong. Eppo’s customer base grew over the past year to include Goldbelly, Netlify, Kumu, and at least one unnamed Fortune 50 company, he said.
“We have seen a resurgence in the interest of experimentation with the recent market downturns. Across our existing customers and our customer pipeline, we have seen this pattern: layoffs are centered on teams building net-new future product lines that won’t return revenue quickly, and are instead centering on core product development with a focus on revenue which is inherently experimentation-centric,” Sharma said. “Concretely, despite many customers having layoffs, across the board none of the experimentation teams have had layoffs.”
With the new funding, San Francisco, California-based Eppo plans to expand its team from 15 employees to 25 by the end of the year.