When it comes to measuring the real impact of paid social, beauty brands face a unique challenge, especially given the impact of brand strength and their omni-channel distribution business models. Platform-reported metrics often tell one story, but what's the actual incremental lift?
For Hourglass Cosmetics, a luxury beauty brand with significant retail presence through Sephora and other channels, this question was critical—especially when e-commerce represents just a fraction of total revenue.
Working with Rockerbox's Professional Services team, Hourglass designed and executed a rigorous geo holdout test to measure Meta's true incremental impact on both overall conversions and new customer acquisition. The results provided clear, statistically significant evidence of Meta's impact and validated their investment in the channel.
Hourglass came to Rockerbox with a specific question: How does Meta advertising affect incremental sales and new customer sales, and how can we measure this impact rigorously?
This type of incrementality testing is especially important for brands like Hourglass, where:
The path to a successful test began with collaborative planning between Hourglass, their media agency, and Rockerbox's Professional Services team. The team created a test design which would use a 10 DMA hold out to measure Meta's overall impact across the country. The ten DMAs selected had in aggregate very strong relationships to the control markets, with correlations of 0.96-0.98 for new customer conversions and revenue, and 0.97 to 0.99 for all customer conversions and revenue, resulting in a very high powered test.
The test ran for six weeks from May 5th to June 16th, 2025 during which time all Meta campaigns were paused in the 10 holdout DMAs. Rockerbox analyzed the impact of Meta during the test based on e-commerce transaction data collected directly from the Hourglass site by Rockerbox pixels.
The geo holdout test revealed statistically significant lift across both overall performance and new customer acquisition.
During the test period, the results showed:
For new customer acquisition specifically, the results were equally compelling:
Given that e-commerce is just one of Hourglass's sales channels, the full omni-channel lift from Meta is likely several times higher and above the brand's breakeven threshold.
The geo holdout test provided Hourglass with several critical insights:
The test proved that Meta advertising generates real incremental conversions and revenue beyond what would occur organically. The 10%+ lift in conversions demonstrates that Meta is not simply taking credit in attribution for conversions that would have happened anyway.
With a 10.6% lift in new customer conversions and a prospecting-specific iCPA of $190, Hourglass validated that their Meta prospecting strategy is effectively reaching and converting new audiences.
Important context: These results measure ecommerce impact only. Hourglass's Sephora retail revenue is approximately 8x their ecommerce revenue, meaning the incremental lift observed in ecommerce likely represents just a fraction of Meta's total business impact.
While the ecommerce-only iROAS of 0.51 might appear modest, if Meta drives similar behavior in retail (which is highly likely given the halo effect of digital advertising), the true total iROAS could be 4-5 when retail impact is included.
Rockerbox can directly analyze the impact of media on sales in third party retailers such as Sephora or Amazon when the data is made available. Store level sales data for retail chains makes an ideal data foundation for testing and with access to that information, Rockerbox would have been able to prove the above hypothesis and understand the impact of media on both e-commerce and Sephora revenue.
The high pre-test correlation (0.96-0.99) and strong statistical significance (90-99% confidence) mean these results are highly reliable. This level of rigor gives the marketing team confidence to make strategic budget decisions based on the findings.
This test showcases the power of combining first-party measurement with rigorous incrementality testing:
For brands looking to move beyond platform-reported metrics and understand true incremental impact, geo holdout testing offers a gold standard approach—and Rockerbox makes it accessible and actionable.
Want to validate your channel performance with rigorous incrementality testing? Rockerbox's platform makes it easy to design, execute, and analyze geo holdout tests that deliver statistically significant insights. Learn more about Rockerbox's testing capabilities.