How We Turned Website A/B Test Hypotheses into Measurable CRO Wins for an AI SaaS

Here are the A/B tests that we conducted and their results:

1. A/B-test: Homepage Headline Test

Hypothesis: 

Replacing the generic headline with a more benefit-driven and ad-matched headline will increase message relevance, clarity, and user motivation. This should have resulted in higher CTR on hero CTAs and better engagement with the first fold.

Implementation: 

The benefit-driven headline demonstrated reliable performance superiority over the control headline. The sustained positive trend across weeks one, two, and three indicates that the benefit-driven headline resonated more effectively with users than the control headline. 

This improvement directly supports the hypothesis that aligning headlines with ad messaging and emphasizing clear benefits increases perceived relevance and user motivation to engage with hero CTAs. 

Outcome:

The variant outperformed the control in terms of conversion rate (+12% improvement). Based on the consistent performance pattern and substantial sample size, we implemented the benefit-driven headline as the new default. This change was made to deliver measurable improvements in CTR and first-fold engagement across the homepage.

2. A/B-test: WhatsApp AI page CTA label

Hypothesis:

Replacing a generic CTA button inscription with a lower-friction, WhatsApp-specific label will reduce perceived effort, increase relevance, and improve CTA CTR and trial starts.

Implementation:

After three weeks of testing, the WhatsApp-specific CTA label consistently outperformed the control, delivering a sustained and growing uplift in CTR over the final two weeks. 

Outcome:

The variant represented a +46% relative increase in engagement. This confirms the hypothesis that reducing perceived friction and increasing contextual relevance through a tailored CTA improves user response. Based on the clear positive trend and final results, the variant was rolled out as the new default CTA on the WhatsApp AI page.

‍3. A/B-test: CTA placement on Product page

Hypothesis:

Adding prominent CTAs directly beneath the hero title and subtitle will make key actions more visible, capture high-intent users earlier, and improve overall trial-start and demo-request conversions.

Implementation:

Over the 4-week testing period, the variant with prominent CTAs placed directly beneath the hero section outperformed the control. While overall traffic volume remained relatively limited and statistical significance was not yet fully established, the directional trend has been consistent throughout all four weeks.

Outcome:

The test variant represented a +241% relative uplift in CR. The results supported the hypothesis that surfacing high-intent CTAs above the fold increases visibility of key actions and improves trial-start and demo-request conversions.

Based on these findings, we implemented the variant as the default experience, with continued monitoring as traffic scales to validate long-term impact.

4. A/B-test: Pricing page CTA button color

Hypothesis:

Changing the CTA buttons from grey to a high-contrast, branded color will increase visibility, perceived interactivity, and overall CTR for trial-start actions.

Implementation:

After three weeks of testing, the high-contrast branded CTA variant consistently outperformed the grey CTA control on the pricing page. As traffic scaled throughout the experiment, the improvement remained stable, confirming that the effect was not caused by early-stage variance.

Outcome:

The variant represented a +92% relative uplift in trial-start actions.

These results strongly support the original hypothesis that increasing CTA contrast and visual prominence improves visibility and user interaction on high-intent pages such as pricing.

Based on these findings, we implemented the high-contrast branded CTA as the default experience on the pricing page and continued monitoring performance to validate the long-term impact.