Experiments let you compare your live quiz with different versions to see which performs better. Test changes to questions, design, recommendations, or other quiz elements, and use performance data to choose a winner.
How Experiments work
Each experiment compares two versions of the same quiz:
Live version: The version shoppers currently see.
Variant B: The alternative version you want to test.
Shoppers are divided between the two versions based on your selected traffic split.
You don't need to create a separate quiz or change your existing embed code. The experiment uses different versions of the same quiz. Nothing changes for shoppers until you start the experiment.
Create an experiment
You can create an experiment from either location:
Open your quiz in the editor and click Create experiment next to Publish.
Go to Experiments in the left menu and click + New experiment.
Step 1: Choose your quiz version
If you're starting from the Experiments page, select a quiz with a live version.
Click Next: pick a version.
Select the version you want to test. You can choose your Current draft, a Manual save, or an Autosave.
Preview the selected version on desktop, tablet, or mobile.
Click Use this version.
Step 2: Configure your experiment
Review the Live version and Variant B side by side.
Enter a descriptive name for your experiment.
Choose how traffic should be split between the versions:
Equal: 50% of shoppers see each version. This is typically the fastest option.
90 / 10 holdback: Most shoppers continue seeing the live version.
Custom: Choose your own traffic split.
Click Next: Decide on.
Your experiment is saved as you go, but it won't start until you activate it.
Step 3: Choose your success metric
Select the primary metric that will determine which version wins.
Metric | Best for testing |
Completion rate | Whether more shoppers finish your quiz |
Email opt-in rate | Whether more shoppers subscribe |
Order conversion | Whether more shoppers place an order |
Revenue per visit | Whether your quiz generates more revenue per visit |
Click Other metrics for additional options, including page visits, product page clicks, button clicks, answer selections, phone opt-ins, and average order value.
Step 4: Protect important metrics
You can also select Protected metrics to make sure improvements in one area don't come at the expense of another. For example, you might test a change designed to increase completion rate while protecting order conversion. If a protected metric appears to perform worse, the experiment won't automatically publish a winner.
Step 5: Choose how your experiment ends
Experiments offer four ways to decide when a test should finish.
Option | What happens |
Ask AI to decide | AI reviews the results and can recommend or publish a winner when conditions are met |
Decide when it is clear | Automatically publishes a winner once there's a clear result, after at least seven days |
Decide on a date | Evaluates results on your chosen date and follows your selected fallback if there's no clear winner |
I decide | Keeps the experiment running until you manually end it |
You can change how your experiment ends while it's running. Under While it runs, you can also enable weekly email updates and reminders for experiments running longer than six weeks.
Note: Ask AI to decide uses credits each time AI evaluates your experiment, even if no winner is selected. The cost is displayed during setup, and you can limit the number of AI runs.
Step 6: Review and start your experiment
Review the selected versions, traffic split, success metric, protected metrics, and ending conditions.
Optionally, add a Note for the AI explaining what you changed or what you expect.
Click Start test to begin immediately, or Save for later to keep it as a draft.
Once started, shoppers are automatically divided between the two versions.
Step 7: Monitor your experiment results
Open Experiments to view your tests under Running, Ended, All, or Drafts.
Each metric displays a confidence label:
Collecting: Not enough data is available yet.
Leaning: One version is performing better, but the difference isn't conclusive.
Clear: There's strong evidence that one version performs differently.
Only your selected success metric determines the winner. Other metrics help you understand the overall impact of your changes.
Keep in mind:
Automatic decisions require at least seven full days of data.
Experiments need sufficient traffic to produce reliable results.
A full progress bar doesn't guarantee a clear winner.
Smaller performance differences generally take longer to confirm.
Manage an active experiment
You can review your results, adjust the traffic split, change the ending conditions, or end an experiment. Changing certain settings, including the traffic split, live version, or decision metric, starts a new counting period. Previous results remain available in the experiment history.
Experiments also monitor whether shoppers are being distributed between versions as expected. If the traffic split becomes unreliable, automatic decisions pause, and a warning appears.
Tip: Test both versions on your storefront to make sure they load and work correctly.
Use Analytics to identify opportunities
Before creating an experiment, review your quiz performance in Analytics to identify areas you'd like to improve. For example, you might notice a high drop-off rate on a particular page and decide to test a shorter version of that page.
For more information, see Understand your quiz analytics.