Broadcasts
A/B test a Broadcast
Send up to four versions of a Broadcast, see which wins on clicks, opens, or replies, and send the winner to everyone else.
In this guide 5 sections
An A/B test sends different versions of a Broadcast to different people and measures which works better.
Two ways to run one
| In the app | With your AI or the API | |
|---|---|---|
| What varies | The whole email: design and subject | Subject, design, content, or sender name |
| Versions | 2–4 | 2–4 |
| Who gets the test | Everyone, split evenly | A sample (5–100%), the rest wait for the winner |
| Winner | You pick it after sending, for the record | You pick it, or Banger picks automatically |
In the app
- In the Design step, choose A/B test this. Banger adds a second version.
- Edit each version. Choose + Variant to add up to 4. Versions must differ.
- Continue as usual. Review shows each version’s share, like 50% / 50%.
- Choose Send a test to me and pick a version to check each one.
- Send. The audience is split evenly, and every recipient gets one version.
After sending, the Broadcast shows A/B test results per version: delivered, clicks, replies, lift, and confidence. Pick marks the winner. Since everyone already got a version, picking sends nothing more.
Test on a sample, then send the winner
Ask your AI to set this up. You choose:
- Sample size: for example 20%. The other 80% wait.
- Metric: human clicks (default), human opens, or replies.
- Decision: manual, or automatic after a set time (default 4 hours) once one version is ahead with 95% confidence. At the deadline, the leader wins even if it’s not yet certain.
- Guardrails (optional): a maximum unsubscribe or complaint rate a version may not exceed.
When the winner is picked, the rest of the audience gets it. If you stop the test instead, they get the first version. A test can only start while the Broadcast is a draft or paused.
How results are counted
Opens and clicks count likely humans only. Bot clicks and privacy-proxy opens, like Apple Mail’s, are excluded. Small samples give wide confidence ranges: aim for at least 100 people per version.
Try it with your AI
Set up an A/B test on my draft Broadcast with two subject lines. Test on 20% of the audience, measure clicks, and pick the winner automatically after 4 hours.