Headline Testing Orientation Guide
Headline testing is the process of using live experiments to get to know your audience on a deeper level. By tracking what kinds of headlines are most successful at getting your audience to actually click into -- and read -- your articles, you can better adapt your homepage strategy to maximize your content exposure, audience engagement, and visitor loyalty.
When you create and run a headline test, new visitors that come to your site are served one of the headline options and we record which version people are clicking into the most. What’s unique about our testing methodology is that it actually exploits success of winning headlines by 'playing' them more frequently as they prove to outperform other headlines. Then the winning headline is played permanently to your entire audience.
If you’re interested you can read more information about the methodology we use, called Thompson Sampling, here.
Headline Testing is just one of the ways that you can be using data to make the most of your homepage audience. So that you can quickly incorporate headline tests into your homepage strategy without interrupting your team's workflow, the Headline Testing tool is built into the Heads Up Display itself.
To get started, just make sure that you've installed the Heads Up Display, then just and select any link that’s receiving a pin. From there, simply toggle from the 'Performance' tab to the 'Headline Testing' tab at the top of the card and you’re ready to get started.
Please note that Engaged Headline Testing is a premium feature. If you have any questions about your current plan, reach out to your Customer Success Manager, or send an email to firstname.lastname@example.org.
Starting a New Test
Select a Headline to Test
Select a headline on the page that you want to start a test for, and click on its Heads Up Display pin. Toggle into the Headline Testing tab on its pin card, and enter the various headlines you want to test. The first headline will be auto-filled with the headline that appears by default in the page, and as additional headlines are entered they appear inline on the page so you can preview their appearance before starting the experiment.
While there's no limit to the number of headline variants that can be tested, remember that an experiment's progress will be slower the more headlines are added. When you’re getting started, we recommend choosing two or three headlines, but as your audience grows you can create tests with more and more headline variants.
Then just select 'start' to begin testing.
Multiple Headline Candidates
If any of your homepage articles feature multiple headlines, for example a main headline and a sub-headline, or a section name as well as a headline, you’ll be prompted to specify which one you want to run an experiment for.
Please note that you must select one of the possible headlines, and cannot run multiple tests at the same time.
Viewing your Active Tests
Following a Test's Progress
To monitor the progress of a test, select the pin next to the headline currently being tested. Active deadlines experiments will be indicated with a pie chart representing the number of headlines being tested and the play percentage of each headline. Headlines with a higher play percentage are ones that are performing better.
Editing tests, adding variants, removing variants, and editing the copy of headline variants will automatically stop and restart the test.
Reviewing Completed Tests
A Completed Test
The test ends when the winning headline has been determined with 95% confidence and Chartbeat begins to automatically serve that headline 100% of the time.
To make sure that tests come to a conclusion as efficiently as possible, we have an alternate way to determine a winner, called 'soft convergence'. If 20 minutes have passed and we're 95% confident that no headline is better by a margin of 25%, the leading headline will win.
To see a full report of all currently running, completed, and stopped tests, select the menu icon in the bottom right of the screen, or the 'All Tests' button on the bottom of the Heads Up Display dock on the newest version.
The Test Archive displays a list of all current and completed headline experiments, the engagement and click-through-rate performance graphs, and the number of trials for each headline.
With the “Download All Data” button at the top of the page, you can export your entire headline testing results archive to CSV. Manipulate the data however you prefer, to see successes and trends over time.
Use the Test Archive as a resource to analyze patterns in your audience’s behavior and preferences.
Headline Testing Best Practices
Generally speaking, when you’re deciding on additional headlines to use, you should...
- ...leverage your own experience and insight. When deciding on headline variants, consider what kinds of headlines have been successful in the past. Maybe one that’s interrogative, one with numbers, and another written in the second person.
- ...make sure your headlines are different. You’re testing to see how your audience reacts to different styles and tones, not slight changes in word order or punctuation.
- ...analyze your results. Look back at those headline writing strategies you started with. Were your hypotheses correct? Every time an experiment completes, you have a chance to learn a little bit more about your audience’s preferences and how they respond to different styles of headlines.
- ...apply your findings time and time again. Some of the best headline testers write wrap reports and have daily meetings to go over results. Creating and sharing a summary of the insights you gained from previous experiments helps all editors and content producers benefit and learn.
- ...test again. And again. Test deliberately and in ways that confirm or deny a single hypotheses.
For more tips and best practices for headline testing, check out our Headline Testing launch webinar with our friends from Bleacher Report!