A/B testing Facebook ads

Everyone tells you that you need to test test test your advertisements. Keep on changing small things, let them run and compare results. Makes sense, right? But it was surprisingly hard to find information on how exactly you should DO this — seems like everyone writing about split testing assumes that of course you know how to do it already.

I had one ad running in my Facebook campaign, with some 30,000 impressions and 20 clicks on it. I created 3 variations on it, changing the picture and the ‘call to action’. So now I had four ads:

  1. Picture with white background, “Upgrade your style now!” (the original)
  2. Picture with white background, “Free shipping!”
  3. Picture with black background, “Upgrade your style now!”
  4. Picture with black background, “Free shipping!”

All ads with the same CPC bid, same demographic targeting. Submit the 3 new ones, wait them to be approved and then wait for few days to check results.

Results — none of the new ads were getting any impressions at all, or just few hundreds of them. Meanwhile the original ad kept showing 5,000 – 10,000 times a day. What gives?

Looks like Facebook does the same thing as Google by  default — ads with better historical performance are shown more often. From their documentation: “This means that if your ad has performed well in the past, it has a better chance of showing up in the future.”

Sucks for split testing. I changed the bidding on all ads to be based on impressions, not on clicks, and this helped, all ads were now showing. The impression count is still a bit off, with the best performing ad (most clicks) getting 33,00 impressions in a week and the worst performing getting just 26,500. So the ‘performance drift’ hits you even over just few days.

Now it was time to read the documentation. Oops. Looks like Facebook has an article about ad testing:

So in summary:

  • Create each ad variation in a separate campaign to ensure even distribution of impressions.
  • Only change one thing at a time in the ads (image, one line of text etc).
  • Be sure to have same bid on all ads. Bidding on impressions seems to be cheaper on new ads.