How It Works
TubeBuddy's A/B Testing works by having you create a variation of a video's metadata (this could include the Thumbnail, Title, Tags or Description). We then alternate your video's metadata every 24 hours at Midnight PST (to line up with YouTube Analytics statistics). The test completes either based on a certain number of days that you picked or based on statistical significance being achieved.
Important Notes on the 'Winning' Test
- In a perfect world, when you run a A/B Test in TubeBuddy, the same video would have the highest CTR, the most views and the highest avg view duration. Unfortunately that doesn't happen every time so it might take some user analysis to choose which version of the video you want to go with after the test is complete.
- One important thing to note is that CTR statistics not only have many many more data points (because the average video gets 40 to 80 impressions for each 'view' the video receives), but it also eliminates biases such as day of the week or how close to the publish date the test is. When you look at views, watch time, and all other stats displayed on screen in the lower section, those are 24 hour blocks of time and can be affected by day of the week. For example, maybe your particular audience is more active on Mondays than Tuesday in general. When looking at CTR and the top line of stats, we're getting impressions and clicks (a ratio) and this ratio is not affected by the day of the week or anything else which is why we declare the winner of the video based on CTR.
- There are cases where the winning version has a higher CTR but a lower view count. Generally speaking, you should always go for the higher CTR. For example, if you happen to get 100 views one day and 1,000 views the next day, the winner might still be the first day because your video was shown to 500 people on the first day screen and 20% of them clicked where as the second day it was shown to 100,000 people and only 1% clicked. The title/thumbnail on day 1 obviously was more appealing and clickable so in the future, you'd want to use that version.
- There are also cases where the winner version has a higher CTR but a lower average view duration. This example is a bit trickier and not always as clear cut. Generally speaking, you always want a higher CTR but if you have a higher CTR and much lower avg view duration, then it could mean that your thumbnail doesn't accurately represent your video content. People are clicking to watch based on your thumbnail but then the video itself isn't what they were looking for. In these cases, it's important to look at the 'Watch Time per Impression" box in the top right hand corner. This will show you based on every time your video thumbnail is displayed on screen (whether or not the user clicks), the expected total watch time. So if it's displayed 100 times, clicked twice and each time the user watches 3 minutes of the video. Your watch time per impression is (3 * 2 * 60 seconds) / 100 impressions = 3.6 seconds. The YouTube algorithm is always trying to maximize total watch time per impression so we recommend going with the highest value in those panel in these cases. That said, there might not be statistical significance in these cases and the test could be inconclusive. Try making a more meaningful change to thumbnail and running a new test or, use your best judgement as to whether or not the thumbnail change you made actually could be 'tricking' people or not accurately representing your content.