How It Works
TubeBuddy's A/B Testing works by having you create a variation of a video's metadata. We then alternate your video's metadata every 24 hours at Midnight PST (to line up with YouTube Analytics statistics). After 14 days, we can show you how the original performed against the variation.
- This is not a "true" A/B Test. When you do an A/B test on a Website for example, you can show each visitor a different version of your website. Alternating back and forth thousands of time per day. Unfortunately YouTube Analytics data can only be gathered 1 day at a time. Because of that, we can only switch your video back and forth once every 24 hours. This can potentially skew the results but it's the best we can do with what YouTube makes available.
- We chose a default length of 14 days so that both the original and variation occur on every day of the week once. They are both shown on a Monday once, a Tuesday once, etc. This helps even out the data if for example, you always get a big viewership on particular days of the week.
- Now that YouTube offers real-time analytics it seems like there is at least potential to have tests that alternate faster than once per day. But there are some problems. For example, it would require you to leave a browser tab open all day on the analytics screen so we could capture the data. We would also need to figure out how to capture and parse the data which is a complex process. Finally, updates to Thumbnails can be delayed at times and each time you make an update to your Title/Description/Tags, it takes YouTube time to re-rank your video in Search and Related. That delay could skew the results.
- Careful with new videos. Because we can only alternate the original every 24 hours, if you start an A/B test on the day that a video launches, you generally won't get accurate results. The first day will have many more views than the following days. We are considering updating the code to eliminate views from your Subscribers in the test results because then it would show all of the organic views and eliminates views that came from people who were pushed your video via email or subscriber feed. Again, just be careful with new videos when looking at the results. Most people use A/B Tests for videos that were launched more than 1 week ago and you can use it to find what types of videos your audience clicks on in general and then apply those findings to future videos.
- Generally speaking, the most views a video has, the more 'statistically significant' the results will be. If your videos only get 25 views per day, then there is much more room for chance and random events affecting the results compared to a video that consistently gets 2,500 views per day.
- Results / Suggestions are currently based on Total Views only. We provide a bunch of other statistics in the results but felt it was simplest to base the final recommendation on total views. The reason being - people care about different things. Some people want lots of comments and engagement. Some are more concerned with getting lots of traffic from YouTube Search. Someone else might only be concerned with Avg Watch Time. There really wasn't a "one size fits all" score to give the results and allow people to completely customize weights/priorities seemed like overkill. So, we base our suggestions on Views but encourage you to look at the broad picture of results, identify what's most important to YOU and go in that direction.