A/B TESTING INTERVIEW QUESTIONS


Most Important Frequently Asked A/b Testing Interview Questions



    1. Question 1. When Is A/b Testing A Good Idea? When Is It A Bad Idea?

      Answer :

      A/B testing most commonly fails because the test itself has unclear goals, so you've got to know what you're testing. Use A/B testing to test a theory, for example -- would adding a picture to this landing page increase conversions? Are people more likely to click a red button or a blue button? What if I change the headline to stress the time-limit of the offer? These are all changes that can be easily quantified. People run into trouble with A/B testing when their theories are too vague, like testing two entirely different designs with multiple variants. While it can be done, unless there is a clear landslide winner, testing different designs can lead to softer conclusions and an uncertainty about what actually caused the increase in conversions.

    2. Question 2. How Many Variations Should I Have In A/b Testing?

      Answer :

      Let's say you’ve brainstormed as a marketing team and you have four great ideas for a landing page design. It can be tempting to run all four treatments at once to declare a winner, but similar to the variations issue above, it’s not a true A/B test if you have multiple different treatments running at once. A number of factors from each different design can get in there and muddy the test result waters, so to speak. The beauty of an A/B test is that its results are straightforward and concrete. We suggest running two versions against each other, and then running a second test afterwards to compare the winners. Think of it as a really techy basketball bracket.

    3. Question 3. What Is A Null Hypothesis?

      Answer :

      A null hypothesis is the hypothesis that any difference in outcomes is the result of a sampling error or standard variation. Think about flipping a coin. While you have 50/50 odds for the coin to land on heads, sometimes the outcome in practice is 51/49 or some other variation due to chance. The more you flip the coin, though, the closer you should get to a 50/50 result. In statistics, the way you prove or disprove an idea is to dispute the null hypothesis. Disputing a null hypothesis is a matter of running the experiment long enough to rule out an incidental outcome. This concept is also referred to as reaching statistical significance.

    4. Question 4. How Many Visits To A Page Do I Need To Get Good Results With A/b Testing?

      Answer :

      Before you can test the results of an A/B test, you have to be sure the test has reached statistical significance -- the point at which you can have 95% confidence or more in the results.  

      The good news is, many A/B testing tools have statistical significance built right in so you get an indication as to when your test is ready for interpretation. If you don't have that, however, there are also a number of free calculators and tools out there for understanding the statistical significance.

    5. Question 5. What's Multivariate Testing, And How Does It Compare To A/b Testing?

      Answer :

      • A/B testing is typically used for redesigns to test out the effectiveness of a single design direction or theory against a goal (like driving conversions). Multivariate testing tends to be used for smaller changes over a longer period of time. It will take a number of elements of your site and test out all possible combinations of these elements together for ongoing optimization. In a post in January, my colleague Corey Eridon explained the differences between when you'd use one test over the other in detail, saying.
      • A/B testing is a great testing method if you need meaningful results fast. Because the changes from page to page are so stark, it will be easier to tell which page is most effective. It is also the right method to choose if you don't have a ton of traffic to your site. Because of the multiple variables being tested in a multivariate test, you'll need a highly trafficked site to get meaningful results with MVT.
      • If you do have enough site traffic to pull off a successful multivariate test (though you can still use A/B testing if you're testing brand new designs and layouts!) a great time to use the testing method is when you want to make subtle changes to a page and understand how certain elements interact with one another to incrementally improve on an existing design.

    6. Question 6. Does A/b Testing Negatively Affect Seo?

      Answer :

      There's a myth that A/B testing hurts search engine rankings because it could be classified as duplicate content, which search engines don’t look kindly upon. This myth is most definitely false. In fact, Google’s Matt Cutt advises running A/B tests to improve the functionality of your site. Website Optimizer has a good breakdown of the myth too, and why it doesn’t hold up. If you're still concerned, you can always add a "no index" tag to your variation page.  Detailed instructions on adding a “no index” tag can be found here.

    7. Question 7. How And When Do I Interpret My Split Test Results?

      Answer :

      The test starts. The results begin to roll in. You scramble to check who's winning. But the early stages of a test are not the right time to start interpreting your results. Wait until your test has reached statistical significance  and then revisit your original hypothesis. Did the test definitively prove or disprove your hypothesis? If so, you can start to draw some conclusions. When you interpret your test, try to stay disciplined about attributing your results to the specific changes made. Make sure there are clear connections between the change and the outcome, and there aren't any other forces at play.

    8. Question 8. How Many Variables Should I Test?

      Answer :

      You want your A/B test to be conclusive -- you’re investing time in it, so you want a clear and actionable answer! The problem with testing multiple variables at once is you aren’t able to accurately determine which of the variables made the difference. So while you can say one page performed better than the other, if there are three or four variables on each, you can’t be certain as to why or if one of those variables is actually a detriment to the page, nor can you replicate the good elements on other pages.

    9. Question 9. What Should I Test?

      Answer :

      What you test is up to you, but we recommend starting with a few basic lynchpins of your webpage. 

      Calls-to-Action: Even with the single element of a call-to-action, there are a number of different things you can test. Just make sure you're clear on what aspect of the CTA you're testing. You could test the text -- what the CTA compels the viewer to do; the location -- where the CTA is positioned on the page; the shape and style -- what the CTA looks like. In the example below, for instance, HubSpot tested the shape and style of our demo CTA to see which performed better. The CTA shaped like a button (on the right) rather than the CTA that included a sprocket image (left) performed signficantly better, giving us a 13% increase in conversions.

      Headline: It's typically the first thing a viewer reads on your site, so the potential for impact is significant. Try out different styles of headlines on your A/B tests. Make sure that the difference between each headline's positioning is clear rather than some simple wordsmithing so you can be certain as to what caused the change.  

      Images: What's more effective, an image of a person using your product, or the product on its own? Test different versions of your pages with alternate supporting images to see if there's a difference in action.

      Copy length: Does shortening the text on your page result in a clearer message, or do you need the extra text to explain your offer? Trying out different versions of your body text can help you determine what amount of explanation a reader needs before converting. To make this test work, try to keep the text similar and just test the volume of it.

    10. Question 10. Can You Run A/b Tests On Things Other Than Web Pages?

      Answer :

      Yes! In addition to landing pages and webpages, many marketers run A/B tests on emails, PPC campaigns, and calls-to-action.  

      Email: Email testing variables include the subject line, personalization features, and sender name, among others.

      PPC: For paid search ad campaigns, you can A/B test the headline, body text, link text, or keywords.

      CTAs: With CTAs, try altering the text on the CTA, its shape, color, or placement on the page.

    11. Question 11. How Do I Find A/b Tests From Similar Companies?

      Answer :

      There are a number of sites out there that aggregate A/B testing examples and results. Some allow you to search by company type, and most provide details as to how the company interpreted the test results. When you're first getting started, it's not a bad idea to read through some of these sites to get ideas on what to test for your own company.

      Which Test Won: Anne Holland's website, Which Test Won, has a series of examples as well as some annual contests that you can submit your own tests to. 

      Visual Website Optimizer: Like HubSpot, Visual Website Optimizer provides A/B testing software and has a number of examples on their blogs that you can learn from.

      ABTests.com: This site is no longer being updated, but it does have a good archive of tests and some quick takeaways, as well as links to the original test conductor and write-ups. The site was founded by HubSpot UX director Josh Porter who also has some great advice on A/B Testing over on his personal blog, bokardo.com.

    12. Question 12. What Do I Do If I Don’t Trust The Results?

      Answer :

      If you really don't trust the results and have ruled out any errors or challenges to the test's validity, the best thing to do is to run the same test again. Treat it as an entirely separate test and see if you can replicate the results. If you're able to replicate again and again, you probably have a solid set of results.

    13. Question 13. How Often Should I Run A/b Testing?

      Answer :

      Perspectives vary on this one. There's a good case to be made for always testing and iterating on your site. Just be sure that each test has a clear purpose and will result in a more functional site for your visitors and company. If you're running a lot of tests that are resulting in minimal outcomes or minor victories, reconsider your testing strategy.

    14. Question 14. What Do I Need To Start A/b Testing On My Site?

      Answer :

      The best way to run A/B tests is to use a software tool designed for it. HubSpot offers one in its all-in-one marketing software platform. Other providers include Unbounce and Visual Website Optimizer. If you don't mind messing with a little code, Google also has a free tool called Content Experiments in Google Analytics. It's a little different than traditional A/B testing, but if you're tech savvy, you could try it out.

    15. Question 15. Beyond Sample Size, What Other Validity Traps Are There?

      Answer :

      MECLABS ran a great web clinic last year on a collection of other threats to a test's validity beyond sample size. In it, Dr. Flint McGlaughlin ran through three testing errors and how to mitigate the risk of coming across them in your tests. I'd recommend reading the full transcript of the clinic, but here are a couple of errors from the list. 

      History Effect: Something happens in the outside world to adversely bias your test results.

      Instrumental Effect: An error in your testing software undermines the testing results.

    16. Question 16. Should I A/b Test My Homepage?

      Answer :

      While I wouldn't completely rule out A/B testing your homepage, I will say that it can be difficult to run a conclusive test of your homepage. Your homepage gets a variable mix of traffic, from accidental visitors, to leads, to customers. There's also a typically a ton of content on your homepage, so it can be challenging in one test to determine what's driving a visitor to act or not act. Finally, because of the variety of people who come to your homepage, figuring out a goal for the page and test can be a challenge. You may think that your goal is to test lead-to-customer conversions, but if the sample visiting during the test is heavy on customers instead of prospects, your goal for that group could shift. If you want to test your homepage, think about just testing your CTAs. At the enterprise level, HubSpot's CTA manager allows you to run a split test just on a single CTA on your site rather than the whole page.

    17. Question 17. What If I Don’t Have A Control?

      Answer :

      A control is the existing version of a landing page or webpage that you are testing against.  Sometimes you may want to test two versions of a page that never existed before ... and that’s okay. Just choose one of the variations and call that one the control. Try to pick the one that's the most similar to how you currently design pages, and use the other as the treatment.

    18. Question 18. Why Isn’t My Split Exactly 50/50?

      Answer :

      Sometimes during A/B tests you may notice that the traffic numbers of each variation are not identical. This doesn’t mean that anything is wrong with the test, just that random variations work, well, randomly. Think about flipping a coin. You have a 50-50 shot of heads or tails, but sometimes you get three tails in a row. As more traffic hits your site, however, the numbers should become closer to 50-50.

    19. Question 19. What Do You Understand By A/b Testing? What Is The Goal Behind Performing A/b Testing?

      Answer :

      A/B Testing (also known as Split testing) defines a way to compare two versions of an application or a web page that enables you to determine, which one performs better. A/B Testing is one of the easiest ways, where you can modify an application or a web page to create a new version and then comparing both these versions to find the conversion rate. This also lets us know, which is the better performer of the two.

    20. Question 20. What Do You Understand By Data Sampling In A/b Testing?

      Answer :

      The number of samples depend on the number of tests performed. The count of conversion rate is called a sample and process of collecting these samples is called sampling.

    21. Question 21. What Is The Use Of Confidence Interval In A/b Testing?

      Answer :

      Confidence interval is called measurement of deviation from the average on the multiple number of samples. Let us assume that 22% of people prefer product A, with +/- 2% of confidence interval. This interval indicates the upper and lower limit of the people, who opt for Product A and is also called margin of error. For best results in this average survey, the margin of error should be as small as possible.

    22. Question 22. When Should You Think Of Performing A/b Testing?

      Answer :

      Always perform A/B Testing if there is probability to beat the original variation by >5%. Test should be run for considerable amount of time, so that you should have enough sample data to perform statistics and analysis. A/B Testing also enables you to gain maximum from your existing traffic on a webpage.

      The cost of increasing your conversions is minimal as compared to the cost of setting up the traffic on your website. The ROI (return on investment) on A/B Testing is huge, as a few minor changes on a website can result in a significant increase of the conversion rate.

    23. Question 23. Name Few Of The Variations That Can Be Applied To A Web Page?

      Answer :

      Here are a few A/B Testing variations that can be applied on a web page. The list includes − Headlines, Sub headlines, Images, Texts, CTA text and button, Links, Badges, Media Mentions, Social mention, Sales promotions and offers, Price structure, Delivery options, Payment options, Site navigations and user interface.

    24. Question 24. Explain The Step By Step Process For A/b Testing?

      Answer :

      Background Research − First step in A/B Testing is to find out the bounce rate on your website. This can be done with the help of any tool like Google Analytics.

      Collect Data − Data from Google Analytics can help you to find visitor behaviors. It is always advisable to collect enough data from the site. Try to find the pages with low conversion rate or high drop-off rates that can be improved.

      Set Business Goals − Next step is to set your conversion goals. Find the metrics that determines whether or not the variation is more successful than the original version.

      Construct Hypothesis − Once the goal and metrics have been set for A/B Testing, next is to find ideas to improve the original version and how they will be better than the current version. Once you have a list of ideas, prioritize them in terms of expected impact and difficulty of implementation.

      Create Variations/Hypothesis − There are many A/B Testing tools in the market that has a visual editor to make these changes effectively. The key decision to perform A/B Testing successfully is by selecting the correct tool.

      Running the Variations − Present all variations of your website or an app to the visitors and their actions are monitored for each variation. Visitor interaction for each variation is measured and compared to determine how that variation performs.

      Analyze Data − Once an experiment is completed, next is to analyze the results. A/B Testing tool will present the data from the experiment and will tell you the difference between how the different variations of web page is performed. Also if there is any significant difference between variations with the help of mathematical methods and statistics.

    25. Question 25. What Are The Common Type Of Tools That Can Be Used To Collect Data For A/b Testing?

      Answer :

      The most common type of data collection tools includes the Analytics tool, Replay tools, Survey tools, Chat and Email tools.

    26. Question 26. What Is The Purpose Of Using Replay Tools? Name A Few Replay Tools For A/b Testing?

      Answer :

      Replay tools are used to get better insight of user actions on your website. It also allows you to click maps and heat maps of user click and to check how far user is browsing on the website. Replay tools like Mouse Flow allows you to view a visitor's session in a way you are with the visitor.

      Video replay tools give deeper insight into what it would be like for that visitor browsing the various pages on your website. The most commonly used tools are Mouse Flow and Crazy egg.

    27. Question 27. Why Do We Use Survey Tools? Give An Example Of A Survey To Collect Data?

      Answer :

      Survey tools are used to collect qualitative feedback from the website. This involves asking returning visitors some survey questions. The survey asks them general questions and also allows them to enter their views or select from pre-provided choices.

    28. Question 28. Give A Few Examples Of Generating Hypotheses In A/b Testing?

      Answer :

      You can reduce the number of bounce rate by adding more images at the bottom. You can add links of social sites to further increase the conversion rate.

    29. Question 29. What Do You Understand By Creating Variations In A/b Testing? What Are The Common Tools That Can Be Used For This?

      Answer :

      There are different types of variations that can be applied to an object like using bullets, changing numbering of the key elements, changing the font and color, etc. There are many A/B Testing tools in the market that has a visual editor to make these changes effectively. The key decision to perform A/B testing successfully is by selecting the correct tool.

      Most commonly available tools are Visual Website Optimizer, Google Content Experiments and Optimizely.

    30. Question 30. What Is Wysiwyg Editor In Vwo? How Do You Create Variations Using That?

      Answer :

      Visual Website Optimizer or VWO enables you to test multiple versions of the same page. It also contains ‘what you see is what you get’ (WYSIWYG) editor that enables you to make the changes and run tests without changing the HTML code of the page. You can update headlines, numbering of elements and run a test without making changes to IT resources.

      To create variations in VWO for A/B Testing, open your webpage in WYSIWYG editor and you can apply many changes to any web page. These include Change Text, Change URL, Edit /Edit HTML, Rearrange and Move.

    31. Question 31. What Are The Other Type Of Testing That Can Be Performed Using Vwo?

      Answer :

      Visual Website Optimizer also provides an option of multivariate testing and contains other number of tools to perform behavioral targeting, heat maps, usability testing, etc.

    32. Question 32. Where Else Can You Use A/b Tests On Things Other Than Web Pages?

      Answer :

      These tests can be applicable on several other places like Email, Mobile Apps, PPC and CTAs as well.

    33. Question 33. What Do You Understand By A/b Testing - Analyze Results?

      Answer :

      Once an experiment is completed, next is to analyze the results. A/B Testing tool will present the data from the experiment and will tell you the difference between how the different variations of that web page are performed. It will also show if there is a significant difference between variations using mathematical methods and statistics.

    34. Question 34. How Do You Integrate Google Analytics With Optimizely?

      Answer :

      To integrate Optimizely to Universal Google Analytics, first select the ON button on the side panel. Then you must have an available Custom to populate with Optimizely experiment data.

    35. Question 35. Integrate Google Analytics With Optimizely?

      Answer :

      The Universal Google Analytics tracking code must be placed at the bottom of the <head> section of your pages. Google Analytics integration will not function properly unless the Optimizely snippet is above the Analytics snippet.

    36. Question 36. What Is The Difference Between Universal Ga And Classic Google Analytics?

      Answer :

      Google Analytics has two options for analyzing the data, which are Universal Analytics and Classic Google Analytics. New Universal Analytics features allow you to use 20 concurrent A/B tests sending data to Google Analytics, however the Classic version allows only up to five.

    37. Question 37. Does A/b Testing Negatively Affect Seo? Explain Different Ways That Ensures To Run A/b Tests Without Worrying About Losing The Potential Seo Value?

      Answer :

      This is a myth that A/B Testing hurts search engine rankings because it could be classified as duplicate content. The following four ways can be applied to ensure that you don’t lose the potential SEO value, while running A/B Tests.

      Don’t Cloak − Cloaking is when you show one version of your webpage to Googlebot agent and other version to your website visitors.

      Use ‘rel=canonical’ − When you have A/B Tests with multiple URL’s, you can add ‘rel=canonical’ to the webpage to indicate to Google which URL you want to index. Google suggests to use canonical element and not noindex tag as it is more in line with its intention.

      Use 302 redirects and not 301’s − Google recommends to use the temporary direction method − a 302 over the permanent 301 redirect.

      Don’t run experiments for a longer period of time − Please note that when your A/B Test is completed, you should remove the variations as soon as possible and make changes to your webpage and start using the winning conversion.



Topic: A/b Testing Interview Questions
Posted by

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)