AB Testing: Everything You Need to Know

ab test

Here’s a scenario that many ecommerce businesses are familiar with.

You’ve spent months researching various niches and found a fantastic product using Oberly Supply. It’s in high demand with relatively little competition. You’ve tested suppliers and found one that offers a quality product at a reasonable price. You’ve created a beautiful ecommerce storefront using Shopify and have automated the dropshipping process with Oberlo. You’ve attracted customers using a mix of social media marketing, email campaigns, and Google ads. You’re getting hundreds of new visitors to your site every day.

But you’re not making sales… and you don’t know why.

Maybe there’s a problem with the location of the “Buy Now” button that’s making it hard for visitors to purchase your product. Maybe they are frustrated by your long and confusing checkout process. Or maybe they just don’t like the color or theme of your site.

It would be great if we could set up a poll to ask visitors why they’re unhappy with the site. We could ask them for suggestions for creating a better user experience. But for most sites, this simply isn’t possible. Fortunately, ecommerce businesses can conduct a simple yet powerful experiment to find out what works and what doesn’t.

That experiment is called AB testing.

With A/B testing, you can compare two different versions of your site to see how your visitors react to specific changes. By testing a variety of changes and responses, you can find the best way to increase conversion rates, increase sales and maximize profits. You can also avoid changes that might harm your business.

In this article, we’ll explore A/B testing in detail and look at some of the most common parts of your site that can be tested. We’ll also examine how A/B testing has helped other small businesses improve their conversion rate and increase their revenue. Finally, we’ll show you how to setup your first A/B test.

What Is AB Testing?

A/B testing, also called split testing, is a method for testing changes to a website. We can see which changes have a positive effect, a negative effect, or no effect on the audience.

The simplest form of A/B testing involves creating two webpages.

The first page is the “A version” (the control), which is usually the page that you are currently showing your visitors. The second page is the “B version” (the variation), which is the same as the control except for one element that you have changed. That element could be almost anything. It could be the title of your page or the timing of a pop-up email capture. It could be the position of an image or even the font you are using.

When people visit your site, you randomly send half your visitors to the “A version” and half your visitors to the “B version”. You then record user data for each version. After enough people have visited the two versions of your page, you can compare the data you collected from the two pages. The results will show which version is more effective at achieving your goal.

a/b testing

Source: VWO

Here’s an example of an A/B test.

Mary runs an online store that sells custom-made leather dog collars. The first thing her site visitors see is a landing page with a hero image of a dog wearing one of her collars.

She gets an average of 1,000 new visitors to her site every day. However, after analyzing her visitor data, she discovers that 20% of her visitors don’t click on the “Shop Now” link to view her products. She thinks this number is too high, so she decides to try modifying her site. Her goal is to reduce the bounce rate (the number of visitors who leave the website without viewing any other pages).

She shows her website to some friends who are dog owners, and they mention that the text over the image is a little confusing. It doesn’t actually specify what she is selling.

She decides to run an A/B test to see whether changing the text so that it includes the word “collars” will improve the bounce rate.

For two weeks, she sends half her visitors to her current page and half her visitors to a variation with different text. After analyzing the visitor data, she discovers that the bounce rate for the control page stayed the same (around 20%). However, the bounce rate for the variant page dropped to just 5%.

Because of these results, she decides to make the variant page her new permanent landing page.

Why Is A/B Testing Important?

Whether you’re struggling to make your first sale, or you’re already selling thousands of products a day, there’s always room for improvement.

For ecommerce businesses, the ultimate goal is to sell as many products as possible to maximize profits. To do this, you need a website that appeals to your visitors and converts as many browsers into buyers as possible.

However, it’s impossible to make a website that appeals to everyone at once. With A/B testing, you can take the guesswork out of designing your website.

Here’s a simple example. Your ecommerce site is currently making $5,000 a month at a 1% conversion rate. You think that moving your call to action might increase your conversion rate and improve profits. However, you’re worried that if you’re wrong, moving it might result in a decrease to your current sales.

Fortunately, you don’t need to guess. A simple A/B test shows that moving your CTA from the bottom of the page to the middle of the page results in a conversion rate increase to 2%. That’s an extra $5,000 a month, just from making a minor change to your site.

A/B testing can also show you what not to do.

For example, before spending thousands of dollars on professional product photos for your site, you could try an A/B test. You could compare a product page with professional photos and a variant page with photos you took yourself. You might find that there is little or no difference to the conversion rates, and decide that your money would be better spent elsewhere.

How Does A/B Testing Work?

Here are some key guidelines:

  • The best way to guarantee accurate results when performing an A/B test is to test only one variable at a time.  By limiting your test to just one variable, you can be confident that the variable you have changed is directly responsible for any changes in visitor behavior.
  • It’s also possible to conduct a test with multiple variables using a multivariate test. In a multivariate test, the control stays the same but the variation has several changed elements. For example, you may try changing your headline, text color and font size at the same time. Multivariate testing allows you to see how variables work together. However, this makes it harder to be sure which variable has affected visitor behavior.
  • It’s important to collect data from as many visitors as possible. A small sample size affects the reliability of the test, as there is a greater possibility that results have been influenced by chance.
  • Try to run an A/B test for an extended period of time. Visitors browsing habits can be affected by various factors such as the time of day and the day of the week. This makes short-term tests less accurate. Aim for at least two weeks.
  • Don’t stop at just one A/B test. If you find that your variant performs better than your control, why not make it your new control and run some additional tests against some other variants? You might find another variant that is even better.
  • A/B testing can be used for other aspects of your online business, not just your website. For example, you can use A/B testing to refine your email marketing plan. Marketers can use A/B testing to see which versions of their messages create the best results. You can try to improve things like email opens, clicks back to the website, and registrations or purchases via the site. You can test variables such as subject lines and headlines, visual layout of the email, personalization (like “Mrs. Doe” vs. “Jane”), and special offers.

What Variables Can Be Tested?

As we mentioned before, you can test almost any element of your website. Here are some ideas to get you started.

Headlines and sub-headlines

Most people use headlines and sub-headlines to quickly browse through websites. It’s important to have the right style and text in your headlines so your visitors don’t skip past important information. You can test which headlines and sub-headlines attract the most attention.

ab testing

Call-to-action text and buttons

Factors such as the size, shape, placement and text of your call-to-action buttons can affect whether or not a visitor will click on the link.

ab test

Links and images

Good images can make your product look amazing. But the wrong image can turn away potential customers.

a/b test

Testimonials and case studies

Displaying testimonials and case studies can be one of the most powerful techniques for an e-commerce business. You can show potential customers what benefits they can expect when buying your product. Try testing a variety of testimonials to see which ones resonate most with your audience.

split testing

Pricing structures

You’ve probably heard that a product marked $9.99 sells more than the same product marked $10.00. But what about a product marked $9.97 or $9.95? After testing different prices, you might find that a slightly cheaper price actually results in more sales.

ab testing

Sales promotions

A good promotion can attract customers who wouldn’t usually consider buying your product. Different promotions work better for certain products, so it’s important to test a variety of ideas before settling on one.  

ab testing

Free trial length

A free trial can be a great promotion for attracting customers. However, you don’t want an unnecessarily long trial as you’ll be missing out on potential profit. You can use an A/B test to find the perfect trial length for your product.

ab testing

Free vs. paid delivery

Free delivery is a popular option for many customers. However, you’ll need to incorporate the shipping fees into the product price. Do your customers prefer a more expensive product with free shipping, or a cheaper product with paid shipping? It’s a good idea to test both options with your audience.

ab test

Ecommerce Store AB Testing Case Studies

While the examples above have probably given you some good ideas for things to test, the best way to see how A/B testing can help improve your e-commerce store is to examine some actual case studies. These three ecommerce businesses were able to increase their sales by A/B testing various elements of their online stores.

Free Shipping Increases Orders By 90%

Anti-aging skin care company NuFACE was concerned about conversions. Their customers appeared to be well informed and interested about the products on their site. However, they seemed to be reluctant to make a purchase. The company realized that they needed to offer their customers some kind of incentive. They decided to conduct an A/B test to see if adding a free shipping threshold would improve their sales.

Half their visitors were sent to their current page, which was used as a control:

ab test


The other half of their visitors were sent to a variant page that included the incentive “Free shipping over $75?” placed right above the Shop NuFace button:

ab test


The A/B test showed that when customers were sent to the page with the free shipping incentive, orders increased by 90% with a 96% confidence level. In addition to this, the company’s average order value (AOV) also rose by 7.32%.

A New Pop-Up Design Increases Conversions 400%

The GLD Shop, an ecommerce store that sells customizable jewelry, was seeing good results with their current welcome pop-up. Although an engagement rate of 20.6% and a conversion rate of 11% would make most ecommerce businesses happy, they felt that there was even more potential for making sales. Their philosophy was “why be happy with ‘good’ numbers when you can have ‘great’ numbers?” However, they didn’t want to mess with these good numbers by immediately changing their welcome pop-up. They decided to conduct an A/B test that compared their current pop-up design with a new design.

Half the visitors were shown the original control pop-up:

ab testing


The other half were shown a variant pop-up with a larger, clearer image, contrasting colors and no generic offer text:

ab testing


Despite having exactly the same offer as the control, the new design resulted in an incredible 400% increase in conversions.

Moving Product Location Increases Sales By 106%

Muc-Off is an ecommerce store that sells cleaning products for electronic goods, apparel, motor vehicles and fitness goods. They were concerned that the design of their product page was affecting the visitor bounce rate.

The original page design featured information about Muc-Off at the user’s eye level. They felt that this was causing confusion in the user experience. Visitors came there to purchase products, but they were finding information resources instead of a shop front.

Muc-Off decided to run an A/B test which would analyze tweaks to the design of the category page. Thee air goal was to increase product views and maximize sales.

Half the visitors to the site were shown the original control page:

ab test


The other half of visitors were shown a variant that had replaced the ‘above the fold’ content with eye-catching images of their products, complete with a slider:

split testing


Using heat maps and various analytic tools, the team discovered that moving images of products above the fold increased product views by a total of 43.78%. It also resulted in an increase in sales by 106.26%.

How To Do A/B Testing: Setting Up Your First Test

Now that you’ve seen how A/B testing can benefit your business, it’s time to try an A/B test on your own site.

Before you start testing, it’s important to nail down specific objectives and goals. You also need to choose which key performance indicators (KPIs) and metrics you’ll use to measure them. This way, you’ll have a clear view of what exactly you’re testing for. You’ll be able to turn your test results into meaningful conclusions about how to improve the success of your commerce site.

Here are five steps you’ll need to follow to conduct a successful A/B test.

Step 1 – Determine Your Goal And Which Metrics To Examine

The first thing you need to decide is the goal of your A/B test. Do you want to reduce website bounces, increase website page views? Do you want to increase the time visitors spend on the site, or simply increase your sales?

Once you have a goal in mind, you can choose a metric to examine. It’s a good idea to look at your existing analytics data to get some ideas on where to begin. Try testing high-traffic pages first to gather data quickly, then determine which metrics you’ll measure to discover if your variation is more successful. Some metrics that you can examine are:

  • Bounce rate
  • Exit rate (similar to bounce rate –  the number of visitors who interact with your site but suddenly leave at a certain point)
  • Conversions
  • Registrations
  • Number of product sales

Step 2 – Form A Hypothesis

In order to achieve your goal, you need to work out what elements of your website need to be changed. For example, if your goal is to reduce the number of visitors who abandon the sale during the checkout process, you should examine each part of the checkout process in detail.

You could examine factors such as:

  • How many steps it takes from selecting a product to completing the sale
  • The design of the submission form
  • The amount of personal information required on the submission form

Make a list of all the possible variations that might perform better than your existing version. Prioritize your list based on how successful you think they’ll be, as well as how difficult it might be to implement the test.

Step 3 – Create A Variation

Once you’ve decided which elements you are going to test, you’ll need to create a variation to compete against the control. Your variation should be an exact copy of your control page except for the element you are testing.

Step 4 – Run The Experiment

To conduct an A/B test, you’ll need to use A/B testing software. There are many different products you can choose from, but for this example we’ll use the free Google Analytics Experiments. To use this A/B testing software, you’ll need to have Google Analytics running on your site.

To setup an experiment, log in to your Google Analytics account. Click on “Behavior” and “Experiments” in the sidebar. Then click the “Create Experiment” button.

create an ab test

In the Content Experiments box, add a name for the experiment that is easy to identify.

Next, choose one or more objectives from the dropdown box. You can choose metrics such as bounces, page views and session duration. You can also create a new objective.

Choose the percentage of traffic to send to your variation. If you’re looking for quick results, you can choose a large percentage of visitors. If you’re running a risky experiment, you should choose a small percentage of visitors. If you’re not sure, the safest thing to do is choose 50%.

ab testing tools

Once you’ve entered all the details, click “Next Step”.

On the next page, you’ll need to enter the URL of your control page and the URL of the variant page you want to test. Once you’ve correctly entered the URLs, you should see a preview in the Page Preview box.

ab test

If everything looks ok, click “Next Step”.

The next step is to add the Google Experiment code to your original page. Click “Manually insert the code” and copy all the script code in the box. You’ll need to paste all this code into the top of your original page, immediately after the opening <head> tag. The method for editing page code depends on whether you’re using a CMS or a static HTML site. If you’re uncomfortable editing page code, it’s a good idea to check with a professional first.

ab test

Once you’ve entered the code, click “Next Step”.

The next screen will confirm whether or not the code has been correctly added to the page. If the code is not validated, but you’re sure it has been entered correctly, you can still start the experiment.

what is ab testing

Once you’ve started the experiment, all you have to do is sit back and wait for the results to come in.

Step 5 – Analyze The Results

After the experiment has finished, Google Analytics will show you which page performed better on your previously defined metrics. You can analyze the results and decide whether to keep the control, change to the variant, or run some more experiments.

AB Testing Tools

There are dozens of A/B testing software options that can help you achieve your ecommerce goals. Here are some great choices:

  • VWO is an all-in-one testing and conversion optimization platform for visitor research. You can build optimization roadmaps and run different types of continuous tests.
  • Optimizely is an experimentation platform that offers testing for everything from designs to algorithms. It also offers personalization and recommendation features.
  • Google Analytics Experiments allows you to test almost any website or app variation to see how it performs while you’re optimizing for a specific goal. You can run client-side (within a web browser) or server-side experiments.
  • Unbounce is a landing page builder that offers a drag-and-drop interface for building without any prior coding knowledge experience. Once your page is up, you can test multiple variations for as long as you like.
  • UsabilityHub offers “five second tests” to optimize websites by measuring a visitor’s first impressions. Visitors are shown designs for five seconds and asked what they remembered. You’re then sent a report to see the most memorable features of your material.
  • Convert specializes in flexible and easy-to-use conversion optimization testing. This includes A/B tests, multivariate tests, split-URL testing, and multipage testing.

Time to Get Testing

There’s no doubt that your brand can benefit from A/B testing in some form. A/B testing can help you boost overall conversions and sales. It can help you identify what is wrong with your current e-commerce site, and help you find a plan for moving forward. The most important thing is to keep experimenting. You never know, you may find that the best results will come from an idea you wouldn’t have tried otherwise.

Start using Oberlo today

All accounts have access to the forever free Starter Plan

Sign Up Free

Oberlo uses cookies to provide necessary site functionality and improve your experience. By using our website, you agree to our privacy policy.