July 24 - A Beginner’s Guide to A/B Testing on Canada.ca

Back to GC Web Priorities Meetings - Réunions des Priorités Web du GC

Meeting information

Agenda

1. Updates

2. Roundtable

Notes and action item

Updates

DTO

• No updates this week

PCO

• Campaigns - We’ve reached out to encourage people to start early working on their campaigns
• Check out the Slack thread about campaigns, tips such as avoiding carousels
• Big thanks to Principle Publisher for new biography for Minister Mckinnon
• Today is International Self Care Day

Principle Publisher

• Removed mention of Minister Seamus O’Regan and replaced with new Minister Steven MacKinnon
• We have removed the Share this page widget from AEM, if you’re not on AEM, please remove by end of August

PCO - Impact Canada - Harry MacKay

A Beginner’s Guide to A/B Testing on Canada.ca

• We were established in 2017
• Partners such as:
  • ECCC
  • NRCan
  • PHAC, Health Canada etc…
• What is A/B testing?
  • A lot of website design is guided by top down principles of usability, composition etc..
  • AB testing is a way to empirically test two or more versions of a product
  • Variants are randomly assigned to subjects as they enter the page
• Considerations for good test:
  • DTO has provided valuable information on A/B testing
  • Canada.ca is mandated to provide clear information to the public
  • Ideally, A/B testing focus on one single element at a time
• Examples of A/B testing on Canada.ca
  • Example of CRA test, in this example the element you were supposed to click was more prominent, with a 20% increase in desire outcome
  • OCAO ran a similar test, making button more “obviously clickable” with favorable results

• Considering statistical power
  • You have to make sure you have enough page traffic for meaningful results
  • The sample size is determined by incoming page views
  • The only thing you really have control over to increase amount of people, is to run the test for a longer time
  • This means knowing the average amount of visitors per day for your page is critical
    • Checklist of statistical power
      • Average number of daily visitors - Canada.ca analytics
      • Baseline rate of target behavior
      • Expected effect size of manipulation during testing
  • Adobe provides a tool that can estimate how many visitors you will need to collect with a given lift rate and baseline conversion rate
• Adobe Target is the software we use for A/B testing, Principle Publisher manages access to this software
  • Adobe Target GCpedia page: https://www.gcpedia.gc.ca/wiki/Adobe_Target_User_Guide

Privacy and ethical considerations

• Personal information is not collected
• Use your own discretion using Adobe Target in environments with Protected information
• Departments are mandated to provide clear information
  • This may limit ability to change wording/text on you page
• You may want to submit your plans to ethics review if you have it at your department
• Stop at a predetermined time to avoid biases
• The Sequential Analysis approach has predefined stopping rules, such as in a drug trial, to protect against the ethical ob subjecting users to underperforming version of critical web page

Key Takeways

• A/B testing is a powerful tool to improve usability
• Statistical power is the biggest concern when planning your project
  • Average lift (improvement) is around 2%, Not every page on Canada.ca receives enough traffic to run a test
• The testing is relatively low cost

QUESTIONS

Q: How do you consider this vs. other kinds of testing?

A: It’s a good method once your website has been set up. It’s really for optimizing. Doesn’t address accessibility very well. It has its place in combination with other kinds of testing which for example might be better for specific users or other considerations.

Comment: A/B testing is good for settling points, for example disagreements between clients or the Minister's Office.

Comment, we recently did a presentation on Adobe Target, sharing here: https://gcxgce.sharepoint.com/:p:/t/1000163/ESGBlOV_32RNuYTcPJKTU8EBO9J9Quk0Ef9jMHej1NYSAg?e=4nGzEc

Comment:

Doing usability testing on a prototype can also help overcome some of the potential ethical considerations that Harry mentioned.

For example, if you want to test the impact of removing an entire paragraph of content and don't want to risk some users having information that other users do not, you can use usability testing instead of an A/B test.

Roundtable

ISED

• Reminder; we are still looking for departments to share any AI content pages you have so we can build the subtopic - aiming for September to launch

Contact: christopher.colyer@ised-isde.gc.ca