Difference between revisions of "July 24 - A Beginner’s Guide to A/B Testing on Canada.ca"
Rob.boudreau (talk | contribs) |
Rob.boudreau (talk | contribs) |
||
(4 intermediate revisions by the same user not shown) | |||
Line 25: | Line 25: | ||
'''PCO - [https://impact.canada.ca/en Impact Canada] - Harry MacKay''' | '''PCO - [https://impact.canada.ca/en Impact Canada] - Harry MacKay''' | ||
− | '''A Beginner’s Guide to A/B Testing on Canada.ca''' | + | '''[[:en:images/2/24/AB_testing_IIU_EN_2024-04-26_for_posting.pptx|A Beginner’s Guide to A/B Testing on Canada.ca]]''' |
* We were established in 2017 | * We were established in 2017 | ||
Line 54: | Line 54: | ||
**** Expected effect size of manipulation during testing | **** Expected effect size of manipulation during testing | ||
** Adobe provides a tool that can estimate how many visitors you will need to collect with a given lift rate and baseline conversion rate | ** Adobe provides a tool that can estimate how many visitors you will need to collect with a given lift rate and baseline conversion rate | ||
− | * [ | + | * [https://business.adobe.com/ca/products/target/adobe-target.html Adobe Target] is the software we use for A/B testing, Principle Publisher manages access to this software |
− | ** Adobe Target GCpedia page: | + | ** Adobe Target GCpedia page: https://www.gcpedia.gc.ca/wiki/Adobe_Target_User_Guide |
Privacy and ethical considerations | Privacy and ethical considerations | ||
Line 82: | Line 82: | ||
Comment: A/B testing is good for settling points, for example disagreements between clients or the Minister's Office. | Comment: A/B testing is good for settling points, for example disagreements between clients or the Minister's Office. | ||
− | Comment, we recently did a presentation on Adobe Target, sharing here: | + | Comment, we recently did a presentation on Adobe Target, sharing here: https://gcxgce.sharepoint.com/:p:/t/1000163/ESGBlOV_32RNuYTcPJKTU8EBO9J9Quk0Ef9jMHej1NYSAg?e=4nGzEc |
'''Comment:''' | '''Comment:''' | ||
Line 95: | Line 95: | ||
* Reminder; we are still looking for departments to share any AI content pages you have so we can build the subtopic - aiming for September to launch | * Reminder; we are still looking for departments to share any AI content pages you have so we can build the subtopic - aiming for September to launch | ||
− | Contact: christopher.colyer@ised-isde.gc.ca | + | Contact: [mailto:Christopher.colyer@ised-isde.gc.ca christopher.colyer@ised-isde.gc.ca] |
Latest revision as of 17:12, 19 August 2024
Back to GC Web Priorities Meetings - Réunions des Priorités Web du GC
Meeting information
Agenda
1. Updates
2. Roundtable
Notes and action item
Updates
DTO
- No updates this week
PCO
- Campaigns - We’ve reached out to encourage people to start early working on their campaigns
- Check out the Slack thread about campaigns, tips such as avoiding carousels
- Big thanks to Principle Publisher for new biography for Minister Mckinnon
- Today is International Self Care Day
Principle Publisher
- Removed mention of Minister Seamus O’Regan and replaced with new Minister Steven MacKinnon
- We have removed the Share this page widget from AEM, if you’re not on AEM, please remove by end of August
PCO - Impact Canada - Harry MacKay
A Beginner’s Guide to A/B Testing on Canada.ca
- We were established in 2017
- Partners such as:
- ECCC
- NRCan
- PHAC, Health Canada etc…
- What is A/B testing?
- A lot of website design is guided by top down principles of usability, composition etc..
- AB testing is a way to empirically test two or more versions of a product
- Variants are randomly assigned to subjects as they enter the page
- Considerations for good test:
- DTO has provided valuable information on A/B testing
- Canada.ca is mandated to provide clear information to the public
- Ideally, A/B testing focus on one single element at a time
- Examples of A/B testing on Canada.ca
- Example of CRA test, in this example the element you were supposed to click was more prominent, with a 20% increase in desire outcome
- OCAO ran a similar test, making button more “obviously clickable” with favorable results
- Considering statistical power
- You have to make sure you have enough page traffic for meaningful results
- The sample size is determined by incoming page views
- The only thing you really have control over to increase amount of people, is to run the test for a longer time
- This means knowing the average amount of visitors per day for your page is critical
- Checklist of statistical power
- Average number of daily visitors - Canada.ca analytics
- Baseline rate of target behavior
- Expected effect size of manipulation during testing
- Checklist of statistical power
- Adobe provides a tool that can estimate how many visitors you will need to collect with a given lift rate and baseline conversion rate
- Adobe Target is the software we use for A/B testing, Principle Publisher manages access to this software
- Adobe Target GCpedia page: https://www.gcpedia.gc.ca/wiki/Adobe_Target_User_Guide
Privacy and ethical considerations
- Personal information is not collected
- Use your own discretion using Adobe Target in environments with Protected information
- Departments are mandated to provide clear information
- This may limit ability to change wording/text on you page
- You may want to submit your plans to ethics review if you have it at your department
- Stop at a predetermined time to avoid biases
- The Sequential Analysis approach has predefined stopping rules, such as in a drug trial, to protect against the ethical ob subjecting users to underperforming version of critical web page
Key Takeways
- A/B testing is a powerful tool to improve usability
- Statistical power is the biggest concern when planning your project
- Average lift (improvement) is around 2%, Not every page on Canada.ca receives enough traffic to run a test
- The testing is relatively low cost
QUESTIONS
Q: How do you consider this vs. other kinds of testing?
A: It’s a good method once your website has been set up. It’s really for optimizing. Doesn’t address accessibility very well. It has its place in combination with other kinds of testing which for example might be better for specific users or other considerations.
Comment: A/B testing is good for settling points, for example disagreements between clients or the Minister's Office.
Comment, we recently did a presentation on Adobe Target, sharing here: https://gcxgce.sharepoint.com/:p:/t/1000163/ESGBlOV_32RNuYTcPJKTU8EBO9J9Quk0Ef9jMHej1NYSAg?e=4nGzEc
Comment:
Doing usability testing on a prototype can also help overcome some of the potential ethical considerations that Harry mentioned.
For example, if you want to test the impact of removing an entire paragraph of content and don't want to risk some users having information that other users do not, you can use usability testing instead of an A/B test.
Roundtable
ISED
- Reminder; we are still looking for departments to share any AI content pages you have so we can build the subtopic - aiming for September to launch
Contact: christopher.colyer@ised-isde.gc.ca