Difference between revisions of "July 24 - A Beginner’s Guide to A/B Testing on Canada.ca"

From wiki
Jump to navigation Jump to search
(Created page with "Back to GC Web Priorities Meetings - Réunions des Priorités Web du GC ==Meeting information== ===Agenda=== 1. Updates 2. Roundtable =Notes and action item= ===Updates==...")
 
 
(6 intermediate revisions by the same user not shown)
Line 8: Line 8:
 
===Updates===
 
===Updates===
 
'''DTO'''
 
'''DTO'''
−
*We noticed IRCC has added Sign in to their landing page, way to go and nice to see it starting to roll out
+
 
 +
* No updates this week
 +
 
 
'''PCO'''
 
'''PCO'''
−
*Thanks to everyone for helping with Dogs on vacation content for CFIA
+
 
−
*Thanks for supporting on the file for new content for wildfires
+
* Campaigns - We’ve reached out to encourage people to start early working on their campaigns
 +
* Check out the [https://gc-web-comms.slack.com/archives/C018SF3J4KF/p1721760193367859 Slack thread about campaigns], tips such as avoiding carousels
 +
* Big thanks to Principle Publisher for new biography for Minister Mckinnon
 +
* Today is International Self Care Day
 +
 
 
'''Principle Publisher'''
 
'''Principle Publisher'''
−
*The Share this page widget has been removed from AEM, we ask the non AEM teams to follow suit within month please
 
−
*Rational: We looked at the numbers and it was hardly ever used, also that functionality is built into a lot of browsers
 
−
'''CFIA - Lessons learned from New dog cross-border regulations'''
 
−
*A lot of our work deals with import/export issues
 
−
*Recently there were changes to requirements to bring your dog into the US on vacation
 
−
*The attention meant we had to interact with media more than usual
 
−
*Observations from this recent work:
 
−
*#Good idea to meet with your partners at the beginning of the exercise, for instance we met with PCO, good to have backing from the Centre
 
−
*#We need more support from management to harness the power of plain language to communicate about regulations/laws
 
−
*#We would love to be part of the conversation about content design communication checklists and templates
 
−
*#*Would be good to meet with others to talk about lessons learned
 
−
*#We relied a lot on Canada.ca design for our content for expertise how to organize things
 
−
'''Principle Publisher - Migration to a new search solution'''
 
−
*GC search is a common and shared search service provided by PP
 
−
*Indexes most publicly available GC web pages, 4.6 million searches a month
 
−
*We work with departments to configure the search to their needs
 
−
*New solution is now live, started on Jun 11, 2024
 
−
*New vendor is Coveo Solutions, a Quebec based company
 
−
*We worked to index 125 websites, roughly 2.9 million pages, built from scratch
 
−
Future of search
 
−
*New system provides a starting point to allow GC Search to grow
 
−
*Lays the foundation for: AI machine learning and analytics
 
−
Input from Conveo
 
−
*Search function works on a “learned” basis, based on input from users
 
−
*For example, will bring up documents that are clicked often, based on particular searches, this is an example of machine learning
 
−
*Uses “automated answering”
 
−
*Also uses smart snippet, copy and pastes answers from relevant documents, similar to AI, but has the advantage that GOC writes the answers which gives us more control
 
−
*We can work with metadata to improve searches, can work with you to determine how to best use and write metadata
 
−
'''Questions'''
 
  
−
'''Q''': How does a smart snippet work?
+
* Removed mention of Minister Seamus O’Regan and replaced with new Minister Steven MacKinnon
 +
* We have removed the Share this page widget from AEM, if you’re not on AEM, please remove by end of August
 +
 
 +
'''PCO - [https://impact.canada.ca/en Impact Canada] - Harry MacKay'''
 +
 
 +
'''[[:en:images/2/24/AB_testing_IIU_EN_2024-04-26_for_posting.pptx|A Beginner’s Guide to A/B Testing on Canada.ca]]'''  
 +
 
 +
* We were established in 2017
 +
* Partners such as:
 +
** ECCC
 +
** NRCan
 +
** PHAC, Health Canada etc…
 +
* What is A/B testing?
 +
** A lot of website design is guided by top down principles of usability, composition etc..
 +
** AB testing is a way to empirically test two or more versions of a product
 +
** Variants are randomly assigned to subjects as they enter the page
 +
* Considerations for good test:
 +
** DTO has provided valuable information on A/B testing
 +
** Canada.ca is mandated to provide clear information to the public
 +
** Ideally, A/B testing focus on one single element at a time
 +
* Examples of A/B testing on Canada.ca
 +
** Example of CRA test, in this example the element you were supposed to click was more prominent, with a 20% increase in desire outcome
 +
** OCAO ran a similar test, making button more “obviously clickable” with favorable results
 +
 
 +
* Considering statistical power
 +
** You have to make sure you have enough page traffic for meaningful results
 +
** The sample size is determined by incoming page views
 +
** The only thing you really have control over to increase amount of people, is to run the test for a longer time
 +
** This means knowing the average amount of visitors per day for your page is critical
 +
*** Checklist of statistical power
 +
**** Average number of daily visitors - Canada.ca analytics
 +
**** Baseline rate of target behavior
 +
**** Expected effect size of manipulation during testing
 +
** Adobe provides a tool that can estimate how many visitors you will need to collect with a given lift rate and baseline conversion rate
 +
* [https://business.adobe.com/ca/products/target/adobe-target.html Adobe Target] is the software we use for A/B testing, Principle Publisher manages access to this software
 +
** Adobe Target GCpedia page: https://www.gcpedia.gc.ca/wiki/Adobe_Target_User_Guide
 +
 
 +
Privacy and ethical considerations
 +
 
 +
* Personal information is not collected
 +
* Use your own discretion using Adobe Target in environments with Protected information
 +
* Departments are mandated to provide clear information
 +
** This may limit ability to change wording/text on you page
 +
* You may want to submit your plans to ethics review if you have it at your department
 +
* Stop at a predetermined time to avoid biases
 +
* The Sequential Analysis approach has predefined stopping rules, such as in a drug trial, to protect against the ethical ob subjecting users to underperforming version of critical web page
  
−
'''Aː''' Our smart snippet function looks at the html code on the page to guess which snippet is the best answer to the question. We can also customize that further, and work with you to pick the best snippet.
+
Key Takeways
  
−
'''Q:''' How do we manage promoting search terms at the whole of GOC level?
+
* A/B testing is a powerful tool to improve usability
 +
* Statistical power is the biggest concern when planning your project
 +
** Average lift (improvement) is around 2%, Not every page on Canada.ca receives enough traffic to run a test
 +
* The testing is relatively low cost
  
−
'''A:''' This is an issue that doesn’t necessarily have anything to do with the new search tool. We are looking at issues such as putting time limits on boosting terms.
+
'''QUESTIONS'''
  
−
DTO: This is something that the Search Working Group will be looking at.
+
'''Q:''' How do you consider this vs. other kinds of testing?
  
−
'''Q:''' Does metadata need to be visible on source code? Talking about metadata in AEM but not in the source code.
+
'''A:''' It’s a good method once your website has been set up. It’s really for optimizing. Doesn’t address accessibility very well. It has its place in combination with other kinds of testing which for example might be better for specific users or other considerations.  
  
−
'''A:''' Strictly speaking no it doesn’t need to be visible in source code. There are ways, but for that specific case, we will have to investigate.
+
Comment: A/B testing is good for settling points, for example disagreements between clients or the Minister's Office.  
  
−
'''Q:''' For the time being do we work on metadata the same way?
+
Comment, we recently did a presentation on Adobe Target, sharing here: https://gcxgce.sharepoint.com/:p:/t/1000163/ESGBlOV_32RNuYTcPJKTU8EBO9J9Quk0Ef9jMHej1NYSAg?e=4nGzEc
  
−
'''A:''' Yes, for now you don’t have to change anything.
+
'''Comment:'''  
  
−
'''Q:''' Can we get view access to the dashboards?
+
Doing usability testing on a prototype can also help overcome some of the potential ethical considerations that Harry mentioned.
  
−
'''A:''' We have very detailed analytics about the searches, at this time they are accessible only to Coveo, because there’s some expertise in reading the dashboard. But this could evolve so that departments could have access to them over time. Whether it be full access or some key indicators, we will discuss this.
+
For example, if you want to test the impact of removing an entire paragraph of content and don't want to risk some users having information that other users do not, you can use usability testing instead of an A/B test.  
  
−
'''Q:''' Is it possible even to have minimal access to dashboards?
+
== Roundtable ==
 +
'''ISED'''
  
−
'''A:''' It wasn’t part of the initial contract, but there seems to be high interest and we will discuss it.
+
* Reminder; we are still looking for departments to share any AI content pages you have so we can build the subtopic - aiming for September to launch
  
−
Input from Service Canada Analytics team: Remember that there’s also a lot of analytical info available in Adobe Analytics and we will also be talking with Conveo to see what we can make available. We need to make sure also that both systems of Analytics are complementary, and not muddy the waters.
+
Contact: [mailto:Christopher.colyer@ised-isde.gc.ca christopher.colyer@ised-isde.gc.ca]

Latest revision as of 17:12, 19 August 2024

Back to GC Web Priorities Meetings - Réunions des Priorités Web du GC

Meeting information

Agenda

1. Updates

2. Roundtable

Notes and action item

Updates

DTO

  • No updates this week

PCO

  • Campaigns - We’ve reached out to encourage people to start early working on their campaigns
  • Check out the Slack thread about campaigns, tips such as avoiding carousels
  • Big thanks to Principle Publisher for new biography for Minister Mckinnon
  • Today is International Self Care Day

Principle Publisher

  • Removed mention of Minister Seamus O’Regan and replaced with new Minister Steven MacKinnon
  • We have removed the Share this page widget from AEM, if you’re not on AEM, please remove by end of August

PCO - Impact Canada - Harry MacKay

A Beginner’s Guide to A/B Testing on Canada.ca

  • We were established in 2017
  • Partners such as:
    • ECCC
    • NRCan
    • PHAC, Health Canada etc…
  • What is A/B testing?
    • A lot of website design is guided by top down principles of usability, composition etc..
    • AB testing is a way to empirically test two or more versions of a product
    • Variants are randomly assigned to subjects as they enter the page
  • Considerations for good test:
    • DTO has provided valuable information on A/B testing
    • Canada.ca is mandated to provide clear information to the public
    • Ideally, A/B testing focus on one single element at a time
  • Examples of A/B testing on Canada.ca
    • Example of CRA test, in this example the element you were supposed to click was more prominent, with a 20% increase in desire outcome
    • OCAO ran a similar test, making button more “obviously clickable” with favorable results
  • Considering statistical power
    • You have to make sure you have enough page traffic for meaningful results
    • The sample size is determined by incoming page views
    • The only thing you really have control over to increase amount of people, is to run the test for a longer time
    • This means knowing the average amount of visitors per day for your page is critical
      • Checklist of statistical power
        • Average number of daily visitors - Canada.ca analytics
        • Baseline rate of target behavior
        • Expected effect size of manipulation during testing
    • Adobe provides a tool that can estimate how many visitors you will need to collect with a given lift rate and baseline conversion rate
  • Adobe Target is the software we use for A/B testing, Principle Publisher manages access to this software

Privacy and ethical considerations

  • Personal information is not collected
  • Use your own discretion using Adobe Target in environments with Protected information
  • Departments are mandated to provide clear information
    • This may limit ability to change wording/text on you page
  • You may want to submit your plans to ethics review if you have it at your department
  • Stop at a predetermined time to avoid biases
  • The Sequential Analysis approach has predefined stopping rules, such as in a drug trial, to protect against the ethical ob subjecting users to underperforming version of critical web page

Key Takeways

  • A/B testing is a powerful tool to improve usability
  • Statistical power is the biggest concern when planning your project
    • Average lift (improvement) is around 2%, Not every page on Canada.ca receives enough traffic to run a test
  • The testing is relatively low cost

QUESTIONS

Q: How do you consider this vs. other kinds of testing?

A: It’s a good method once your website has been set up. It’s really for optimizing. Doesn’t address accessibility very well. It has its place in combination with other kinds of testing which for example might be better for specific users or other considerations.

Comment: A/B testing is good for settling points, for example disagreements between clients or the Minister's Office.

Comment, we recently did a presentation on Adobe Target, sharing here: https://gcxgce.sharepoint.com/:p:/t/1000163/ESGBlOV_32RNuYTcPJKTU8EBO9J9Quk0Ef9jMHej1NYSAg?e=4nGzEc

Comment:

Doing usability testing on a prototype can also help overcome some of the potential ethical considerations that Harry mentioned.

For example, if you want to test the impact of removing an entire paragraph of content and don't want to risk some users having information that other users do not, you can use usability testing instead of an A/B test.

Roundtable

ISED

  • Reminder; we are still looking for departments to share any AI content pages you have so we can build the subtopic - aiming for September to launch

Contact: christopher.colyer@ised-isde.gc.ca

Retrieved from "https://wiki.gccollab.ca/index.php?title=July_24_-_A_Beginner’s_Guide_to_A/B_Testing_on_Canada.ca&oldid=124051"