Changes

[[File:Res ai poster Dec8.jpg|alt=|center|frameless|996x996px]]

('''Français''': [[Vers un cadre d’IA responsable pour la conception de systèmes décisionnels automatisés au MPO: une étude de cas de la détection et de l’atténuation des biais|Vers un cadre d’IA responsable pour la conception de systèmes décisionnels automatisés au MPO: une étude de cas de la détection et de l’atténuation des biais - wiki (gccollab.ca)]]

 

== Executive Summary ==

== Executive Summary ==

Automated decision systems are computer systems that automate, or assist in automating, part or all of an administrative decision-making process. Currently, automated decision systems driven by Machine Learning (ML) algorithms are being used by the Government of Canada to improve service delivery. The use of such systems can have benefits and risks for federal institutions, including the Department of Fisheries and Oceans Canada (DFO). There is a need to ensure that automated decision systems are deployed in a manner that reduces risks to Canadians and federal institutions and leads to more efficient, accurate, consistent, and interpretable decisions.

Automated decision systems are computer systems that automate, or assist in automating, part or all of an administrative decision-making process. Currently, automated decision systems driven by Machine Learning (ML) algorithms are being used by the Government of Canada to improve service delivery. The use of such systems can have benefits and risks for federal institutions, including the Department of Fisheries and Oceans Canada (DFO). There is a need to ensure that automated decision systems are deployed in a manner that reduces risks to Canadians and federal institutions and leads to more efficient, accurate, consistent, and interpretable decisions.

For the Fiscal Year (2020 – 2021), the Office of the Chief Data Officer (OCDO), and Information Management and Technical Services (IMTS) co-sponsored an Artificial Intelligence (AI) pilot project to determine the extent to which AI, coupled with other data analytics techniques, can be used to gain insights, to automate tasks, and to optimize outcomes for different priority areas and business objectives at DFO. The project was funded through the 2020 – 2021 Results Fund and was developed in partnership with programs and business areas within DFO. The results included the development of three proof of concepts:

In order to support these goals, the Office of the Chief Data Steward (OCDS) is developing a data ethics framework which will provide guidance on the ethical handling of data and the responsible use of Artificial Intelligence (AI). The guidance material on the responsible use of AI addresses 6 major themes that have been identified as being pertinent to DFO projects using AI. These themes are Privacy and Security, Transparency, Accountability, Methodology and Data Quality, Fairness, and Explainability. While many of these themes have a strong overlap with the domain of data ethics, the theme of Fairness covers many ethical concerns unique to AI due to the nature of the impacts that bias can have on AI models.

 

* Predictive models for detecting vessels’ fishing behavior, to fight non-compliance with fishing regulations.

* A predictive model for the identification and  classification of endangered North Atlantic Right Whale upcalls, to minimize vessel strikes to endangered whales.

 

Supported by the (2021 – 2022) Results Fund, the OCDO and IMTS are prototyping automated decision systems, based on the outcome of the AI pilot project. The effort includes defining an internal process to detect and mitigate bias, as a potential risk of ML-based automated decision systems. A case study is designed to apply this process to assess and mitigate bias in the predictive model for detecting vessels’ fishing behavior.  

Eventually, in the long term, the goal will be to develop a comprehensive responsible AI framework to ensure responsible use of AI within DFO and to ensure compliance with the Treasury Board Directive on Automated Decision-Making.  

Supported by the (2021 – 2022) Results Fund, the OCDS and IMTS are prototyping automated decision systems, based on the outcome of the AI pilot project. The effort includes defining an internal process to detect and mitigate bias, as a potential risk of ML-based automated decision systems. A case study is designed to apply this process to assess and mitigate bias in a predictive model for detecting vessels’ fishing behavior. The process defined in this work and the results of the field study will contribute towards the guidance material that will eventually form the responsible AI component of the data ethics framework.

== Introduction ==

== Introduction ==

Unlike traditional automated decision systems, Machine Learning (ML)-based automated decision systems do not follow explicit rules authored by humans <ref>V. Fomins, "The Shift from Traditional Computing Systems to  Artificial Intelligence and the Implications for Bias," ''Smart  Technologies and Fundamental Rights,'' pp. 316-333, 2020.</ref>. ML models are not inherently objective. Data scientists train models by feeding them a data set of training examples, and the human involvement in the provision and curation of this data can make a model's predictions susceptible to bias. Due to this, applications of ML-based automated decision systems have far-reaching implications for society. These range from new questions about the legal responsibility for mistakes committed by these systems to retraining for workers displaced by these technologies. There is a need for a framework to ensure that accountable and transparent decisions are made, supporting ethical practices.

Unlike traditional automated decision systems, ML-based automated decision systems do not follow explicit rules authored by humans <ref>V. Fomins, "The Shift from Traditional Computing Systems to  Artificial Intelligence and the Implications for Bias," ''Smart  Technologies and Fundamental Rights,'' pp. 316-333, 2020.</ref>. ML models are not inherently objective. Data scientists train models by feeding them a data set of training examples, and the human involvement in the provision and curation of this data can make a model's predictions susceptible to bias. Due to this, applications of ML-based automated decision systems have far-reaching implications for society. These range from new questions about the legal responsibility for mistakes committed by these systems to retraining for workers displaced by these technologies. There is a need for a framework to ensure that accountable and transparent decisions are made, supporting ethical practices.

=== Responsible AI ===

=== Responsible AI ===

Responsible AI is a governance framework that documents how a specific organization is addressing the challenges around AI from both an ethical and a legal point of view.

Through a study of established responsible AI frameworks from other organizations and an inspection of DFO use cases in which AI is employed, a set of responsible AI themes have been identified for a DFO framework:

[[File:5gp.png|thumb|426x426px|alt=|'''Figure 1: Five main guiding principles for Responsible AI''']]In an attempt to ensure Responsible AI practices, organizations have identified guiding principles to guide the development of AI applications and solutions.  According to the research “The global landscape of AI ethics guidelines” <ref>A. Jobin, M. Ienca and E. Vayena, "The global landscape of AI ethics guidelines," ''Nature Machine Intelligence,'' p. 389–399, 2019</ref> , some principles are mentioned more often than others. However, Gartner has concluded that there is a global convergence emerging around five ethical principles <ref>S. Sicular , E. Brethenoux , F. Buytendijk and J. Hare, "AI Ethics: Use 5 Common Guidelines as Your Starting Point," Gartner, 11 July 2019. [Online]. Available: <nowiki>https://www.gartner.com/en/documents/3947359/ai-ethics-use-5-common-guidelines-as-your-starting-point</nowiki>. [Accessed 23 August 2021].</ref> :

·      Human-centric and socially beneficial

* Privacy and Security

·      Explainable and transparent

The process of bias detection and mitigation is dependent upon the identification of the context within which bias is to be assessed. Given the breadth of sources from which bias can originate, exhaustive identification of the sources relevant to a particular system and the quantification of their impacts can be impractical. As such, it is recommended to instead '''view bias through the lens of harms that can be induced by the system'''  <ref name=":0">S. Bird, M. Dudík, R. Edgar, B. Horn, R. Lutz, V. Milan, M. Sameki, H. Wallach and K. Walker, "Fairlearn: A toolkit for assessing and improving fairness in AI," Microsoft, May 2020. [Online]. Available: <nowiki>https://www.microsoft.com/en-us/research/publication/fairlearn-a-toolkit-for-assessing-and-improving-fairness-in-ai/</nowiki>. [Accessed 30 11 2021].</ref>. Common types of harm can be represented through formal mathematical definitions of fairness in decision systems. These definitions provide the foundation for the quantitative assessment of fairness as an indicator of bias in systems.

·      Secure and safe

Harms are typically considered in the context of error rates and/or rates of representation. Differences in error rates across different groups of people induce disproportionate performance of the system, leading to unjust consequences or reduced benefits for certain groups. Differences in rates of representation, even in the absence of differences in rates of error, can lead to unfair distributions of benefits or penalties across groups of people. The types of harm applicable to a particular system are highly dependent upon the intended usage of the system. A flow chart to assist in identification of the metric of fairness most relevant to a system is provided in Figure 3.

Having laid out the potentials for harm in the system, the next step is to identify protected attributes. In the context of bias detection and mitigation, a protected attribute refers to a categorical attribute for which there are concerns of bias across the attribute categories. Common examples of protected attributes are race and gender. Relevant protected attributes for the application of the system must be identified in order to investigate disproportionate impacts across the attribute categories.

Having laid out the potentials for harm in the system, the next step is to identify protected attributes. In the context of bias detection and mitigation, a protected attribute refers to a categorical attribute for which there are concerns of bias across the attribute categories. Common examples of protected attributes are race and gender. Relevant protected attributes for the application of the system must be identified in order to investigate disproportionate impacts across the attribute categories.

'''False positive (negative) rate''': The percentage of negative (positive) instances incorrectly classified as positive (negative) in a sample of instances to which binary classification is applied.

'''False positive (negative) rate''': The percentage of negative (positive) instances incorrectly classified as positive (negative) in a sample of instances to which binary classification is applied.

'''Protected attribute''': An attribute that partitions a population into groups across which fairness is expected. Examples of common protected attributes are gender and race.  

'''Protected attribute''': An attribute that partitions a population into groups across which fairness is expected. Examples of common protected attributes are gender and race.

=== Results of Bias Assessment   ===

=== Results of Bias Assessment   ===

In this investigation Fairlearn <ref name=":0" />, has been applied to implement the FPR disparity measurement experiments. As a point of reference, IBM AI Fairness 360 <ref name=":1" />, applies a threshold of 10 on similar disparity metrics as the point beyond which the model is considered to be unfair in an interactive demo [7]. Results from the initial fishing detection model are shown in Figure 6. Due to an excessively high FPR for the troll gear type, there is an FPR disparity difference of 52.62. '''<u>These results highlight an unacceptable level of bias present in the model which must be mitigated</u>'''.

In this investigation Fairlearn <ref name=":0" />, has been applied to implement the FPR disparity measurement experiments. As a point of reference, IBM AI Fairness 360 <ref name=":1" />, applies a threshold of 10 on similar disparity metrics as the point beyond which the model is considered to be unfair in an interactive demo [7]. Results from the initial fishing detection model are shown in Figure 6. Due to an excessively high FPR for the troll gear type, there is an FPR disparity difference of 52.62. '''<u>These results highlight an unacceptable level of bias present in the model which must be mitigated</u>'''.

[[File:Chart 1.png|alt=Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.|center|thumb|703x703px|'''Figure 6: Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.''']]

[[File:Chart 1.png|alt=Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.|center|thumb|464x464px|'''Figure 6: Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.''']]

==== Bias Mitigation ====

==== Bias Mitigation ====

Bias mitigation algorithms implemented in Fairlearn and other similar tools can be applied at various stages of the ML pipeline. In general, there is a trade-off between model performance and bias such that mitigation algorithms induce a loss in model performance. Initial experimentation has demonstrated this occurrence, leading to a notable loss in performance to reduce bias. This can be observed in the results shown in Figure 7 where a mitigation algorithm has been applied to reduce the FPR disparity to 28.83 at the cost of a loss in fishing detection accuracy.  

Bias mitigation algorithms implemented in Fairlearn and other similar tools can be applied at various stages of the ML pipeline. In general, there is a trade-off between model performance and bias such that mitigation algorithms induce a loss in model performance. Initial experimentation has demonstrated this occurrence, leading to a notable loss in performance to reduce bias. This can be observed in the results shown in Figure 7 where a mitigation algorithm has been applied to reduce the FPR disparity to 28.83 at the cost of a loss in fishing detection accuracy.  

[[File:Chart2.png|alt=Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.|center|thumb|691x691px|'''Figure 7: Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.''']]

[[File:Chart2.png|alt=Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.|center|thumb|479x479px|'''Figure 7: Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.''']]

Based on these results, it was determined that efforts on bias mitigation should be focused in the data cleaning and preparation stages of the ML pipeline as this is the earliest possible point where this can be achieved. Through additional exploratory data analysis, notable issues were identified in terms of disproportionate representation across gear types and imbalance between positive and negative instances of fishing activity for some gear types. Through adjustments to the data cleaning process as well as the application of techniques such as data balancing and data augmentation, a new version of the model training data, better suited to the task of bias mitigation, was produced. The results of these modifications can be seen in Figure 8. The FPR has been greatly reduced across all gear types while preserving an acceptable level of fishing detection accuracy. Notably, the FPR disparity difference has been reduced from its original value of 52.62 down to 2.97.  

Based on these results, it was determined that efforts on bias mitigation should be focused in the data cleaning and preparation stages of the ML pipeline as this is the earliest possible point where this can be achieved. Through additional exploratory data analysis, notable issues were identified in terms of disproportionate representation across gear types and imbalance between positive and negative instances of fishing activity for some gear types. Through adjustments to the data cleaning process as well as the application of techniques such as data balancing and data augmentation, a new version of the model training data, better suited to the task of bias mitigation, was produced. The results of these modifications can be seen in Figure 8. The FPR has been greatly reduced across all gear types while preserving an acceptable level of fishing detection accuracy. Notably, the FPR disparity difference has been reduced from its original value of 52.62 down to 2.97.  

[[File:Chart3.png|alt=Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.|center|thumb|704x704px|'''Figure 8: Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.''']]

[[File:Chart3.png|alt=Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.|center|thumb|484x484px|'''Figure 8: Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.''']]

=== Bias Assessment Next Steps ===

=== Bias Assessment Next Steps ===

== The Path Forward ==

== The Path Forward ==

Responsible AI is the only way to mitigate AI risks, and bias risks are considered a subset of such risks. As DFO moves towards adopting AI to support decision-making and improve service delivery, there is a need to ensure that these decisions are not only bias-aware, but also accurate, human-centric, explainable, and privacy-aware.

The development of a process for bias identification and mitigation is a step towards a framework that supports responsible use of AI. In order to fully develop this framework, guidance for additional processes is required. In particular, the theme of Explainability is another topic with requirements that are unique to the use of AI. Next steps in this area will require the identification of tools and the development of guidance to support the use of interpretable models and explainability algorithms for black-box models. Further to this, a more general process is required to enable project teams to assess their compliance across all themes of responsible AI. The OCDS is currently undertaking the development of these resources.

<references />

<references />