View source for Towards a Responsible AI Framework for the Design of Automated Decision Systems in DFO: a Case Study of Bias Detection and Mitigation

[[File:CaseStudyofResponsibleAIatDFO.jpg|alt=|center|frameless|996x996px]]

== Executive Summary ==
Automated Decision Systems (ADS) are computer systems that automate, or assist in automating, part or all of an administrative decision-making process. Currently, automated decision systems driven by Machine Learning (ML) algorithms are being used by the Government of Canada to improve service delivery. The use of such systems can have benefits and risks for federal institutions, including the Department of Fisheries and Oceans Canada (DFO). There is a need to ensure that automated decision systems are deployed in a manner that reduces risks to Canadians and federal institutions and leads to more efficient, accurate, consistent, and interpretable decisions.

For the Fiscal Year (2020 - 2021), the Office of Chief Data Officer (OCDO), and Information Management and Technical Services (IMTS) co-sponsored an Artificial Intelligence (AI) pilot project to determine the extent to which AI, coupled with other data analytics techniques, can be used to gain insights, to automate tasks, and to optimize outcomes for different priority areas and business objectives at DFO. The project was funded through the 2020-21 Results Fund and was developed in partnership with programs and business areas within DFO. The results included the development of three proof of concepts:

* Predictive models for detecting vessels’ fishing behavior, to fight     noncompliance with fishing regulations.
* A predictive model for the identification and classification of     endangered North Atlantic Right Whale upcalls, to minimize vessels’ strike     to endangered whales.
* A predictive model for clustering ocean data, to advance ocean     science.

Supported by the (2021 – 2022) Results Fund, the OCDO and IMTS are prototyping automated decision systems, based on the outcome of the AI pilot project. The effort includes defining an internal process to detect and mitigate bias, as a potential risk of ML-based automated decision systems. A case study is designed to apply this process to assess and mitigate bias in the predictive model for detecting vessels’ fishing behavior. 

Eventually, in the long term, the goal will be to develop a comprehensive responsible AI framework to ensure responsible use of AI within DFO and to ensure compliance with Treasury’s Board directive on automated decision making.

== Introduction ==
Unlike traditional Automated Decision Systems (ADS), Machine Learning (ML)-based ADS do not follow explicit rules authored by humans. ML models are not inherently objective. Data scientists train models by feeding them a data set of training examples, and the human involvement in the provision and curation of this data can make a model's predictions susceptible to bias. Due to this, applications of ML-based ADS have far-reaching implications for society. These range from new questions about the legal responsibility for mistakes committed by these systems to retraining for workers displaced by these technologies. There is a need for a framework to ensure that accountable and transparent decisions are made, supporting ethical practices.

=== Responsible AI ===
[[File:AiGuidingPrinciples.png|thumb|367x367px|The research “ The global landscape of AI ethics guidelines “ , source: <nowiki>https://www.nature.com/articles/s42256-019-0088-2</nowiki>]]
Responsible AI is a governance framework that documents how a specific organization is addressing the challenges around artificial intelligence (AI) from both an ethical and legal point of view. 
In an attempt to ensure Responsible AI practices, organizations have identified guiding principles to guide the development of AI applications and solutions. According to the research “The global landscape of AI ethics guidelines” [1], some principles are mentioned more often than others. However, Gartner has concluded that there is a global convergence emerging around five ethical principles:
[[File:5gp.png|center|thumb|339x339px]]


The definition of the various guiding principles is included in [1].

=== Treasury Board Directive on Automated Decision-Making ===
Treasury Board’s Directive on Automated Decision-Making is defined as a policy instrument to promote ethical and responsible use of Artificial Intelligence. It outlines the responsibilities of federal institutions using AI-automated decision systems.

==== The Algorithmic Impact Assessment Process ====
According to the Directive, an Algorithmic Impact Assessment (AIA)  must be conducted before the production of any ADS, to assess the risks of the system. The assessment must be updated at regular intervals when there is a change to the functionality or the scope of the Automated Decision System.

The AIA tool is a questionnaire that determines the impact level of an automated decision system. It is composed of 48 risk and 33 mitigation questions. Assessment scores are based on many factors including systems design, algorithm, decision type, impact, and data. The tool results in an impact categorization level between I (presenting the least risk) and IV (presenting the greatest risk).   Based on the impact level (which will be published), the Directive may impose additional requirements. The process is described in Figure 2.

== Bias Detection and Mitigation ==
To comply with the TBS Directive on ADS, DFO is working on establishing an internal process, as part of the Result Fund project for 2021 – 2022,  to detect and mitigate potential bias in ADS.

=== Bias Detection and Mitigation Process ===
The process of bias detection and mitigation is dependent upon the identification of the context within which bias is to be assessed. Given the breadth of sources from which bias can originate, exhaustive identification of the sources relevant to a particular system and the quantification of their impacts can be impractical. As such, it is recommended to instead '''view bias through the lens of harms that can be induced by the system''' [2]. Common types of harm can be represented through formal mathematical definitions of fairness in decision systems. These definitions provide the foundation for the quantitative assessment of fairness as an indicator of bias in systems.

Harms are typically considered in the context of error rates and/or rates of representation. In the case of error rates, differences in false positive/negative rates across different groups of people induce disproportionate performance of the system, leading to unjust consequences or reduced benefits for certain groups. Differences in rates of representation, even in the absence of differences in rates of error, can lead to unfair distributions of benefits or penalties across groups of people. The types of harm applicable to a particular system are highly dependent upon the intended usage of the system. A flow chart to assist in identification of the metric of fairness most relevant to a system is provided in Figure 3.


Having laid out the potentials for harm in the system, the next step is to identify protected attributes. In the context of bias detection and mitigation, a protected attribute refers to a categorical attribute for which there are concerns of bias across the attribute categories. Common examples of protected attributes are race and gender. Relevant protected attributes for the application of the system must be identified in order to investigate disproportionate impacts across the attribute categories.

Once the assessment of potential harms and affected groups has been conducted, a bias detection and mitigation tools can be applied to attempt to remedy issues of unfairness in the system. This overall process can thus be broken up into four steps:

1.      Identification of potential harms

2.      Identification of protected attributes

3.      Bias detection

4.      Bias mitigation

=== Bias Detection and Mitigation Tools ===
There is a growing number of technical bias detection and mitigation tools, which can supplement the AI practitioner’s capacity to avoid and mitigate AI bias. As part of the 2021 -2022 Result fund project, DFO has explored the following bias detection tools that AI practitioners can use to detect and remove bias.

==== '''Microsoft’s FairLearn''' ====
An open-source toolkit by Microsoft that allows AI practitioners to detect and correct the fairness of their AI systems. To optimize the trade-offs between fairness and model performance, the toolkit includes two components: an interactive visualization dashboard and bias mitigation algorithm [2].

==== '''IBM AI Fairness 360''' ====
An open-source toolkit by IBM that helps AI practitioners to easily check for biases at multiple points along their machine learning pipeline, using the appropriate bias metric for their circumstances. It comes with more than 70 fairness metrics and 11 unique bias mitigation algorithms [3].

In the following section, details are provided on an investigation into one of the AI pilot project POCs using this proposed process. Through this investigation, further information can be gleaned into the practical considerations of bias detection and mitigation.

== Case Study: Predictive Model for Detecting Vessels’ Fishing Behavior ==
Through the AI work funded by the 2020-2021 Results Fund, a POC was developed for the automated detection of vessel fishing activity using a machine learning model. The resulting predictive model takes the vessel tracks as input and outputs the vessel tracks annotated with its activity i.e. fishing or not-fishing, as illustrated in Figure 4. The model uses features such as vessel speed deviation, course deviation, and gear type to detect vessel activity. The insight gained from the predictive model is then combined with other data sources, such as fisheries management areas and license conditions to detect non-compliance with fisheries regulations.

The fishing detection predictive model will be used as the backbone for building an automated decision system for the detection of non-compliant fishing behavior. After consultation with TBS, DFO has established that the TBS directive on automated decision systems applies to the production of the model.

To comply with the Directive, and to ensure that this automated decision system is both effective and fair, an investigation has been conducted into the potential for bias in the fishing detection model, and bias mitigation techniques have been applied.

== Terms and Definitions ==
'''Bias''': In the context of automated decision systems, the presence of systematic errors, or misrepresentation in data and system outcomes. Bias may originate from a variety of sources including societal bias (e.g., discrimination) embedded in data, unrepresentative data sampling or preferential treatment imparted through algorithms.

'''Disparity''': A systematic lack of fairness across different groups. Formally measured as the maximal difference between the values of a metric applied across the groups defined by a ''protected attribute''.

'''False negative''': An error in binary classification in which a positive instance is incorrectly classified as negative.

'''False positive''': An error in binary classification in which a negative instance is incorrectly classified as positive.

'''False positive (negative) rate''': The percentage of negative (positive) instances incorrectly classified as positive (negative) in a sample of instances to which binary classification is applied.

'''Protected attribute''': An attribute that partitions a population into groups across which fairness is expected. Examples of common protected attributes are gender and race.

== Results of Bias Assessment   ==


The bias assessment process has been conducted following the four steps proposed as indicated in the Bias Detection and Mitigation Process. Details on the results of the investigation for each of the steps are provided below.

=== Identification of Potential Harms ===
The investigation was initiated with an assessment of the potential for harms induced by bias in the system. For the use case of non-compliance detection, there always exists an objective ground truth. That is to say, through complete knowledge of the actions taken by a fishing vessel and the conditions specified in a fishing license, it can be objectively determined whether or not a vessel is in compliance with its license conditions. When a system is able to correctly make this determination with perfect consistency, there would be no potential for harm through biased decisions made by the system. However, due to the practical constraints of incomplete information (e.g., fishing activity of the vessels), perfect accuracy is not realistically achievable. Thus, the potential for harm exists in the instances where the system makes errors in its decisions.

In the binary (i.e., yes or no) detection of fishing activity, the system is susceptible to two types of errors:

1.      False positives – Reporting fishing activity where it has not occurred

2.      False negatives – Reporting an absence of fishing activity when in fact it has occurred

When the fishing detections are checked against license conditions, these errors can then take on two alternate forms:

1.      False positives may lead to incorrect reporting of non-compliance

2.      False negatives may lead to missed detections of non-compliance

As the system is designed to assist in the identification of cases in which punitive action is required, the false positives have the potential to cause harms to the operators of the vessels that are incorrectly flagged for non-compliance. It is important to note that the intended use of this system is not to make final determinations on the occurrence of non-compliant behavior but rather to assist fishery officers in the identification of cases that warrant further investigation. False positives are therefore unlikely to result in undue punitive actions. However, they result in unwarranted scrutiny of vessel activities, which is both undesirable and unfair to vessel operators. While perfect detection of fishing activity may be unattainable, it is nevertheless critical to put in best efforts to reduce error rates and to mitigate disproportionate (i.e., biased) impacts across different groups of people.

Although false positives have been identified as the primary source of potential harm through this investigation, it is important to note that some degree of less direct harm exists in false negatives as well. Missed detections of non-compliance may result in illegal fishing going uncaught. This has a cost to both DFO as well as the commercial fisheries. Should certain fisheries or geographic areas be more susceptible to false negatives than others, this may lead to disproportionate distributions of these costs.

=== Bias Detection ===
To assess model fairness across different gear types, the data is first partitioned according to the gear type and then False Positive Rate (FPR) disparity [3] is measured. The FPR is given by the percentage of negative instances (of fishing activity) that are mislabeled by the model as being positives instances. The FPR disparity is measured as the greatest difference between the FPR of each grouping of vessels by gear type. The greater this difference is measured to be, the greater the degree of bias and unfairness in the system.

In this investigation FairLearn [2], has been applied to implement the FPR disparity measurement experiments. As a point of reference, IBM AI Fairness 360 [3], applies a threshold of 10 on similar disparity metrics as the point beyond which the model is considered to be unfair in an interactive demo [4]. Results from the initial fishing detection model are shown in Table 1. Due to an excessively high FPR for the troll gear type, there is an FPR disparity difference of 52.62. These results highlight an unacceptable level of bias present in the model which must be mitigated.

{| class="wikitable"
|
|'''Fixed'''
|'''Longline'''
|'''Pole and Line'''
|'''Purse Seine'''
|'''Trawl'''
|'''Troll'''
|'''Unknown'''
|-
|'''False Positive Rate'''
|15.98
|11.78
|1.41
|2.67
|5.18
|54.03
|4.62
|-
|'''Accuracy'''
|90.25
|92.95
|98.92
|96.65
|96.63
|90.64
|97.17
|} 

'''Table 1''': Results for FPR and detection accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.

=== Bias Mitigation ===


Bias mitigation algorithms implemented in FairLearn and other similar tools can be applied at various stages of the machine learning pipeline. In general, there is a trade-off between model performance and bias such that mitigation algorithms induce a loss in model performance. Initial experimentation has demonstrated this occurrence leading to a notable loss in performance to reduce bias. This can be observed in the results shown in Table 2 where a mitigation algorithm has been applied to reduce the FPR disparity to 28.83 at the cost of a loss in fishing detection accuracy. 
{| class="wikitable"
|
|'''Fixed'''
|'''Longline'''
|'''Pole and Line'''
|'''Purse Seine'''
|'''Trawl'''
|'''Troll'''
|'''Unknown'''
|-
|'''False Positive Rate'''
|13.46
|13.00
|11.68
|10.83
|12.33
|39.52
|10.69
|-
|'''Accuracy'''
|93.01
|88.51
|91.75
|91.08
|95.08
|78.15
|95.30
|}
'''Table 2''': Results for FPR and accuracy after bias mitigation via FairLearn. The FPR disparity difference is 28.83.

Based on these results, it was determined that efforts on bias mitigation should be focused in the data cleaning and preparation stages of the machine learning pipeline as this is the earliest possible point where this can be achieved. Through additional exploratory data analysis, notable issues were identified in terms of disproportionate representation across gear types and imbalance between positive and negative instances for some gear types. Through adjustments to the data cleaning process as well as the application of techniques such as data balancing and data augmentation, a new version of the model training data, better suited to the task of bias mitigation, was produced. The results of these modifications can be seen in Table 3. The FPR has been greatly reduced across all gear types while preserving an acceptable level of fishing detection accuracy. Notably, the FPR disparity difference has been reduced from its original value of 52.62 down to 2.97.
{| class="wikitable"
|
|'''Fixed'''
|'''Longline'''
|'''Pole and Line'''
|'''Purse Seine'''
|'''Trawl'''
|'''Troll'''
|'''Unknown'''
|-
|'''False Positive Rate'''
|4.90
|4.59
|1.93
|2.43
|2.33
|4.44
|1.98
|-
|'''Accuracy'''
|88.38
|88.98
|95.65
|96.82
|95.53
|90.22
|95.76
|}
'''Table 3''': Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.

=== Bias Assessment Next Steps ===
The results of this investigation have demonstrated successful identification and mitigation of bias present in an automated decision system. It is important to note that there may be diverse possible interpretations of harms and protected attributes. Furthermore, the bias and fairness of a system may drift over time. Continued monitoring is essential to identify potential unfairness both due to previously unidentified sources of bias or through drift in the system. A continuation of this investigation into a field study stage will address these tasks.

== The Path Forward ==
Responsible AI is the only way to mitigate AI risks, and bias risks are considered a subset of such risks. As DFO moves towards adopting AI to support decision-making and improve service delivery, there is a need to ensure that these decisions are not only bias-aware, but also accurate, human-centric, explainable, and privacy-aware.

DFO is in the process of defining guiding principles to guide the development of AI applications and solutions. Once defined, various tools will be considered and/or developed to operationalize such principles.


== Bibliography ==
{| class="wikitable"
|[1]
|A. Jobin, M. Ienca and E. Vayena, "The global    landscape of AI ethics guidelines," ''Nature Machine Intelligence,'' p.    389–399, 2019.
|-
|[2]
|S. Bird, M. Dudík, R. Edgar, B. Horn, R. Lutz, V.    Milan, M. Sameki, H. Wallach and K. Walker, "Fairlearn: A toolkit for    assessing and improving fairness in AI," Microsoft, May 2020.    [Online]. Available: <nowiki>https://www.microsoft.com/en-us/research/publication/fairlearn-a-toolkit-for-assessing-and-improving-fairness-in-ai/</nowiki>.    [Accessed 30 November 2021].
|-
|[3]
|IBM Developer Staff, "AI Fairness 360,"    IBM, 14 November 2018. [Online]. Available: <nowiki>https://developer.ibm.com/open/projects/ai-fairness-360/</nowiki>.    [Accessed 28 July 2021].
|-
|[4]
|"AI Fairness 360 - Demo," IBM Reasearch    Trusted AI. [Online]. [Accessed 26 July 2021].
|-
|[5]
|V. Fomins, "The Shift from Traditional    Computing Systems to Artificial Intelligence and the Implications for    Bias," ''Smart Technologies and Fundamental Rights,'' pp. 316-333,    2020.
|}


----[ER1]Earlier  you have indicated that the process will follow the 4 previously mentioned  steps. I think you missed step 2 here “Identification of protected  attributes”. Isn’t it the gear type? Can you add something to elude to that?