Changes

In this investigation Fairlearn <ref name=":0" />, has been applied to implement the FPR disparity measurement experiments. As a point of reference, IBM AI Fairness 360 <ref name=":1" />, applies a threshold of 10 on similar disparity metrics as the point beyond which the model is considered to be unfair in an interactive demo [7]. Results from the initial fishing detection model are shown in Figure 6. Due to an excessively high FPR for the troll gear type, there is an FPR disparity difference of 52.62. '''<u>These results highlight an unacceptable level of bias present in the model which must be mitigated</u>'''.

In this investigation Fairlearn <ref name=":0" />, has been applied to implement the FPR disparity measurement experiments. As a point of reference, IBM AI Fairness 360 <ref name=":1" />, applies a threshold of 10 on similar disparity metrics as the point beyond which the model is considered to be unfair in an interactive demo [7]. Results from the initial fishing detection model are shown in Figure 6. Due to an excessively high FPR for the troll gear type, there is an FPR disparity difference of 52.62. '''<u>These results highlight an unacceptable level of bias present in the model which must be mitigated</u>'''.

[[File:Chart 1.png|alt=Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.|center|thumb|703x703px|'''Figure 6: Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.''']]

[[File:Chart 1.png|alt=Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.|center|thumb|464x464px|'''Figure 6: Results for FPR and accuracy are shown for each gear type. The FPR disparity difference is measured as the difference between the highest and lowest FPR, giving a value of 52.62.''']]

==== Bias Mitigation ====

==== Bias Mitigation ====

Bias mitigation algorithms implemented in Fairlearn and other similar tools can be applied at various stages of the ML pipeline. In general, there is a trade-off between model performance and bias such that mitigation algorithms induce a loss in model performance. Initial experimentation has demonstrated this occurrence, leading to a notable loss in performance to reduce bias. This can be observed in the results shown in Figure 7 where a mitigation algorithm has been applied to reduce the FPR disparity to 28.83 at the cost of a loss in fishing detection accuracy.  

Bias mitigation algorithms implemented in Fairlearn and other similar tools can be applied at various stages of the ML pipeline. In general, there is a trade-off between model performance and bias such that mitigation algorithms induce a loss in model performance. Initial experimentation has demonstrated this occurrence, leading to a notable loss in performance to reduce bias. This can be observed in the results shown in Figure 7 where a mitigation algorithm has been applied to reduce the FPR disparity to 28.83 at the cost of a loss in fishing detection accuracy.  

[[File:Chart2.png|alt=Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.|center|thumb|691x691px|'''Figure 7: Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.''']]

[[File:Chart2.png|alt=Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.|center|thumb|479x479px|'''Figure 7: Results for FPR and accuracy after bias mitigation via Fairlearn. The FPR disparity difference is 28.83.''']]

Based on these results, it was determined that efforts on bias mitigation should be focused in the data cleaning and preparation stages of the ML pipeline as this is the earliest possible point where this can be achieved. Through additional exploratory data analysis, notable issues were identified in terms of disproportionate representation across gear types and imbalance between positive and negative instances of fishing activity for some gear types. Through adjustments to the data cleaning process as well as the application of techniques such as data balancing and data augmentation, a new version of the model training data, better suited to the task of bias mitigation, was produced. The results of these modifications can be seen in Figure 8. The FPR has been greatly reduced across all gear types while preserving an acceptable level of fishing detection accuracy. Notably, the FPR disparity difference has been reduced from its original value of 52.62 down to 2.97.  

Based on these results, it was determined that efforts on bias mitigation should be focused in the data cleaning and preparation stages of the ML pipeline as this is the earliest possible point where this can be achieved. Through additional exploratory data analysis, notable issues were identified in terms of disproportionate representation across gear types and imbalance between positive and negative instances of fishing activity for some gear types. Through adjustments to the data cleaning process as well as the application of techniques such as data balancing and data augmentation, a new version of the model training data, better suited to the task of bias mitigation, was produced. The results of these modifications can be seen in Figure 8. The FPR has been greatly reduced across all gear types while preserving an acceptable level of fishing detection accuracy. Notably, the FPR disparity difference has been reduced from its original value of 52.62 down to 2.97.  

[[File:Chart3.png|alt=Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.|center|thumb|704x704px|'''Figure 8: Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.''']]

[[File:Chart3.png|alt=Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.|center|thumb|484x484px|'''Figure 8: Results for FPR and accuracy after improvements were made to the data preparation process. The FPR disparity difference is 2.97.''']]

=== Bias Assessment Next Steps ===

=== Bias Assessment Next Steps ===