AI's Got Some Explaining to Do
Why We're Demanding Answers from Our Smartest Machines
Image generated by Gemini AI
It's no secret that Artificial Intelligence (AI) is becoming ever-more present in a variety of aspects of our lives. Making key decisions in everything from medical diagnosis to loan decisions. With this in mind, there is increased importance on understanding how these models are making these decisions: ensuring that they are making their decisions on relevant information and that they are making these decisions fairly.
That's where Explainable AI (XAI) comes in, it's an area that's (rightly) been getting increased attention lately, as we aim to shed some light on some decisions that these systems are making.
What is Explainability AI (XAI)?
XAI is focused on understanding how AI systems make decisions, and as AI becomes more integrated into critical areas of our lives, it's crucial that we can trust and validate its decisions [1].
XAI isn't just about transparency; it's about accountability and fairness [2]. Imagine an AI system denying you a loan or misdiagnosing a medical condition…wouldn't you want to know why?
Techniques for Peering into the AI Mind
Scientists and researchers have developed several techniques to make AI more explainable:
1. Model Attention:
Helps us understand which parts of the input data the AI focuses on most. It's particularly useful in natural language processing [3].
2. Saliency Maps:
Provide visual explanations for image-based decisions, highlighting the most important parts of an image for the AI's prediction [4].
Figure 1: Saliency map highlighting the most important part of the image for predicting ‘person’. Source [4]
3. LIME (Local Interpretable Model-agnostic Explanations):
Creates a simpler, interpretable model to explain individual predictions [5].
Figure 2: Using LIME to highlight important words for the model, blue words indicate ”atheism”, orange words indicate “christian”
4. Shapley Values:
Borrowed from game theory, this method assigns importance values to different features in the decision-making process [6].
Figure 3: A “beeswarm” plot from SHAP to examine the impact of different features on income from census data
The Challenge of Modern AI Models
While these XAI techniques work well for traditional ML models, modern AI systems like Large Language Models (LLMs) present new challenges.
These massive models, from OpenAI and others, can process and generate human-like text, but understanding their decision-making process is far from straightforward [7].
Researchers are exploring new methods like attention visualisation and prompt engineering to shed light on these complex systems.
These new approaches aim to help us understand how LLMs work and explain their outputs. As AI continues to advance, finding ways to make it more transparent and explainable remains a key priority.
Real-World Impact: When Explainability Matters Most
Explainability isn't just an academic exercise, it has real-world implications across various sectors:
1. Healthcare:
A recent study published in Nature Medicine [8] examined AI models used for analysing chest X-rays. The researchers discovered that these models were inadvertently developing the ability to predict demographic information such as race, gender, and age from the images, despite not being explicitly trained to do so.
This unintended capability led to "fairness gaps" in diagnostic accuracy across different demographic groups.
Without explainable AI techniques, such critical biases might go unnoticed, potentially leading to disparities in healthcare outcomes and misdiagnoses for certain populations.
Having personally worked in healthcare for 8 years, I've seen firsthand the importance of understanding how decisions are made. Explainable AI techniques not only help identify potential biases but also enable healthcare professionals to integrate AI with confidence and to ensure equitable care for all patients.
2. Finance:
In areas like credit scoring and fraud detection, understanding why an AI system made a particular decision is essential for regulatory compliance and customer trust [9].
3. Autonomous Vehicles:
Explainable AI techniques are vital for understanding and validating the decision-making processes of self-driving cars, enhancing safety and reliability.
Learning from AI's Mistakes
Sometimes, the true value of explainability becomes apparent when it reveals flaws in model learning.
A classic example is the 'Husky vs Wolf' case study: researchers trained a model to differentiate between huskies and wolves, only to discover that the model was primarily focusing on the presence of snow in the background, rather than the features of the animals themselves [5].
Figure 4: Image determining husky vs wolf based on the presence of snow rather than animal features. Source[5]
This case highlighted how models can learn spurious correlations that don't generalise well, emphasising the need for explainable AI to uncover such issues.
The Road Ahead: Challenges and Opportunities
While we've made significant progress in explainable AI, several challenges remain:
1. Balancing Accuracy and Explainability:
There's often a perceived trade-off between model performance and explainability. However, recent advancements suggest that this gap may be narrowing.
2. Standardisation:
As the field evolves, there's a growing need for standardised approaches and metrics for evaluating AI transparency [10].
3. Ethical Considerations:
Ensuring that explanations are accessible and meaningful to diverse user groups, including those without technical backgrounds, is an ongoing challenge.
Looking ahead, we can expect to see increased integration of explainable AI in critical sectors, advancements in natural language explanations, and the development of more comprehensive regulatory frameworks. As AI continues to evolve, so too will our methods for understanding and explaining these complex systems.
Conclusion: The Power of Understanding
Explainability in AI and ML is not just a technical challenge; it's a crucial component of responsible AI development. As these technologies become more deeply integrated into our lives, the ability to understand, trust, and validate their decisions becomes paramount.
By continuing to advance the field of explainable AI, we can harness the full potential of these powerful technologies while ensuring transparency, fairness, and accountability. After all, the goal isn't just to create smarter machines, but to create a smarter, fairer world for all of us.
If you're interested in the possibilities of explainable AI and how it can benefit your organisation, we invite you to explore our AI and ML services: whether you're looking to implement transparent AI solutions or enhance the explainability of your existing models, we’re here to help.
Review our services page to learn more about our offerings in AI, ML, data science and NLP or get in touch to discuss your specific needs.
References
[1] F. Doshi-Velez and B. Kim, "Towards A Rigorous Science of Interpretable Machine Learning," arXiv preprint arXiv:1702.08608, 2017.
[2] C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead," Nature Machine Intelligence, vol. 1, no. 5, pp. 206-215, 2019.
[3] A. Vaswani et al., "Attention is All you Need," in Advances in Neural Information Processing Systems, 2017, pp. 5998-6008.
[4] R. R. Selvaraju et al., "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization," in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 618-626.
[5] M. T. Ribeiro, S. Singh, and C. Guestrin, "Why Should I Trust You?: Explaining the Predictions of Any Classifier," in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135-1144.
[6] S. M. Lundberg and S. I. Lee, "A Unified Approach to Interpreting Model Predictions," in Advances in Neural Information Processing Systems, 2017, pp. 4765-4774.
[7] T. Brown et al., "Language Models are Few-Shot Learners," in Advances in Neural Information Processing Systems, 2020, pp. 1877-1901.
[8] M. Zhang et al., "The limits of fair medical imaging AI in real-world generalization," Nature Medicine, vol. 30, no. 2, pp. 389-399, Feb. 2024.
[9] B. H. Misheva, "Explainable AI in Credit Risk Management," arXiv preprint arXiv:2103.00949, Mar. 2021.
[10] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso, "Machine Learning Interpretability: A Survey on Methods and Metrics," Electronics, vol. 8, no. 8, p. 832, 2019.