FPU2025- Robust and Trustworthy AI Systems: Addressing Vulnerabilities and Points of Failure of AI Systems Deployed in High-Risk Applications

Despite their performances, AI systems are not yet considered as reliable enough to be fully autonomous in complex environments without human supervision. Beyond the classical software vulnerabilities that are inherent to any piece of software, AI systems open up new surfaces of vulnerabilities. Addressing these challenges is pivotal for deploying secure and dependable AI systems, particularly in high-risk scenarios. To this end, this proposal will provide a comprehensive assessment of the reliability and robustness of AI systems in multiple domains, including cybersecurity, healthcare, finance, and agriculture. This project focus on three specific areas to advance robust and trustworthy AI systems in high-risk scenarios: 1/ Vulnerabilities, failures modes and attack surfaces. It will investigate the vulnerabilities, failure modes, and attack surfaces of AI-based systems deployed in high-risk applications, including (1) evasion attacks, (2) poisoning attacks and (3) model extraction and inference attacks. 2/ Risk mitigation strategies and adversarial defenses. It will explore strategies to mitigate the risk of the different failure modes, adversarial threats and minimize the attack surfaces of AI-based systems, including algorithms that prioritize security compliance and potential certification schemes. 3/ Interpretability and explainability. It will develop processes and tools for testing, evaluating, and analyzing AI-based systems.