Building Trustworthy AI for Critical Systems

Picture the scene. It is the day you set off on holiday. At the airport, you walk across the tarmac hand in hand with your children towards the aircraft waiting ahead, engines already running. By the boarding stairs, you notice a sign reading "AI inside". Artificial intelligence on board. Do you still get on the plane? This perfectly legitimate question is precisely the one the white paper "End-to-end Trustworthy AI Engineering Lifecycle" seeks to answer.

According to Fateh Kaakai, a researcher in trustworthy AI at Thales cortAIx Labs and co-author of the study, the answer is reassuring: “You can board the aircraft because a framework is in place that guarantees the safe integration of AI with the required level of security.”

Anticipating the AI revolution in critical systems

Thales was among the first major industrial players to anticipate the profound transformation that artificial intelligence would bring to the design of its products and solutions. Through its cortAIx initiative, and as part of the multi-stakeholder confiance.ai project, the Group has been working for many years on integrating AI into critical systems - systems in which even the slightest failure can have serious consequences.

This long-term effort has consistently taken account of the regulatory environment, including the AI Act, which came into force in the European Union in 2024. The legislation bans certain high-risk uses of AI and significantly strengthens transparency requirements.

Trust as a prerequisite for deployment

These sustained efforts have led to a clear and firmly held conviction: trust in AI is a prerequisite for its large-scale deployment, especially in critical systems, where its use must contend with a wide range of constraints. These include the inherent complexity of such systems; extremely demanding requirements in terms of reliability and availability; the need to embed AI in components with strict limitations on size, weight and power consumption; connectivity between subsystems that may be intermittent, low-bandwidth or even contested through jamming; and, finally, the need to address ethical considerations.

When the Group brings an AI-enabled predictive maintenance tool to market for the aviation sector, it must offer its customers the highest possible guarantees of reliability and transparency. As Juliette Mattioli, an AI expert at Thales, explains: “A trustworthy product does what it is supposed to do, everything it is supposed to do, and nothing more.”

“When AI based on machine learning is integrated,” she continues, “the system’s output can vary depending on context, which is a first major challenge. The second is transparency and explainability. We need to put precise methods in place so that the system’s behaviour can be understood not only by the user, but also by the designer who debugs it and the auditor who certifies it. We must be able to provide evidence, simply saying that it works is not enough.”

Towards robust and explainable industrial processes

Establishing a genuine "chain of trus" is therefore the central challenge facing industrial players who integrate AI into critical systems. Whether it is pilot assistance in fighter aircraft, dialysis equipment in hospitals or cybersecurity systems, reliability must be beyond reproach.

This is why Thales has worked relentlessly to establish industrial processes that are robust, dependable, explainable and compliant with regulations at national, European and global levels. The article "End-to-end Trustworthy AI Engineering Lifecycle” represents the culmination of work begun as early as 2019, aimed at establishing AI engineering as a discipline in its own right, governed by specific rules and methodologies.

“The methods and tools we are proposing challenge a significant proportion of current practices in software engineering, algorithm engineering and systems engineering,” warns Juliette Mattioli. “This is something genuinely new, made necessary by the fact that we are embedding artificial intelligence into our systems.”

Enhancing existing processes for effective AI integration

What, in practical terms, are the changes proposed by Thales? The first step is to identify which system functions can legitimately be developed using AI, taking into account both safety considerations and operational benefits.

“In a commercial aircraft, for example, flight control systems do not incorporate AI,” Fateh Kaakai points out. The European Union Aviation Safety Agency (EASA) does not currently envisage authorising AI for the most critical functions. “That said, many other functions are suitable for AI integration, such as decision support, airworthiness, object detection, or survivability in constrained environments. Identifying which functions are genuinely compatible with AI is a first layer of safety, and it requires a deep understanding of the limitations of these technologies.”

Once the relevant functions have been identified, the second challenge is to bring rigour to engineering processes while fully accounting for existing regulations and standards. “This is particularly true in aviation, which has benefited from decades of adjustments, experimentation and standards development,” Fateh Kaakai adds. “What we are contributing today is the missing piece needed to ensure that the limitations of AI technologies are properly addressed and no longer pose any major risk to systems or their users.”

A new W-shaped development lifecycle

This additional safeguard can be summed up in a single letter, one that represents a small revolution: W. For decades, algorithm engineering has followed a well-established V-shaped development cycle, encompassing all stages from requirements capture and specification through design, implementation, integration, verification, validation, deployment and maintenance.

Thales is now developing a W-shaped cycle. The first "V" focuses on the design of reliable algorithms with built-in trust guarantees. The second "V" covers their implementation and subsequent integration (software and/or hardware) on target platforms. "Put like that, it sounds fairly straightforward," notes Kaakai, "but it has far-reaching implications. This first V involves a wide range of activities, dedicated toolchains and highly specific skills."

All of this unfolds against a backdrop of ever-increasing complexity in AI-based algorithms, a trend that is only set to accelerate.

Skills at the heart of the transformation

Implementing these changes places skills firmly at the centre of the equation. The W-shaped engineering process calls for a new approach to professional roles, reflecting the emergence in recent years of an entire new generation of profiles in higher education: data scientists, data engineers, machine learning engineers, and more.

These new algorithm engineering skills must be integrated into multidisciplinary teams alongside more traditional roles (systems, software and hardware engineering) as well as specialised expertise in areas such as functional safety, cybersecurity and human factors.

The lifecycle for trustworthy AI engineering, as set out in “End-to-end Trustworthy AI Engineering Lifecycle”, provides a comprehensive framework for addressing the challenges inherent in AI technologies. Through this work, Thales reaffirms its commitment to developing AI-based systems that are not only effective, safe and secure, but also responsible, transparent and ethical.