OCTOBER 2024
DO AI SYSTEMS COMPLY WITH THE AI ACT?
SUMMARY.
The EU’s AI Act establishes a comprehensive framework for AI governance. It categorises systems by risk level to ensure safety and transparency. High-impact models, such as General-Purpose AI, face specific requirements for documentation and cybersecurity. However, concerns remain about whether the Act sufficiently addresses risks from generative AI, prompting discussions on enforcement.
WHAT IS THE AI ACT?
Just as a reminder, a regulation is a binding legislative act issued by the European Union that applies directly to all member stateswithout needing national laws to implement it. It ensures uniformity across the Union by setting consistent standards and rules that members must follow immediately upon its adoption. Specifically, the Regulation EU 2024/1689, called AI Act, is the first-ever comprehensive legal framework on AI worldwide.
Its objective is to promote trustworthy AI both within Europe and internationally, ensuring that AI systems uphold fundamental rights, safety. Meanwhile, it aims to address the risks posed by highly capable and influential AI models.
The AI Act is the result of months of discussions and work, as the topic is highly controversial. It was published in the Official Journal of the European Union on 12 July 2024 and will enter into force in stages.
It excludes sectors outside EU jurisdiction and it does not interfere with member states’ national security powers. Additionally, it does not apply to AI systems used purely for military, defence, research, innovation, or personal, non-professional purposes.
The adopted framework is risk-based, categorising AI systems into four distinct risk levels according to their potential societal impact:
UNACCEPTABLE RISK
AI systems in this category will be banned within the EU due to their potential dangers. This includes systems that can manipulate human behaviour, social scoring that ranks individuals based on personal traits, and biometric identification in public spaces (although this has a restricted scope).
*CHAPTER II, ART. 5
HIGH RISK
This category covers AI systems that may endanger individual safety or fundamental rights. It includes applications in critical areas like transport, healthcare, and education. Examples include automated student assessments or robotic-assisted surgery. Before placing a high-risk AI system on the market or putting it into service within the EU, companies must conduct a preliminary meet an extensive list of requirements to ensure the system’s safety. As a practical measure, the regulation also requires the European Commission to establish and maintain a public-access database where providers must submit information about their high-risk AI systems, ensuring transparency for all stakeholders.
*CHAPTER III, ART. 6
LIMITED RISK
AI systems that don’t fall under the high-risk category are generally allowed. However, in cases where certain uses or interactions with end users are involved, the AI Regulation recognises a limited risk. This can and should be mitigated by meeting basic transparency requirements. For instance, ensuring users are informed they are interacting with an AI, enabling them to make responsible choices based on the AI’s information.
*ART. 50
MINIMAL OR NO RISK
AI systems that do not fall within the specified risk categories of the AI Act are not subject to particular restrictions and may be used without limitation. Such applications are now widespread and represent most of the AI systems we encounter daily. Examples include spam detection tools, AI-enabled video games, and stock management systems.
AND WHAT ABOUT GENERATIVE AI?
The AI Act defines an “AI system” as a machine-based system capable of operating autonomously, adapting after deployment, and generating outputs (like predictions or decisions) that can influence physical or digital spaces.
While it does not define “AI models” in general, it specifically defines General-Purpose AI (GPAI) model. A GPAI model (commonly known as foundation model) is «an AI model, including when trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable to competently perform a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications».
*ART. 3(44B)
Their broad scope of application, economic and social impact, has brought about challenges related to safety, privacy, and ethics, prompting numerous debates.
Specifically, the AI Act requires the providers of these models to maintain up-to-date documentation on: a general description of the model (parameters, input and output methodologies, etc.), architecture and strategies adopted for training, the sources of the dataset employed for training and the bias mitigation techniques adopted, and lastly, information on computational resources and energy consumption.
*ART. 53
In addition, when they pose a systemic risk, i.e. when they exhibit high-impact capabilities, assessed through indicators and benchmarks such as the computing power needed for training being greater than 1025 flops (floating point operations per second), or when deemed as such by the European Commission ex officio, they are subject to more stringent obligations: not only their inclusion in a public database, but also the implementation of advanced cybersecurity measures (such as adversarial testing).
*ART. 51
Despite marking an important step toward regulating artificial intelligence, the AI Act has been met with criticism, particularly regarding its approach to generative AI. By generally classifying generative AI as limited risk, it applies only minimal transparency requirements, which some argue do not sufficiently address the risks of misinformation and manipulation posed by these systems.
Furthermore, there are concerns that foundational models, capable of broad societal influence, may evade rigorous oversight. Effective enforcement remains a question, with doubts about whether the European AI Office will have adequate resources and adaptability for rapidly advancing technologies.
To integrate the Act in response to these concerns, in summer 2024, the European AI Office announced a call for academics, providers and different stakeholders to participate in the drafting of the first General-Purpose AI Code of Practice. The drafting process will be iterative and will be completed by April 2025. After publication, the AI Office and the AI Board will assess the appropriateness of the Code and the Commission may decide to approve it, giving it general validity in the Union through an implementing act.
HOW WELL ARE WE DOING?
The AI Act is a remarkable regulatory effort and prompting worldwide discussions regarding the importance of regulating artificial intelligence. However, a study conducted in 2023 (when its draft was not yet final) by Stanford University showed how most of the generative AI models used by millions of people on a daily basis, such as GPT-4, Stable Diffusion v2, and LLaMA, are in fact nowhere close to meeting the transparency requirements of the AI Act(Bommasani et al., 2023)
The study revealed how the majority of these models’ providers fail to disclose information particularly in areas like data sourcing, energy reporting, and risk mitigation. Furthermore, they do not implement efficient strategies to mitigate biases and address the ethical impact of their products.
Interestingly, open-source providers often perform better in terms of transparency (despite being subject to less stringent regulation under the AI Act). On the other hand, providers of closed models tend to excel in controlling deployment.
Although that study predated the Act’s adoption, its findings remain relevant, with new evidence corroborating these challenges.
In 2024, ETH Zurich and INSAIT introduced COMPL-AI, a framework developed to help companies gauge their compliance with the AI Act.
By evaluating models across areas such as cybersecurity, environmental impact, and data governance, the tool has highlighted ongoing shortcomings among major providers. Many models performed reasonably well in controlling harmful content but showed weak results in reducing bias and strengthening cybersecurity (Guldimann et al., 2024).
FUTURE PERSPECTIVES
These weaknesses highlight the need for a balanced focus in AI development, moving beyond pure performance and accuracy to meet ethical and regulatory requirements. COMPL-AI’s benchmarks indicate that these essential aspects are not yet sufficiently prioritised in today’s generative AI models.
The EU AI Act’s regulatory demands call for specific standards and well-defined technical deliverables. However, the lack of precise guidelines poses challenges in aligning AI systems with legal expectations.
In addition, many of the Act’s requirements are underexplored in AI research, especially in areas such as traceability, corrigibility, and resilience against cyberattacks. Existing benchmarks often fall short of capturing these regulatory aspects.
The EU regulation is therefore likely to prompt a shift in research priorities, encouraging the creation of new benchmarks and assessment tools to ensure that LLMs meet standards for safety, transparency, and ethics.
REFERENCES.
Bommasani, R., Klyman, K., Zhang, D., & Liang, P. Do foundation model providers comply with the eu ai act?, 2023. URL https://crfm.stanford.edu/2023/06/15/eu-ai-act.html.
Guldimann, P., Spiridonov, A., Staab, R., Jovanović, N., Vero, M., Vechev, V., … & Vechev, M. (2024). COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act. arXiv preprint arXiv:2410.07959.
Regulation EU 2024/1689.