To ensure that an Agentic AI solution complies with the transparency requirements of the AI Act, your organization must demonstrate that the system is traceable, documentable, and verifiable. This applies in particular if your Agentic AI application is classified as high-risk under the regulation. In this article, we answer the most frequently asked questions about transparency, documentation, and human oversight at Agentic AI in light of the AI Act.
What transparency requirements does the AI Act impose on high-risk AI systems?
The AI Act requires providers and users of high-risk AI systems to ensure transparency at four levels: technical documentation, logging of system behavior, understandable information for users, and demonstrable human oversight. These requirements are intended to enable accountability and prevent arbitrary or uncontrollable decisions.
Specifically, this means that a high-risk AI system must meet the following transparency requirements:
- Technical documentation: a complete set of documents for the system, including its design, operation, training data, and known limitations.
- Automatic logging: The system must log events so that it is possible to verify later which decisions were made and on what basis.
- User Information: People who work with the system or are affected by it must understand that they are dealing with AI and what the system does.
- Accountability: The logic behind decisions must be transparent to regulators and stakeholders.
High-risk AI is defined in Article 6 of the AI Act and includes, among other things, systems used in eight specific domains, including access to essential services, employment and workforce management, and the administration of justice. Systems that perform profiling of natural persons are always considered high-risk, regardless of the context. Most of the obligations for high-risk Annex III systems will take effect on August 2, 2026.
What makes Agentic AI different from regular AI under the AI Act?
Agentic AI differs from conventional AI in that the system takes initiative on its own, links decisions sequentially, and carries out actions without human approval at every step. This autonomous nature makes the application more complex to assess under the AI Act, because the chain of responsibility is less direct than in a single-decision model.
Whereas a traditional AI model processes a single input and produces a single output, an Agentic AI system can go through a series of steps: retrieving information, making a decision, performing an action, evaluating the result, and then initiating a new step. This makes it more difficult to determine exactly which step led to which result.
This is relevant to the AI Act for two reasons. First, greater autonomy increases the potential risk: the more the system acts independently, the greater the impact of an error or an undesirable outcome. Second, the multi-step operation requires logging and documentation at the level of individual actions, not just at the level of the final result. If your Agentic AI solution makes decisions that directly affect people, there’s a good chance it will be classified as high-risk.
What documentation is required for an Agentic AI solution?
For an Agentic AI solution classified as high-risk, comprehensive technical documentation is required. This documentation must be up to date, accessible to regulators, and sufficiently detailed to fully understand the system’s operation, limitations, and risks.
The required documentation must include, at a minimum:
- System Description: A detailed description of the system’s purpose, operation, and limitations.
- Risk management system: a documented analysis of risks and the measures taken to manage them.
- Training Data and Data Quality: Information about what data were used, how they were selected, and what quality checks were performed.
- Performance metrics: measurable indicators that demonstrate the system’s accuracy and reliability.
- Change History: An overview of substantial changes to the system, including an assessment of whether those changes alter the risk classification.
Additional obligations apply to General-Purpose AI (GPAI) models: providers must prepare technical documentation in accordance with Annex XI, inform downstream providers about capabilities and limitations in accordance with Annex XII, and make a summary of the training data used publicly available. If your Agentic AI solution is built on a GPAI model, these obligations also apply to the underlying layer of the system.
How do you explain what led to an Agentic AI decision?
Explainability in Agentic AI requires a combination of technical logging, structured reasoning traces, and understandable reporting. The goal is to enable you to reconstruct, after the fact, which input, which step, and which weighting led to a specific decision or action.
In practice, this means that you need to organize explainability at three levels:
- Action Level: Every action taken by the system is logged with a timestamp, context, and the reason for that action.
- Decision-making level: The reasoning behind a decision is documented, including which alternatives were considered and why a particular course of action was chosen.
- Outcome level: The final result is linked to the chain of actions and decisions that preceded it, ensuring that a complete audit trail is available.
This poses a technical challenge for systems that use large language models or neural networks, because the internal workings of these models are not always immediately interpretable. A practical approach is to incorporate structured interim reports into the workflow of the Agentic AI system, so that each step is explicitly documented in understandable language. Regulators and stakeholders do not need to understand the underlying model architecture, but they do need to be able to understand why a decision was relevant to them.
When is human oversight required for Agentic AI?
Human oversight is required for all high-risk AI systems, including Agentic AI applications that fall into that category. The AI Act requires providers and users to take measures to ensure that humans can effectively monitor, correct, and, if necessary, shut down the system during operation.
The law distinguishes between two forms of oversight. In the case of oversight during use, people must be able to recognize abnormal behavior and intervene before harm occurs. Post-use oversight involves the ability to review decisions and correct their consequences. Both forms are relevant to Agentic AI, precisely because the system independently carries out multiple steps.
In practice, this means that you build explicit checkpoints into your Agentic AI solution: moments when a human assesses the progress before the system continues. This is particularly required when the system makes decisions that are irreversible or have direct consequences for people, such as denying a service, initiating a payment, or modifying customer data. The greater the autonomy and the impact, the more frequent and thorough human oversight must be.
How can you verify that your Agentic AI solution complies with the AI Act?
You can verify whether your Agentic AI solution complies with the AI Act by first determining the risk classification, then identifying the obligations for that class, and finally systematically assessing whether your system and organization meet each requirement. This is an ongoing process, not a one-time check.
A practical approach consists of the following steps:
- Classify the system: determine whether your Agentic AI solution is high-risk based on Article 6 and Annex III. Also check whether the system performs profiling of individuals, as that is always considered high-risk.
- Identify the obligations: For each obligation, determine whether you are classified as a provider, deployer, or importer. Anyone who offers the system under their own name or makes substantial modifications to it assumes all the obligations of a provider.
- Audit the documentation: verify that the technical documentation is complete, up-to-date, and accessible to regulatory authorities.
- Test the logging: Verify that the system automatically records sufficient data to allow decisions to be reconstructed retrospectively.
- Evaluate the oversight mechanism: assess whether the built-in checkpoints for human oversight work in practice and whether employees know how to intervene.
- Repeat in case of changes: Any substantial modification to the system requires a new assessment of the risk classification and compliance.
Organizations that are not yet fully compliant can, in the meantime, adopt a Code of Practice to establish a presumption of compliance until harmonized standards become available. This provides a structured path toward full compliance without having to wait for final standards.
How Pegamento Helps Agentic AI Achieve AI Act Compliance
Complying with the transparency requirements of the AI Act is not a mere administrative formality, but an integral part of building and managing a responsible Agentic AI solution. At Pegamento, we understand this challenge because we don’t view Agentic AI as a standalone product, but rather as an evolution from task automation to self-thinking assistants that take the initiative and act independently. What used to be called RPA has evolved into Agentic AI: systems that not only follow instructions but also reason, prioritize, and act.
Our approach is built on proven modules that you can combine without the need for costly customization, and that were designed from the outset with auditability and transparency in mind. Specifically, we offer:
- Built-in audit trails so that every decision and action taken by the system is traceable.
- Structured checkpoints for human oversight at the moments that really matter.
- Documentation support that meets the requirements of the AI Act, including technical dossiers and risk assessments.
- A single point of contact for development, implementation, management, and compliance, so you don’t have to manage a complex supplier structure.
- ISO 27001-certified security as the foundation, supplemented by ISO 9001 and ISO 26000, to ensure information security and quality.
Would you like to know how your current or planned Agentic AI solution measures up against the AI Act requirements? Contact us, and we’ll work with you to determine the best approach for your situation.
Frequently Asked Questions
Does the AI Act also apply to Agentic AI solutions that we use internally and do not offer to third parties?
Yes, the AI Act distinguishes between providers (who develop or market the system) and deployers (who put the system into use). Even if you use an Agentic AI solution exclusively internally, as a deployer you are subject to obligations such as human oversight, logging, and compliance with the provider’s instructions. Furthermore, if you have developed the system yourself or substantially modified it, all obligations of a provider apply, including technical documentation and conformity assessment.
What are the most common mistakes made when setting up logging for Agentic AI?
The most common mistake is that organizations only log the end result instead of every individual step in the decision-making chain. For Agentic AI, it is essential that intermediate actions, considered alternatives, and the context at the time of decision-making are recorded. A second common mistake is the lack of a retention policy: logs must be retained long enough to support audits and complaint procedures, and the AI Act sets specific requirements for this regarding high-risk systems.
How do I determine whether a modification to my existing Agentic AI system is 'substantial' and therefore requires a new conformity assessment?
A modification is substantial if it significantly affects the intended operation, risk level, or performance of the system. Examples include adding new decision-making capabilities, changing the underlying AI model, expanding into a new application domain, or adjusting the thresholds for automated decisions. When in doubt, it is wise to document the modification and explicitly assess it against the original risk classification; if the outcome changes, a full reassessment is required.
Can we already start building a compliant Agentic AI solution, or do we have to wait until all harmonized standards are available?
You don’t have to wait: the core requirements of the AI Act have already been established, and most obligations for high-risk Annex III systems take effect on August 2, 2026. You can get started right away by determining the risk classification, compiling technical documentation, and setting up logging and monitoring mechanisms based on the current text of the law. As long as harmonized standards are not yet available, you can adhere to a recognized Code of Practice to establish a presumption of compliance.
How do I explain a decision made by my Agentic AI to a customer who doesn’t have a technical background?
The AI Act does not require you to explain the technical workings of the model, but it does require that the data subject understands that an AI system played a role and what the relevant outcome is for them. In practical terms, this means generating structured, human-readable summaries within the workflow that describe, in understandable language, what information was taken into account and what conclusion the system reached. Additionally, always provide a clear point of contact where data subjects can go with questions or objections regarding the decision.
What happens if my Agentic AI solution is built on an external GPAI model, such as a large language model API? Who is then responsible for AI Act compliance?
Responsibility is layered: the provider of the GPAI model is responsible for obligations at the model level, such as technical documentation in accordance with Annex XI and informing downstream providers about capabilities and limitations. If you integrate the GPAI model into a high-risk Agentic AI solution and offer it under your own name or substantially modify it, you assume all of the obligations of a provider for the end product. It is therefore crucial to specify contractually what information the GPAI provider must provide to you so that you can complete your own technical dossier.
How frequently should human oversight checkpoints occur, and who in our organization is authorized to fulfill that oversight role?
The AI Act does not prescribe a fixed frequency, but stipulates that oversight must be effective: the supervisor must be able to recognize anomalous behavior in a timely manner and intervene before harm occurs. The frequency therefore depends on the speed, autonomy, and impact of your specific Agentic AI solution. The person fulfilling the oversight role must have sufficient knowledge of the system to assess anomalies and be authorized to pause or stop the system; this person does not need to be a technical expert, but the role should not be merely a formality without any real ability to intervene.
