Agentic AI poses specific risks in customer service related to autonomous decision-making, a lack of transparency, and potential violations of customer rights. The EU AI Act requires organizations that use Agentic AI in customer interactions to systematically manage these risks, ensure human oversight, and, in certain cases, maintain comprehensive compliance documentation. In this article, we answer the most frequently asked questions about Agentic AI in customer service and explain exactly what the AI Act requires of your organization.
What specific risks does Agentic AI pose in customer service?
Agentic AI in customer service carries risks that go beyond those of traditional chatbots or automated menu systems. Because Agentic AI makes decisions independently, takes action, and can act on its own without employee intervention, this increases the likelihood of errors that directly impact customers.
The main risks are:
- Autonomous decisions without context: An AI agent handling a complaint or processing a request may sometimes miss the nuances that a human would pick up on, such as when dealing with a vulnerable customer or in an emotionally charged situation.
- Lack of transparency: Customers do not always know that they are communicating with an AI system, which undermines trust and raises legal concerns.
- Profiling and Discrimination: If Agentic AI analyzes customer behavior to determine priority or treatment, there is a risk that certain groups will be systematically disadvantaged.
- Data Quality and Error Propagation: An autonomous agent that acts based on outdated or incorrect information may make false promises or mislead customers.
- Loss of human oversight: The more tasks Agentic AI performs autonomously, the greater the likelihood that employees will lose track of what is being communicated on behalf of the organization.
These risks are not hypothetical. In sectors such as government, healthcare, and housing authorities—where customer contact has a direct impact on people—they carry even greater weight.
How does the AI Act classify agentic AI in customer service?
The AI Act defines four risk levels: unacceptable risk (prohibited), high risk (strictly regulated), limited risk (minimal transparency requirements), and minimal risk (unregulated). Agentic AI in customer service generally falls into the limited- or high-risk category, depending on what the system actually does.
Low-risk applications, such as an AI chatbot that answers questions or provides information, are subject to transparency requirements. The customer must be aware that he or she is communicating with an AI system. That sounds simple, but in practice, organizations often fail to implement this.
A high-risk classification is imminent as soon as Agentic AI:
- Performing profiling of individual customers (always considered high risk under the AI Act);
- Makes decisions regarding access to essential services, such as whether or not to transfer a call to emergency services or to assess a customer request with financial implications;
- Is used in sectors covered by Annex III of the AI Act, including government and public services.
Prohibited practices also apply in full to customer service. Agentic AI may not use manipulative techniques that influence behavior without the customer’s awareness, nor may it exploit vulnerable groups based on age, disability, or socioeconomic status. These prohibitions have been in effect since February 2, 2025.
What requirements apply to high-risk AI systems used in customer interactions?
If your Agentic AI system is classified as high-risk, extensive obligations will apply starting August 2, 2026. Organizations acting as deployers—that is, those that deploy a third-party AI system—also bear a share of this responsibility.
The core obligations for high-risk AI are:
- Risk Management System: An ongoing process that identifies, evaluates, and manages risks throughout the entire life cycle of the system.
- Data quality requirements: Training and operational data must be relevant, representative, and as free of errors as possible.
- Technical documentation: A comprehensive description of the system, its capabilities, limitations, and test results.
- Logging and Traceability: The system must automatically maintain logs so that the decisions it has made can be reconstructed later.
- Transparency for users: Customers and employees must understand how the system works and what decisions it makes.
- Human oversight: There must be mechanisms in place that allow people to monitor, correct, and, if necessary, shut down the system.
- Fundamental Rights Impact Assessment: For certain applications, a formal assessment of the impact on fundamental rights is required.
As a deployer, you are also required to provide adequate training to employees who work with the AI system and to report serious incidents to the competent authority.
How do you ensure human oversight of autonomous AI agents?
You can ensure human oversight at Agentic AI by intentionally building in intervention points, defining clear escalation paths, and actively involving employees in monitoring AI behavior. The AI Act explicitly requires this for high-risk systems, but it is also a best practice for lower-risk applications.
In practice, this means the following:
- Define the limits of autonomy: Determine which actions the Agentic AI is permitted to perform independently and in which situations a human must always be involved, such as complaints about service, financial decisions, or emotionally charged conversations.
- Build in escalation mechanisms: Ensure that the AI agent can detect on its own when a situation is beyond its scope and seamlessly hand it off to a staff member, including the full conversation history.
- Actively monitor for anomalies: Use dashboards and logging to see if the AI agent is behaving as expected. Unexpected patterns, such as a sudden spike in escalations or low customer satisfaction scores, are early warning signs of problems.
- Train employees in AI literacy: The AI Act requires organizations to promote AI literacy starting February 2, 2025. Employees must understand how the AI agent works, what its limitations are, and how they can intervene.
Human oversight is not an afterthought, but a structural component of a responsible Agentic AI system. An AI agent that operates completely autonomously without any form of human oversight does not meet the requirements of the AI Act.
What are the consequences of non-compliance with the AI Act?
Non-compliance with the AI Act can result in substantial fines, reputational damage, and the shutdown of AI systems. The fine structure is tiered and will take full effect on August 2, 2025.
The three stages of penance are:
- Violation of prohibited practices (Article 5): Up to 35 million euros or 7% of global annual revenue, whichever is greater.
- Non-compliance with other obligations: Up to 15 million euros or 3% of global annual revenue.
- Inaccurate or misleading information provided to authorities: Up to 7.5 million euros or 1% of global annual revenue.
For medium-sized organizations, the lower of the percentage or the fixed amount applies, which somewhat limits the financial impact. But even aside from fines, the consequences of non-compliance are very real: regulators can order AI systems to be shut down, customers can take legal action, and reputational damage in sectors such as government and healthcare is difficult to repair. In January 2026, Finland became the first member state to grant formal enforcement powers to its national authority, a sign that enforcement is becoming a reality.
What steps are you taking to implement Agentic AI in compliance with the AI Act?
In accordance with the AI Act, you implement Agentic AI in customer service by starting with a clear risk classification, followed by establishing the required governance, documentation, and oversight mechanisms. A structured approach prevents you from having to fix later on what went wrong during the initial setup.
Follow these steps:
- Map out its usage: What exactly does Agentic AI do? What decisions does it make? What data does it use? This forms the basis for risk classification.
- Determine the risk category: Does the system fall under “limited risk” or “high risk”? Does it perform profiling? Does it impact access to essential services? Document your reasoning.
- Prepare technical documentation: Describe the system, how it works, its limitations, test results, and the measures you have taken to manage risks.
- Set up logging: Ensure that the system automatically tracks which decisions it makes, when, based on what input, and with what result.
- Define human oversight: Specify who is responsible for monitoring, how the escalation process works, and how to shut down the system if necessary.
- Train your employees: Foster AI literacy throughout the organization. Employees who work with the AI agent need to understand how the system works and what their role is.
- Establish transparent communication: Always inform customers that they are communicating with an AI system, especially in low-risk applications.
- Schedule periodic reviews: The AI Act requires ongoing risk management. Schedule regular times to review the system and update the documentation.
How Pegamento Helps Implement Responsible Agentic AI in Customer Service
We at Pegamento understand that the combination of Agentic AI and regulations such as the AI Act raises many questions. Especially for organizations already struggling with fragmented systems, staff shortages, and increasing customer volumes, it’s tempting to postpone addressing these risks. But that’s exactly where things go wrong.
Our AI-driven intelligence was built from the ground up with human oversight as a guiding principle. What we offer:
- Risk Classification as a Starting Point: We’ll help you determine which category your use of Agentic AI falls into and what that means in practical terms for your obligations.
- Built-in escalation routes: Our Agentic AI assistants are designed to seamlessly hand off conversations to agents, including the full context of the conversation, so customers don’t have to repeat themselves.
- Transparency by design: Customers always know when they are communicating with an AI agent, in accordance with the transparency requirements of the AI Act.
- Logging and reporting: All interactions are logged and can be viewed via central dashboards, so you can always account for what the system has done.
- Everything under one roof: From implementation to management and compliance support, you have a single point of contact instead of multiple vendors that aren’t coordinated.
- Built on proven modules: No costly custom development, but a smart combination of proven building blocks that we tailor to your situation and industry.
We are ISO 27001 certified (information security), supplemented by ISO 9001 and ISO 26000, which means that compliance and quality are structurally embedded in our operations—not something you have to add as an afterthought. Would you like to know how your organization can use Agentic AI responsibly and in compliance with the AI Act? Contact us, and we’d be happy to work with you to find a solution.
Frequently Asked Questions
Does the AI Act apply even if we purchase an Agentic AI system from a third-party vendor rather than developing it ourselves?
Yes, even as a deployer—the party that deploys a third-party AI system—you have obligations under the AI Act. You are responsible for the proper use of the system, training employees, and reporting serious incidents. It is therefore essential to check with your supplier to see what technical documentation and declarations of conformity are available, and to specify in the contract who is responsible for what.
How do I know if my current Agentic AI system already complies with the transparency requirements of the AI Act?
At a minimum, verify that customers are clearly informed at all times that they are communicating with an AI system—this applies from the very first interaction and also to automated emails or chat messages. Also check whether the system offers customers the option to switch to a human agent. If this option is not actively enabled, it constitutes non-compliance with the transparency requirements that are already in effect.
What is the difference between an AI chatbot and Agentic AI, and does that matter for AI Act classification?
A traditional chatbot follows fixed scripts and provides predetermined responses, while Agentic AI makes independent decisions, performs actions within systems, and can take the initiative. This difference is crucial for the AI Act: the greater the autonomy and impact on the customer, the higher the likelihood of a high-risk classification. A chatbot that only provides information is generally classified as low risk, but an AI agent that processes customer requests, updates files, or assigns priorities can quickly be classified as high risk.
What should I do if my Agentic AI system makes a mistake that has harmed a customer?
Document the incident immediately and in full using the designated logging system, and use the audit trail to investigate which decision the system made and based on what input. For high-risk systems, you are required to report serious incidents to the competent national authority. Additionally, ensure a remediation process for the affected customer and evaluate whether adjustments to the system or the autonomy limits are necessary to prevent recurrence.
How often should I review my risk classification and documentation after the initial implementation?
The AI Act requires ongoing risk management, which means your documentation must remain up to date with every significant change to the system, the data used, or the context of use. As a practical guideline, many organizations conduct at least one formal evaluation per quarter, supplemented by ad hoc reviews in response to system changes, new use cases, or notable anomalies in the monitoring data. Schedule these review periods in your governance calendar so that compliance becomes a structured process rather than a one-time activity.
Are there any exceptions or relaxed requirements for small organizations or nonprofit institutions?
The AI Act provides some relief for small and medium-sized enterprises (SMEs) and startups, such as lower fixed fine amounts, with the lower of the percentage or the fixed maximum applying. For nonprofit organizations, there are no structural exceptions to the substantive obligations—the system’s risk category and the impact on customers determine the requirements, not the organization’s legal form. However, regulators may take an organization’s size and resources into account when enforcing the law.
How do I handle Agentic AI in customer service if my organization operates in multiple EU countries?
The AI Act is an EU regulation that applies directly in all member states, so the core obligations are the same everywhere. However, national regulators—such as the Finnish authority mentioned earlier—may have their own enforcement priorities and reporting procedures. Make sure you know which national authority has jurisdiction in each country where you operate, align your incident reporting procedures accordingly, and take into account any additional sector-specific legislation—such as in healthcare or financial services—which may vary by member state.
