Agentic AI poses specific risks in customer service related to autonomous decision-making, a lack of transparency, and potential violations of customer rights. The EU AI Act requires organizations that use Agentic AI in customer interactions to systematically manage these risks, ensure human oversight, and, in certain cases, maintain comprehensive compliance documentation. In this article, we answer the most frequently asked questions about Agentic AI in customer service and explain exactly what the AI Act requires of your organization.
What specific risks does Agentic AI pose in customer service?
Agentic AI in customer service carries risks that go beyond those of traditional chatbots or automated menu systems. Because Agentic AI makes decisions independently, takes action, and can act on its own without employee intervention, this increases the likelihood of errors that directly impact customers.
The main risks are:
- Autonomous decisions without context: An AI agent handling a complaint or processing a request may sometimes miss the nuances that a human would pick up on, such as when dealing with a vulnerable customer or in an emotionally charged situation.
- Lack of transparency: Customers do not always know that they are communicating with an AI system, which undermines trust and raises legal concerns.
- Profiling and Discrimination: If Agentic AI analyzes customer behavior to determine priority or treatment, there is a risk that certain groups will be systematically disadvantaged.
- Data Quality and Error Propagation: An autonomous agent that acts based on outdated or incorrect information may make false promises or mislead customers.
- Loss of human oversight: The more tasks Agentic AI performs autonomously, the greater the likelihood that employees will lose track of what is being communicated on behalf of the organization.
These risks are not hypothetical. In sectors such as government, healthcare, and housing authorities—where customer contact has a direct impact on people—they carry even greater weight.
How does the AI Act classify agentic AI in customer service?
The AI Act defines four risk levels: unacceptable risk (prohibited), high risk (strictly regulated), limited risk (minimal transparency requirements), and minimal risk (unregulated). Agentic AI in customer service generally falls into the limited- or high-risk category, depending on what the system actually does.
Low-risk applications, such as an AI chatbot that answers questions or provides information, are subject to transparency requirements. The customer must be aware that he or she is communicating with an AI system. That sounds simple, but in practice, organizations often fail to implement this.
A high-risk classification is imminent as soon as Agentic AI:
- Performing profiling of individual customers (always considered high risk under the AI Act);
- Makes decisions regarding access to essential services, such as whether or not to transfer a call to emergency services or to assess a customer request with financial implications;
- Is used in sectors covered by Annex III of the AI Act, including government and public services.
Prohibited practices also apply in full to customer service. Agentic AI may not use manipulative techniques that influence behavior without the customer’s awareness, nor may it exploit vulnerable groups based on age, disability, or socioeconomic status. These prohibitions have been in effect since February 2, 2025.
What requirements apply to high-risk AI systems used in customer interactions?
If your Agentic AI system is classified as high-risk, extensive obligations will apply starting August 2, 2026. Organizations acting as deployers—that is, those that deploy a third-party AI system—also bear a share of this responsibility.
The core obligations for high-risk AI are:
- Risk Management System: An ongoing process that identifies, evaluates, and manages risks throughout the entire life cycle of the system.
- Data quality requirements: Training and operational data must be relevant, representative, and as free of errors as possible.
- Technical documentation: A comprehensive description of the system, its capabilities, limitations, and test results.
- Logging and Traceability: The system must automatically maintain logs so that the decisions it has made can be reconstructed later.
- Transparency for users: Customers and employees must understand how the system works and what decisions it makes.
- Human oversight: There must be mechanisms in place that allow people to monitor, correct, and, if necessary, shut down the system.
- Fundamental Rights Impact Assessment: For certain applications, a formal assessment of the impact on fundamental rights is required.
As a deployer, you are also required to provide adequate training to employees who work with the AI system and to report serious incidents to the competent authority.
How do you ensure human oversight of autonomous AI agents?
You can ensure human oversight at Agentic AI by intentionally building in intervention points, defining clear escalation paths, and actively involving employees in monitoring AI behavior. The AI Act explicitly requires this for high-risk systems, but it is also a best practice for lower-risk applications.
In practice, this means the following:
- Define the limits of autonomy: Determine which actions the Agentic AI is permitted to perform independently and in which situations a human must always be involved, such as complaints about service, financial decisions, or emotionally charged conversations.
- Build in escalation mechanisms: Ensure that the AI agent can detect on its own when a situation is beyond its scope and seamlessly hand it off to a staff member, including the full conversation history.
- Actively monitor for anomalies: Use dashboards and logging to see if the AI agent is behaving as expected. Unexpected patterns, such as a sudden spike in escalations or low customer satisfaction scores, are early warning signs of problems.
- Train employees in AI literacy: The AI Act requires organizations to promote AI literacy starting February 2, 2025. Employees must understand how the AI agent works, what its limitations are, and how they can intervene.
Human oversight is not an afterthought, but a structural component of a responsible Agentic AI system. An AI agent that operates completely autonomously without any form of human oversight does not meet the requirements of the AI Act.
What are the consequences of non-compliance with the AI Act?
Non-compliance with the AI Act can result in substantial fines, reputational damage, and the shutdown of AI systems. The fine structure is tiered and will take full effect on August 2, 2025.
The three stages of penance are:
- Violation of prohibited practices (Article 5): Up to 35 million euros or 7% of global annual revenue, whichever is greater.
- Non-compliance with other obligations: Up to 15 million euros or 3% of global annual revenue.
- Inaccurate or misleading information provided to authorities: Up to 7.5 million euros or 1% of global annual revenue.
For medium-sized organizations, the lower of the percentage or the fixed amount applies, which somewhat limits the financial impact. But even aside from fines, the consequences of non-compliance are very real: regulators can order AI systems to be shut down, customers can take legal action, and reputational damage in sectors such as government and healthcare is difficult to repair. In January 2026, Finland became the first member state to grant formal enforcement powers to its national authority, a sign that enforcement is becoming a reality.
What steps are you taking to implement Agentic AI in compliance with the AI Act?
In accordance with the AI Act, you implement Agentic AI in customer service by starting with a clear risk classification, followed by establishing the required governance, documentation, and oversight mechanisms. A structured approach prevents you from having to fix later on what went wrong during the initial setup.
Follow these steps:
- Map out its usage: What exactly does Agentic AI do? What decisions does it make? What data does it use? This forms the basis for risk classification.
- Determine the risk category: Does the system fall under “limited risk” or “high risk”? Does it perform profiling? Does it impact access to essential services? Document your reasoning.
- Prepare technical documentation: Describe the system, how it works, its limitations, test results, and the measures you have taken to manage risks.
- Set up logging: Ensure that the system automatically tracks which decisions it makes, when, based on what input, and with what result.
- Define human oversight: Specify who is responsible for monitoring, how the escalation process works, and how to shut down the system if necessary.
- Train your employees: Foster AI literacy throughout the organization. Employees who work with the AI agent need to understand how the system works and what their role is.
- Establish transparent communication: Always inform customers that they are communicating with an AI system, especially in low-risk applications.
- Schedule periodic reviews: The AI Act requires ongoing risk management. Schedule regular times to review the system and update the documentation.
How Pegamento Helps Implement Responsible Agentic AI in Customer Service
We at Pegamento understand that the combination of Agentic AI and regulations such as the AI Act raises many questions. Especially for organizations already struggling with fragmented systems, staff shortages, and increasing customer volumes, it’s tempting to postpone addressing these risks. But that’s exactly where things go wrong.
Our AI-driven intelligence was built from the ground up with human oversight as a guiding principle. What we offer:
- Risk Classification as a Starting Point: We’ll help you determine which category your use of Agentic AI falls into and what that means in practical terms for your obligations.
- Built-in escalation routes: Our Agentic AI assistants are designed to seamlessly hand off conversations to agents, including the full context of the conversation, so customers don’t have to repeat themselves.
- Transparency by design: Customers always know when they are communicating with an AI agent, in accordance with the transparency requirements of the AI Act.
- Logging and reporting: All interactions are logged and can be viewed via central dashboards, so you can always account for what the system has done.
- Everything under one roof: From implementation to management and compliance support, you have a single point of contact instead of multiple vendors that aren’t coordinated.
- Built on proven modules: No costly custom development, but a smart combination of proven building blocks that we tailor to your situation and industry.
We are ISO 27001 certified (information security), supplemented by ISO 9001 and ISO 26000, which means that compliance and quality are structurally embedded in our operations—not something you have to add as an afterthought. Would you like to know how your organization can use Agentic AI responsibly and in compliance with the AI Act? Contact us, and we’d be happy to work with you to find a solution.
Frequently Asked Questions
Geldt de AI Act ook als wij een Agentic AI-systeem van een externe leverancier afnemen en niet zelf ontwikkelen?
Ja, ook als deployer — de partij die een AI-systeem van een derde inzet — draag je verplichtingen onder de AI Act. Je bent verantwoordelijk voor het correct gebruik van het systeem, het trainen van medewerkers en het melden van ernstige incidenten. Het is dus essentieel om bij je leverancier te controleren welke technische documentatie en conformiteitsverklaringen beschikbaar zijn, en contractueel vast te leggen wie welke verantwoordelijkheden draagt.
Hoe weet ik of mijn huidige Agentic AI-systeem al voldoet aan de transparantieverplichtingen van de AI Act?
Controleer minimaal of klanten op elk moment duidelijk geïnformeerd worden dat ze met een AI-systeem communiceren — dit geldt al vanaf de eerste interactie en ook bij geautomatiseerde e-mails of chatberichten. Kijk ook of het systeem klanten de mogelijkheid biedt om naar een menselijke medewerker over te stappen. Als dit niet actief is ingericht, is er sprake van non-compliance met de transparantieverplichtingen die al van kracht zijn.
Wat is het verschil tussen een AI-chatbot en Agentic AI, en maakt dat uit voor de AI Act-classificatie?
Een traditionele chatbot volgt vaste scripts en geeft vooraf bepaalde antwoorden, terwijl Agentic AI zelfstandig beslissingen neemt, acties uitvoert in systemen en initiatief kan nemen. Dit verschil is cruciaal voor de AI Act: hoe meer autonomie en impact op de klant, hoe groter de kans op een hoog-risico classificatie. Een chatbot die alleen informatie verstrekt valt doorgaans onder beperkt risico, maar een AI-agent die klantverzoeken verwerkt, dossiers bijwerkt of prioriteiten toekent kan al snel als hoog risico worden aangemerkt.
Wat moet ik doen als mijn Agentic AI-systeem een fout maakt die een klant heeft benadeeld?
Documenteer het incident direct en volledig via de ingerichte logging, en onderzoek via de audittrail welke beslissing het systeem heeft genomen en op basis van welke input. Bij hoog-risico systemen ben je verplicht ernstige incidenten te melden aan de bevoegde nationale autoriteit. Zorg daarnaast voor een herstelproces richting de getroffen klant en evalueer of aanpassingen aan het systeem of de autonomiegrenzen noodzakelijk zijn om herhaling te voorkomen.
Hoe vaak moet ik mijn risicoclassificatie en documentatie herzien na de initiële implementatie?
De AI Act vereist doorlopend risicomanagement, wat betekent dat je documentatie actueel moet blijven bij elke significante wijziging van het systeem, de gebruikte data of de toepassingscontext. Als praktische richtlijn hanteren veel organisaties minimaal één formele evaluatie per kwartaal, aangevuld met ad-hoc reviews bij systeemwijzigingen, nieuwe use cases of opvallende afwijkingen in de monitoringdata. Plan deze evaluatiemomenten vast in je governance-kalender zodat compliance geen eenmalige activiteit wordt maar een structureel proces.
Zijn er uitzonderingen of verlichte eisen voor kleine organisaties of non-profitinstellingen?
De AI Act kent voor kleine en middelgrote ondernemingen (kmo’s) en start-ups enkele verlichtingen, zoals lagere vaste boetebedragen waarbij het laagste van het percentage of het vaste maximum geldt. Voor non-profitorganisaties bestaan geen structurele uitzonderingen op de inhoudelijke verplichtingen — de risicocategorie van het systeem en de impact op klanten bepalen de eisen, niet de rechtsvorm van de organisatie. Wel kunnen toezichthouders bij de handhaving rekening houden met de omvang en middelen van een organisatie.
Hoe ga ik om met Agentic AI in klantenservice als mijn organisatie actief is in meerdere EU-landen?
De AI Act is een EU-verordening die rechtstreeks van toepassing is in alle lidstaten, dus de kernverplichtingen zijn overal gelijk. Toch kunnen nationale toezichthouders — zoals de eerder genoemde Finse autoriteit — eigen handhavingsprioriteiten en meldprocedures hanteren. Zorg dat je weet welke nationale autoriteit bevoegd is per land waar je opereert, stem je incidentmeldingsprocedures hierop af en houd rekening met eventuele aanvullende sectorwetgeving, zoals in de zorg of financiële dienstverlening, die per lidstaat kan verschillen.
