The AI Act sets out clear rules for automated decision-making in customer service: AI systems that make decisions that directly affect people are, in many cases, subject to the regulation’s transparency and oversight requirements. Whether a system is classified as high-risk depends on the nature of the decision and the domain in which it is used. In this article, we answer the most frequently asked questions about what the AI Act means for your customer service organization. For more background on how AI works in customer interactions, visit our page on AI-driven intelligence.
Which AI systems used in customer service are covered by the AI Act?
Most AI systems used in customer service do not automatically fall under the strict “high-risk” category of the AI Act. Chatbots, virtual assistants, and automated routing tools are generally classified as low-risk systems, subject to less stringent transparency requirements. Systems that profile customers or make decisions regarding access to essential services are always considered high-risk.
The AI Act distinguishes four risk levels: prohibited applications, high-risk AI, low-risk AI, and AI with minimal or no risk. For customer service, the distinction between the last three categories is the most relevant.
Low-risk applications in customer service
Chatbots and virtual agents that provide information or guidance to customers generally fall under the low-risk category. The most important requirement here is transparency: customers must know that they are communicating with an AI system, not a human. This also applies to automated email responses and AI-powered knowledge bases.
High-Risk AI in Customer Service
A system is considered high-risk as soon as it profiles natural persons, or when it makes decisions regarding access to essential services such as insurance, credit, or emergency services. Consider, for example, an AI system that automatically determines whether a customer is eligible for a payment plan, or that categorizes customers based on behavioral patterns. Such systems fall under Annex III of the AI Act and are subject to strict obligations.
What are the obligations regarding high-risk automated decisions?
In the case of high-risk automated decision-making, extensive obligations apply to both the system provider and the organization that deploys it. The provider must establish a risk management system, maintain technical documentation, and ensure reliable, representative training data. As the organization using the system, you are required to assign human oversight, retain logs for at least six months, and inform employees before the system is put into use.
In practical terms, this means the following for deployers—the parties that use a high-risk AI system under their own authority:
- Use the system only in accordance with the provider’s instructions for use
- Assign human supervision to qualified and trained employees
- Retain automatic logs for at least six months
- Informing employees before the system is put into use (Article 26(7))
- Conduct a data protection impact assessment (DPIA) where applicable
Furthermore, under Article 86, customers who are subject to a decision made by a high-risk system have the right to request an explanation of the factors that determined that decision. This right to an explanation is new compared to existing legislation and requires specific adjustments to your processes and communication.
How does the AI Act relate to the GDPR with regard to customer data?
The AI Act and the GDPR complement each other, but they overlap in key areas. While the GDPR grants rights regarding the processing of personal data, the AI Act regulates the security and transparency of the AI systems themselves. In the case of automated decision-making involving customer data, both laws apply simultaneously, which means you must comply with the requirements of both.
Article 22 of the GDPR already grants individuals the right not to be subject solely to automated decision-making that produces legal effects or otherwise significantly affects them. The AI Act builds on this by imposing additional requirements on the systems that make such decisions, regardless of whether personal data is involved.
In practice, this means that when using AI for customer contact, you must always ask two questions: Does the system process personal data (GDPR), and does it make decisions that directly affect customers (AI Act)? If the answer to both questions is yes, you must also conduct a DPIA. Both supervisory authorities—the Dutch Data Protection Authority and the national AI regulator—have the power to enforce compliance.
When is human oversight required for AI decisions?
Human oversight is required for all high-risk AI systems. This means that a trained employee must always be able to understand how the system works, intervene, and correct or cancel decisions. This requirement is not merely a formality: the system itself must be designed in such a way that effective oversight is technically feasible.
The AI Act explicitly warns against automation bias—the tendency of people to blindly accept AI decisions. Providers must design their systems in such a way that users remain aware of the system’s limitations. For customer service organizations, this means that employees must have not only the authority but also the knowledge and tools to critically evaluate AI decisions.
For low-risk systems, such as a chatbot that provides information, formal human oversight is not required by law. However, even in these cases, it is wise to establish escalation procedures so that customers can always reach a representative if the AI is not sufficient.
What do organizations need to do to be AI Act-compliant?
To be AI Act-compliant, your organization must first identify which AI systems you use and which risk category they fall into. Next, for each system, you must determine which obligations apply and who is responsible for compliance. Most obligations for high-risk Annex III systems take effect on August 2, 2026, but preparations take time.
A practical approach involves the following steps:
- AI Assessment: Identify all AI applications used in your customer service, from chatbots to routing algorithms and scoring models.
- Risk Classification: Determine for each system whether it is high-risk, limited-risk, or minimal-risk based on the criteria in the AI Act.
- Division of Responsibilities: Determine who is the provider and who is the deployer, as the obligations differ by role.
- Transparency with customers: Make sure customers know when they are interacting with AI and how they can reach a human representative.
- Establishing Human Oversight: Assign trained staff to monitor and adjust high-risk systems.
- Keep Documentation in Order: Retain logs, user manuals, and reviews in accordance with legal retention requirements.
- AI literacy: Train employees to understand AI decisions, a requirement that will take effect on February 2, 2025.
For organizations that have not yet systematically addressed AI compliance, the AI Act provides a good opportunity to formalize this process. Fines for non-compliance can reach up to 15 million euros or 3% of global annual revenue, depending on the violation.
How Pegamento Helps Ensure AI Act Compliance in Customer Service
We understand that the combination of the AI Act and automated decision-making raises many questions, especially if you use multiple systems that don’t communicate well with each other. At Pegamento, we help you deploy AI in customer service in a responsible and effective way. Our Agentic AI for customer service solutions are built on a foundation of transparency and human oversight—exactly what the AI Act requires.
What we can do for you:
- Risk classification of your current AI applications, so you know where you stand and which steps to prioritize
- Customized solutions using standard building blocks, without costly customization, that meet the transparency and oversight requirements of the AI Act
- Agentic AI assistants that not only follow instructions but also take the initiative on their own, and are designed with built-in escalation paths to human employees. This represents the evolution from traditional RPA bots to self-thinking assistants that operate within clear parameters.
- Everything under one roof: from consulting and implementation to management and support, without having to coordinate multiple vendors
- ISO 27001-certified information security, supplemented by ISO 9001 and ISO 26000, as the foundation for a compliance-proof infrastructure
Would you like to know how your organization is doing in terms of AI Act compliance? Contact us, and we’d be happy to help you figure it out.
Frequently Asked Questions
Geldt de AI Act ook voor kleine en middelgrote bedrijven in klantenservice?
Ja, de AI Act is van toepassing op alle organisaties die AI-systemen inzetten binnen de EU, ongeacht hun omvang. Wel biedt de verordening enige ruimte voor mkb-bedrijven en start-ups: zij komen in aanmerking voor vereenvoudigde documentatieverplichtingen en kunnen gebruikmaken van regulatoire sandboxes om systemen te testen. Toch gelden de kernverplichtingen rondom transparantie, menselijk toezicht en AI-geletterdheid ook voor kleinere organisaties.
Wat is het verschil tussen een 'aanbieder' en een 'deployer' onder de AI Act, en hoe weet ik welke rol mijn organisatie heeft?
Een aanbieder (provider) is de partij die een AI-systeem ontwikkelt en op de markt brengt, terwijl een deployer de organisatie is die het systeem onder eigen verantwoordelijkheid inzet voor een specifiek doel. Als je een kant-en-klare AI-oplossing van een leverancier gebruikt in jouw klantenservice, ben je doorgaans deployer. Als je een eigen AI-systeem bouwt of een bestaand systeem ingrijpend aanpast, kun je ook als aanbieder worden aangemerkt — met bijbehorende zwaardere verplichtingen.
Hoe leg ik klanten uit dat een beslissing door AI is genomen, zonder technisch jargon te gebruiken?
Het recht op uitlegbaarheid (Artikel 86) vereist geen technische uitleg, maar een begrijpelijke toelichting op de bepalende factoren achter een beslissing. In de praktijk kun je dit vertalen naar heldere klanttaal, zoals: ‘Uw aanvraag is afgewezen op basis van uw betalingsgeschiedenis en het openstaande saldo.’ Zorg dat je klantenservicemedewerkers zijn getraind om deze uitleg mondeling te geven, en overweeg een gestandaardiseerde schriftelijke verklaring aan te bieden voor klanten die hier formeel om verzoeken.
Wat zijn de meest voorkomende fouten die organisaties maken bij het voorbereiden op AI Act-compliance?
Een veelgemaakte fout is het onderschatten van de scope: organisaties denken dat alleen grote, zichtbare AI-systemen onder de wet vallen, maar ook routeringsalgoritmen en geautomatiseerde scoringsmodellen kunnen hoog-risico zijn. Daarnaast wordt AI-geletterdheid van medewerkers vaak als laatste prioriteit behandeld, terwijl dit al verplicht is per 2 februari 2025. Tot slot vergeten veel organisaties de verantwoordelijkheidsverdeling tussen aanbieder en deployer contractueel vast te leggen met hun leveranciers.
Moet ik bestaande AI-systemen die al in gebruik zijn ook aanpassen aan de AI Act?
Ja, ook bestaande systemen moeten uiteindelijk voldoen aan de AI Act. Voor hoog-risico Annex III-systemen geldt een overgangsperiode tot 2 augustus 2026, maar de verplichting rondom AI-geletterdheid is al van kracht. Het is verstandig om nu al een inventarisatie te starten en prioriteit te geven aan systemen die beslissingen nemen over toegang tot diensten of klantprofilering uitvoeren, omdat de aanpassingen voor die categorie het meest ingrijpend zijn.
Hoe richt ik escalatiepaden in zodat klanten altijd bij een menselijke medewerker terecht kunnen?
Een effectief escalatiepad begint met een duidelijke trigger: wanneer een AI-systeem een klacht niet kan oplossen, een klant expliciet om een medewerker vraagt, of wanneer een beslissing financiële of juridische gevolgen heeft, moet de overdracht automatisch plaatsvinden. Zorg dat de medewerker die het gesprek overneemt direct beschikt over de volledige gesprekshistorie en de door de AI gegenereerde informatie, zodat de klant zijn verhaal niet opnieuw hoeft te doen. Test deze paden regelmatig en documenteer de uitkomsten als onderdeel van je toezichtsprotocol.
Hoe verhoudt de AI Act-verplichting rondom AI-geletterdheid zich tot bestaande trainingsverplichtingen voor medewerkers?
De AI-geletterdheidsplicht (Artikel 4) is een aanvulling op bestaande trainingsverplichtingen en vereist dat medewerkers die met AI-systemen werken voldoende kennis hebben om de werking, beperkingen en risico’s van die systemen te begrijpen. Dit gaat verder dan een eenmalige introductie: het niveau van training moet afgestemd zijn op de complexiteit van het systeem en de rol van de medewerker. Voor hoog-risico systemen betekent dit dat toezichthoudende medewerkers aantoonbaar bekwaam moeten zijn, wat je het beste vastlegt in een intern trainingsregister.
