The AI Act sets out clear rules for automated decision-making in customer service: AI systems that make decisions that directly affect people are, in many cases, subject to the regulation’s transparency and oversight requirements. Whether a system is classified as high-risk depends on the nature of the decision and the domain in which it is used. In this article, we answer the most frequently asked questions about what the AI Act means for your customer service organization. For more background on how AI works in customer interactions, visit our page on AI-driven intelligence.
Which AI systems used in customer service are covered by the AI Act?
Most AI systems used in customer service do not automatically fall under the strict “high-risk” category of the AI Act. Chatbots, virtual assistants, and automated routing tools are generally classified as low-risk systems, subject to less stringent transparency requirements. Systems that profile customers or make decisions regarding access to essential services are always considered high-risk.
The AI Act distinguishes four risk levels: prohibited applications, high-risk AI, low-risk AI, and AI with minimal or no risk. For customer service, the distinction between the last three categories is the most relevant.
Low-risk applications in customer service
Chatbots and virtual agents that provide information or guidance to customers generally fall under the low-risk category. The most important requirement here is transparency: customers must know that they are communicating with an AI system, not a human. This also applies to automated email responses and AI-powered knowledge bases.
High-Risk AI in Customer Service
A system is considered high-risk as soon as it profiles natural persons, or when it makes decisions regarding access to essential services such as insurance, credit, or emergency services. Consider, for example, an AI system that automatically determines whether a customer is eligible for a payment plan, or that categorizes customers based on behavioral patterns. Such systems fall under Annex III of the AI Act and are subject to strict obligations.
What are the obligations regarding high-risk automated decisions?
In the case of high-risk automated decision-making, extensive obligations apply to both the system provider and the organization that deploys it. The provider must establish a risk management system, maintain technical documentation, and ensure reliable, representative training data. As the organization using the system, you are required to assign human oversight, retain logs for at least six months, and inform employees before the system is put into use.
In practical terms, this means the following for deployers—the parties that use a high-risk AI system under their own authority:
- Use the system only in accordance with the provider’s instructions for use
- Assign human supervision to qualified and trained employees
- Retain automatic logs for at least six months
- Informing employees before the system is put into use (Article 26(7))
- Conduct a data protection impact assessment (DPIA) where applicable
Furthermore, under Article 86, customers who are subject to a decision made by a high-risk system have the right to request an explanation of the factors that determined that decision. This right to an explanation is new compared to existing legislation and requires specific adjustments to your processes and communication.
How does the AI Act relate to the GDPR with regard to customer data?
The AI Act and the GDPR complement each other, but they overlap in key areas. While the GDPR grants rights regarding the processing of personal data, the AI Act regulates the security and transparency of the AI systems themselves. In the case of automated decision-making involving customer data, both laws apply simultaneously, which means you must comply with the requirements of both.
Article 22 of the GDPR already grants individuals the right not to be subject solely to automated decision-making that produces legal effects or otherwise significantly affects them. The AI Act builds on this by imposing additional requirements on the systems that make such decisions, regardless of whether personal data is involved.
In practice, this means that when using AI for customer contact, you must always ask two questions: Does the system process personal data (GDPR), and does it make decisions that directly affect customers (AI Act)? If the answer to both questions is yes, you must also conduct a DPIA. Both supervisory authorities—the Dutch Data Protection Authority and the national AI regulator—have the power to enforce compliance.
When is human oversight required for AI decisions?
Human oversight is required for all high-risk AI systems. This means that a trained employee must always be able to understand how the system works, intervene, and correct or cancel decisions. This requirement is not merely a formality: the system itself must be designed in such a way that effective oversight is technically feasible.
The AI Act explicitly warns against automation bias—the tendency of people to blindly accept AI decisions. Providers must design their systems in such a way that users remain aware of the system’s limitations. For customer service organizations, this means that employees must have not only the authority but also the knowledge and tools to critically evaluate AI decisions.
For low-risk systems, such as a chatbot that provides information, formal human oversight is not required by law. However, even in these cases, it is wise to establish escalation procedures so that customers can always reach a representative if the AI is not sufficient.
What do organizations need to do to be AI Act-compliant?
To be AI Act-compliant, your organization must first identify which AI systems you use and which risk category they fall into. Next, for each system, you must determine which obligations apply and who is responsible for compliance. Most obligations for high-risk Annex III systems take effect on August 2, 2026, but preparations take time.
A practical approach involves the following steps:
- AI Assessment: Identify all AI applications used in your customer service, from chatbots to routing algorithms and scoring models.
- Risk Classification: Determine for each system whether it is high-risk, limited-risk, or minimal-risk based on the criteria in the AI Act.
- Division of Responsibilities: Determine who is the provider and who is the deployer, as the obligations differ by role.
- Transparency with customers: Make sure customers know when they are interacting with AI and how they can reach a human representative.
- Establishing Human Oversight: Assign trained staff to monitor and adjust high-risk systems.
- Keep Documentation in Order: Retain logs, user manuals, and reviews in accordance with legal retention requirements.
- AI literacy: Train employees to understand AI decisions, a requirement that will take effect on February 2, 2025.
For organizations that have not yet systematically addressed AI compliance, the AI Act provides a good opportunity to formalize this process. Fines for non-compliance can reach up to 15 million euros or 3% of global annual revenue, depending on the violation.
How Pegamento Helps Ensure AI Act Compliance in Customer Service
We understand that the combination of the AI Act and automated decision-making raises many questions, especially if you use multiple systems that don’t communicate well with each other. At Pegamento, we help you deploy AI in customer service in a responsible and effective way. Our Agentic AI for customer service solutions are built on a foundation of transparency and human oversight—exactly what the AI Act requires.
What we can do for you:
- Risk classification of your current AI applications, so you know where you stand and which steps to prioritize
- Customized solutions using standard building blocks, without costly customization, that meet the transparency and oversight requirements of the AI Act
- Agentic AI assistants that not only follow instructions but also take the initiative on their own, and are designed with built-in escalation paths to human employees. This represents the evolution from traditional RPA bots to self-thinking assistants that operate within clear parameters.
- Everything under one roof: from consulting and implementation to management and support, without having to coordinate multiple vendors
- ISO 27001-certified information security, supplemented by ISO 9001 and ISO 26000, as the foundation for a compliance-proof infrastructure
Would you like to know how your organization is doing in terms of AI Act compliance? Contact us, and we’d be happy to help you figure it out.
Frequently Asked Questions
Does the AI Act also apply to small and medium-sized businesses in customer service?
Yes, the AI Act applies to all organizations that use AI systems within the EU, regardless of their size. However, the regulation does provide some leeway for SMEs and startups: they are eligible for simplified documentation requirements and can use regulatory sandboxes to test systems. Nevertheless, the core obligations regarding transparency, human oversight, and AI literacy also apply to smaller organizations.
What is the difference between a 'provider' and a 'deployer' under the AI Act, and how do I know which role my organization plays?
A provider is the party that develops and markets an AI system, while a deployer is the organization that deploys the system under its own responsibility for a specific purpose. If you use a ready-made AI solution from a vendor in your customer service, you are generally considered a deployer. If you build your own AI system or make significant modifications to an existing system, you may also be classified as a provider—which comes with more stringent obligations.
How do I explain to customers that a decision was made by AI without using technical jargon?
The right to explainability (Article 86) does not require a technical explanation, but rather a clear explanation of the factors that determined the decision. In practice, you can translate this into clear, customer-friendly language, such as: ‘Your application was denied based on your payment history and outstanding balance.’ Ensure your customer service representatives are trained to provide this explanation verbally, and consider offering a standardized written statement for customers who formally request one.
What are the most common mistakes organizations make when preparing for AI Act compliance?
A common mistake is underestimating the scope: organizations believe that only large, visible AI systems fall under the law, but routing algorithms and automated scoring models can also be high-risk. In addition, AI literacy among employees is often treated as the lowest priority, even though it will be mandatory as of February 2, 2025. Finally, many organizations forget to contractually define the division of responsibility between the provider and the deployer with their suppliers.
Do I also need to adapt existing AI systems that are already in use to comply with the AI Act?
Yes, existing systems must also ultimately comply with the AI Act. For high-risk Annex III systems, a transition period applies until August 2, 2026, but the requirement regarding AI literacy is already in effect. It is wise to start an assessment now and prioritize systems that make decisions regarding access to services or perform customer profiling, as the adjustments for that category are the most significant.
How do I set up escalation paths so that customers can always reach a human agent?
An effective escalation path starts with a clear trigger: when an AI system cannot resolve a complaint, a customer explicitly requests a human agent, or when a decision has financial or legal consequences, the handoff must occur automatically. Ensure that the agent taking over the conversation immediately has access to the full conversation history and the information generated by the AI, so the customer doesn’t have to repeat their story. Test these workflows regularly and document the results as part of your monitoring protocol.
How does the AI Act’s requirement regarding AI literacy relate to existing employee training requirements?
The AI literacy requirement (Article 4) supplements existing training obligations and requires employees who work with AI systems to have sufficient knowledge to understand the functioning, limitations, and risks of those systems. This goes beyond a one-time orientation: the level of training must be tailored to the complexity of the system and the employee’s role. For high-risk systems, this means that supervisory staff must be demonstrably competent, which is best documented in an internal training log.
