AI applications in customer service are considered high-risk when they are used to make decisions that have a significant impact on fundamental rights, access to essential services, or people’s legal status. Examples include automated systems that assess creditworthiness, handle emergency calls, or create customer profiles based on personal characteristics. The EU AI Act imposes specific obligations in this regard that are becoming increasingly relevant for customer service organizations. This article answers the most frequently asked questions about high-risk AI in the context of customer service.
What officially classifies an AI system as “high-risk” under the AI Act?
An AI system is officially considered high-risk under the EU AI Act if it falls within one of the eight domains listed in Annex III of the regulation, or if it constitutes a safety component of a product that is already subject to EU harmonization legislation and requires a third-party conformity assessment. Profiling of natural persons is always high-risk, without exception.
The eight areas listed in Annex III are specifically defined in the law. These include biometrics, critical infrastructure, education and vocational training, employment and human resources management, access to essential services (such as creditworthiness, insurance, and emergency calls), law enforcement, migration and border control, and the administration of justice and democratic processes.
There is one exception to the Annex III classification: a system that performs only a narrow procedural or preparatory task and does not pose a significant risk to fundamental rights may fall outside the high-risk category. However, the provider must provide substantiated documentation to support this. As soon as a system profiles individuals, that possibility is completely ruled out.
Which specific customer service AI applications are considered high-risk?
In the context of customer service, AI applications are considered high-risk when they support or make decisions regarding access to essential services, create personal profiles, or handle emergency calls. Not every chatbot or routing tool is automatically high-risk, but the line is closer to everyday practice than many organizations realize.
Specific examples of customer service AI that are classified as high-risk:
- Automated credit assessment: AI that determines whether a customer is eligible for a payment plan, credit, or financial product is explicitly covered by Annex III.
- Insurance-related decisions: Systems that analyze customer data to assess insurance applications or calculate premiums are high-risk.
- Handling of emergency calls: AI that routes, prioritizes, or assesses emergency calls falls under the category of access to essential services.
- Customer profiling for personalized offers: As soon as a system analyzes customer behavior to create individual profiles, it is, by definition, high-risk due to the profiling provision.
- AI in Recruitment Through Customer Service: Systems that evaluate applicants or employees based on customer service interactions fall under the categories of employment and human resources management.
A standard FAQ chatbot that answers general questions without creating user profiles generally falls outside the high-risk category. But as soon as such a chatbot combines customer data to make personalized decisions, its classification changes.
What is the difference between high-risk AI and prohibited AI in customer service?
Prohibited AI applications are completely banned under the AI Act and may not be deployed, regardless of the context or precautionary measures. High-risk AI is not prohibited, but may only be deployed if strict compliance requirements are met. The distinction is fundamental: prohibited AI is a hard line, while high-risk AI is a regulated category.
In the context of customer service, prohibited uses include: AI systems that use subliminal techniques to manipulate customer behavior without their knowledge; systems that exploit the vulnerabilities of specific groups (such as the elderly or people with disabilities); and real-time biometric identification in public spaces for law enforcement purposes.
High-risk AI in customer service is permitted, but requires, among other things, a conformity assessment, technical documentation, human oversight, and registration in the EU database. The bans have been in effect since February 2, 2025. Anyone still using prohibited AI applications now risks fines of up to 35 million euros or 7% of global annual revenue.
What obligations apply to organizations that use high-risk AI in customer service?
Organizations that use high-risk AI in customer service must comply with a comprehensive set of obligations, depending on their role in the chain. The AI Act distinguishes between providers (who develop the system), deployers (who implement the system), and importers or distributors. Each role entails its own set of responsibilities.
Requirements for Providers of High-Risk AI
Suppliers must conduct a conformity assessment, prepare and maintain technical documentation, affix the CE marking, and register the system in the EU database. In addition, they are required to establish a quality management system and ensure that human supervision is technically feasible.
Requirements for Deployers of High-Risk AI
Organizations that deploy high-risk AI in their customer service (deployers) must follow the provider’s instructions, ensure human oversight, document data management practices, and conduct a fundamental rights impact assessment for certain applications. Important note: a deployer may inadvertently become a provider if they substantially modify the system or alter its intended purpose in such a way that it becomes high-risk. In that case, all provider obligations apply.
In practical terms, this means that customer service organizations must already begin compiling a registry of all AI systems and determine their role in them. Special attention must be paid to circumstances that could inadvertently turn a party into a provider.
When will the high-risk AI regulations take effect for customer service organizations?
The compliance requirements for high-risk AI systems covered by Annex III will become enforceable on August 2, 2026. That is the key date for most customer service AI applications. The prohibitions and the AI literacy requirement have been in effect since February 2, 2025.
The complete timeline is as follows:
- February 2, 2025: Prohibited AI practices (Article 5) and the requirement for AI literacy (Article 4) take effect.
- August 2, 2025: Requirements for GPAI models, notified bodies, governance structures, and penalty provisions take effect.
- August 2, 2026: Compliance requirements for high-risk Annex III systems become enforceable. This is the most relevant date for customer service AI.
- August 2, 2027: Requirements for high-risk AI used as a safety component in regulated products (Annex I) take effect. GPAI models that were already on the market before August 2025 must also comply by that date.
So by 2026, there will be no time left to sit back and relax. Organizations that have not yet taken stock of their AI systems run the risk of being ill-prepared when enforcement begins.
How do you determine whether your current customer service AI tools are high-risk?
To determine whether your customer service AI tools are high-risk, ask three key questions for each system: Does the system create profiles of individuals? Does it influence decisions regarding access to essential services? And does its use fall under one of the eight Annex III domains? If you answer “yes” to any of these questions, the system is likely high-risk.
A practical approach to assessing this:
- Take inventory of all AI systems: Create a registry of every system that uses AI, including chatbots, routing tools, speech recognition, and analytics software.
- Determine your role: Are you a supplier, deployer, importer, or distributor? This determines which obligations apply to you.
- Analyze the intended purpose: What does the system actually do? Does it evaluate customers, create profiles, or make decisions that affect customers?
- Check for profiling: Any system that creates individual customer profiles based on behavior, preferences, or personal characteristics is automatically considered high-risk.
- Request documentation from suppliers: Providers of high-risk AI are required to provide documentation regarding capabilities, limitations, and compliance status.
Are you unsure about the classification of a specific system? The law allows providers to classify an Annex III system outside the high-risk category if it performs only a narrow procedural task and does not pose a significant risk to fundamental rights. However, this requires substantiated documentation and does not apply to systems that perform profiling.
How Pegamento Helps Ensure AI Compliance in Customer Service
Navigating the AI Act is a significant challenge for many customer service organizations, especially when existing systems have been in operation for years and the question is how they align with the new regulations. We help organizations map out their AI landscape and set up responsible, compliant customer service solutions.
What we specifically offer:
- AI Systems Inventory: We help you create a comprehensive overview of all AI applications in your customer service environment and determine the risk classification for each system.
- Agentic AI for Customer Service: Our Agentic AI solutions for customer service are designed with human oversight as a core principle. Agentic AI represents the evolution from task-oriented bots to self-thinking assistants that not only follow instructions but also take the initiative and act independently, within the parameters you set.
- Everything under one roof: From development and implementation to management and support. No complex supplier structures—just a single point of contact for the complete package.
- Customized solutions using standard building blocks: No costly custom work, but a smart combination of proven modules that fit your specific situation and compliance requirements.
- ISO 27001 certified: Information security is our top priority, complemented by ISO 9001 and ISO 26000 certifications that ensure quality and social responsibility.
Would you like to know how your current customer service AI tools measure up against the high-risk criteria of the AI Act? Contact us, and we’d be happy to work with you to develop an approach that suits your organization.
Frequently Asked Questions
What happens if my organization accidentally becomes a provider of high-risk AI?
If, as a deployer, you substantially modify a high-risk AI system or change its intended purpose in a way that makes it high-risk, you automatically fall under the provider obligations. This means you must still conduct a conformity assessment, prepare technical documentation, and register the system in the EU database. It is therefore essential to assess in advance, whenever you modify an AI system, whether that change alters your role in the chain.
How do I conduct a fundamental rights impact assessment for my customer service AI?
You conduct a Fundamental Rights Impact Assessment step by step: identify which fundamental rights the system may affect, analyze the risks for each affected group (such as vulnerable customers), and describe the measures you are taking to mitigate those risks. Deployers of high-risk AI are required to conduct this assessment before putting the system into use. For more information, consult the guidelines from the European AI Office, which publishes practical templates and guidance.
What does the AI literacy requirement mean in practice for my customer service team?
As of February 2, 2025, organizations are required to ensure that employees working with AI systems have sufficient knowledge of how those systems work, their limitations, and the risks they pose. For customer service teams, this means in practice that agents working with AI-supported tools must be trained to recognize AI-generated output, exercise human oversight, and identify anomalous system behavior. Document these training sessions and their content as evidence of compliance.
Can a standard CRM system with AI features also be classified as high-risk?
Yes, that is possible. If a CRM system includes AI features that create customer profiles based on behavior or personal characteristics, or if it makes recommendations that affect access to services, it may fall under the high-risk category—even if it is primarily a CRM. The classification does not depend on the type of software, but on what the system actually does. Ask your CRM vendor explicitly which AI features are active and whether they can provide compliance documentation.
What documentation should I start preparing now, even though the deadline isn’t until August 2026?
Start immediately by creating an AI registry in which you record each system along with its intended purpose, the data used, your role in the chain (provider or deployer), and a preliminary risk classification. In addition, record the instructions and documentation from your AI suppliers, and document how human oversight is organized. The sooner you build this registry, the less work you’ll have as the deadline approaches—and the stronger your position will be if regulators ask questions early on.
What are the most common mistakes organizations make when assessing their AI risk classification?
The most common mistake is assuming that a system is safe because it functions ‘merely as a support tool’ or does not make final decisions. Under the AI Act, even decision support is considered high-risk if it significantly influences the outcome. A second common mistake is overlooking indirect profiling: systems that, on the surface, only segment or personalize data still create individual profiles in practice. Always assess based on what the system actually does, not on how the vendor positions it.
How do I handle high-risk AI from a third-party vendor that does not yet have compliance documentation?
If your vendor cannot provide compliance documentation, that is a serious red flag. As the deployer, you are jointly responsible for compliance and cannot simply rely on the statement that ‘the vendor will take care of it.’ Issue a written notice of default to your supplier and request a concrete timeline for achieving compliance. In the meantime, consider limiting the system’s use to non-high-risk applications, or begin a selection process for an alternative that is demonstrably compliant by August 2026.
