A customer service chatbot is considered high-risk under the EU AI Act if it makes or supports decisions that have significant implications for people’s rights or their access to essential services. This does not apply to most standard FAQ bots, but it does apply to chatbots that, for example, assess creditworthiness, process benefits, or handle emergency calls. In this article, we answer the most frequently asked questions about when a customer service AI chatbot falls into the high-risk category and what that means for your organization.
What criteria determine whether an AI system is high-risk?
An AI system is considered high-risk under the AI Act if it falls under one of the eight use cases listed in Annex III, or if it constitutes a safety component of a regulated product that requires a third-party conformity assessment. The Act provides two independent pathways to high-risk classification, and it is sufficient for a system to meet the criteria for either one.
The eight areas listed in Annex III are specifically defined and include: biometrics, critical infrastructure, education and vocational training, employment and human resources management, access to essential services (such as creditworthiness, insurance, and emergency calls), law enforcement, migration and border control, and the administration of justice and democratic processes.
There is, however, one exception: an Annex III system that performs only a narrow procedural or preparatory task and does not pose a significant risk to fundamental rights may fall outside the high-risk category. However, the provider must then document this with supporting evidence. Furthermore, there is a strict rule: systems that perform profiling of natural persons are always high-risk, without exception.
In practical terms, this means that the question “Is this system high-risk?” can never be answered purely from a technical standpoint. It’s about the intended use and the potential impact on people, not the technology itself.
Does a customer service chatbot fall under a high-risk category under the AI Act?
Most customer service chatbots do not automatically fall into a high-risk category. A chatbot that answers frequently asked questions, provides business hours, or registers a callback request typically has minimal impact on fundamental rights and falls outside the eight Annex III domains. Such systems are considered low-risk AI.
However, the classification changes as soon as a chatbot performs tasks that touch on the protected areas listed in Annex III. A chatbot at an insurance company that helps assess whether someone is eligible for a policy, or a bot at a municipality that determines whether a citizen is entitled to benefits, operates in the domain of access to essential services. That immediately places it in the high-risk category.
Another relevant factor is whether the chatbot performs profiling. If the system uses conversation data or user behavior to build a profile that influences future decisions, then the system is always high-risk, regardless of the sector in which it is used.
What makes a chatbot in the public sector high-risk?
A chatbot in the public sector is considered high-risk when it plays a role in decisions regarding access to government services, benefits, permits, or other citizens’ rights. Government agencies operate in areas that the AI Act explicitly designates as high-risk, which means the threshold for classification is lower than in commercial environments.
Consider a chatbot at a municipal office that handles questions about social assistance applications, or a bot at an implementing agency that guides citizens through appeal procedures. Even if the chatbot does not make a final decision itself but instead filters information or determines the path leading to a decision, it can be considered part of a high-risk AI system.
An additional risk in the public sector is the combination of scale and vulnerability. Government agencies serve large numbers of citizens, including people in financially or socially vulnerable situations. The AI Act places particular emphasis on that combination of reach and impact in its risk assessment.
How does a high-risk chatbot differ from a low-risk chatbot?
The key difference between a high-risk and a minimal-risk chatbot lies in the impact of the output on people. A minimal-risk chatbot provides information, answers questions, or facilitates a conversation without the outcome having any direct consequences for a person’s rights, opportunities, or access to services. A high-risk chatbot influences or supports decisions that do have such consequences.
In practical terms, you can explain the distinction as follows:
- Low-risk: a chatbot that provides product information, answers FAQs, schedules an appointment, or logs a complaint
- High-risk: a chatbot that helps determine whether someone is eligible for a loan, insurance, or benefits; that handles emergency calls; or that performs profiling based on conversation data
An important point to note: the classification is not static. A chatbot deployed today as low-risk may become high-risk as soon as its functionality is expanded or its intended purpose changes. The AI Act explicitly states that a deployer or third party becomes a provider—with all the associated obligations—when they modify the intended purpose of a system in such a way that it becomes high-risk.
What requirements apply to high-risk AI in customer service?
High-risk AI systems used in customer service are subject to extensive requirements regarding documentation, oversight, transparency, and risk management. Most of these requirements for Annex III systems will take effect on August 2, 2026, but preparations are necessary now.
The key obligations for providers of high-risk AI include, among other things:
- Establish and maintain a robust risk management system throughout the system’s entire lifecycle
- Prepare technical documentation demonstrating that the system complies with the requirements of the AI Act
- Ensure data management, including quality control of training data
- Build in logging and traceability so that the system can be audited later
- Enabling human oversight: A person must always be able to override or shut down the system
- Ensure adequate levels of accuracy, robustness, and cybersecurity
- Registration in the EU database for high-risk AI systems
Deployers (organizations that use a high-risk AI system developed by another party) are subject to additional obligations: conducting a fundamental rights impact assessment before the system is put into use, informing employees about how the system works, and monitoring its performance in practice.
When does a transparency requirement apply to chatbots?
A transparency requirement applies to virtually every chatbot that interacts with people, regardless of whether the system is high-risk. The AI Act stipulates that users must be informed when they are interacting with an AI system, unless this is already clear from the context. This is a basic requirement that is already in effect and does not depend on a high-risk classification.
In practical terms, this means that a chatbot that mimics a human must always disclose that it is an AI system. This also applies to voice-activated systems and to bots that communicate on behalf of an organization via WhatsApp, chat, or email. The disclosure must be clear and understandable, not hidden in the fine print.
For high-risk chatbots, the transparency requirements are more extensive. Deployers must inform users that they are interacting with a high-risk AI system and, in certain cases, also explain the logic the system uses. When a chatbot supports a decision that has significant consequences for a person, that person has the right to a meaningful explanation.
A practical consideration: the transparency requirement also applies to internal applications. If your employees are supported by an AI system that evaluates their performance or assigns their tasks, that is also subject to the disclosure requirement under the AI Act.
How Pegamento Helps with AI Compliance in Customer Service
Navigating the AI Act is complex, especially if you also want to run a customer service operation that actually works. At Pegamento, we understand that challenge. We help organizations implement smart, AI-driven solutions that are not only effective but also comply with the requirements of the AI Act.
What we can do for you:
- Mapping Risk Classification: Together, we’ll analyze which AI applications in your customer service fall under which risk categories
- Building in transparency: We ensure that chatbots and AI assistants meet the basic requirements for transparency and human oversight
- Documentation and logging: We set up systems with the proper traceability and audit trails you need to ensure compliance
- Customized solutions using standard building blocks: no costly custom work, but a smart combination of proven modules that you can deploy quickly and easily
- Everything under one roof: from consulting and implementation to management and support—a single point of contact for the complete package
Our AI assistants are built on the principle of Agentic AI: an evolution from executive bots to self-thinking assistants that not only follow instructions but also take the initiative and act independently. That makes them powerful—and requires the right governance. Would you like to know where your organization stands and how to use AI responsibly in customer service? Contact us, and we’d be happy to help you figure it out.
Frequently Asked Questions
How do I know if my existing chatbot needs to be re-evaluated after a feature expansion?
As soon as you expand a chatbot’s functionality—for example, by linking it to a decision-making module or by having it build customer profiles—you must re-evaluate its risk classification. The AI Act explicitly states that a change in the intended purpose may result in a new classification, in which case the deployer becomes the provider and assumes all associated obligations. Practical advice: For every significant update, document the intended purpose and assess it against the Annex III domains before deploying the change.
What is the difference between a provider and a deployer under the AI Act, and why does that matter for my organization?
A provider is the party that develops and markets the AI system, while a deployer is the organization that puts the system into use for a specific purpose. This distinction is crucial because deployers have their own obligations, such as conducting a fundamental rights impact assessment before putting the system into use and monitoring its performance in practice. Furthermore, a deployer can automatically assume the role of provider—with all the more stringent compliance requirements—if it modifies the intended purpose of a purchased system in such a way that it becomes high-risk.
Should I take action now, or can I wait until the obligations for high-risk AI take effect in August 2026?
Waiting until August 2026 is a risk, not a strategy. The preparation time for high-risk compliance—such as setting up a risk management system, configuring logging, and drafting technical documentation—is substantial and can easily take several months. Furthermore, the transparency requirement for chatbots already applies: users must be informed today that they are communicating with an AI system. So start now by conducting a risk classification of your current AI applications.
What exactly is a Fundamental Rights Impact Assessment, and how do I conduct one?
A Fundamental Rights Impact Assessment (FRIA) is a structured analysis in which you identify which fundamental rights of individuals may be affected by your high-risk AI system, the extent of that impact, and the measures you are taking to mitigate risks. The assessment must be conducted before you deploy a high-risk system and must be documented. Specifically, you examine issues such as non-discrimination, privacy, access to legal remedies, and the protection of vulnerable groups—and you document how the system affects each of these aspects.
Does the transparency requirement also apply if my chatbot is clearly labeled as a 'virtual assistant' on the website?
A label such as 'virtual assistant' or 'chatbot' in the interface may be sufficient if it is unmistakably clear to the average user that they are communicating with an AI system. The AI Act states that the disclosure is not required when this 'is evident from the context.' However, caution is advised: the determination of whether something is ‘evident’ rests with the regulator, not with you. An explicit, understandable statement at the beginning of the conversation—such as “You are speaking with an AI assistant”—is the safest approach and prevents disputes.
What are the most common mistakes made when classifying a customer service chatbot under the AI Act?
The most common mistake is reasoning based on the technology rather than the use case: organizations conclude that their chatbot is ‘just a language model’ and therefore poses minimal risk, even though the actual application does indeed fall within an Annex III domain. A second common mistake is ignoring indirect influence: a chatbot that does not make decisions itself but filters information or determines the routing that leads to a decision can still be classified as part of a high-risk system. Finally, many organizations underestimate the profiling criterion—as soon as a system builds a user profile based on conversation data, it is always high-risk.
How do I handle a chatbot that performs both low-risk and high-risk tasks within a single platform?
If a single chatbot both answers simple FAQ questions and performs tasks that touch on Annex III domains, the strictest classification applies to the entire system. You can resolve this by architecturally separating the high-risk functionality into a separate system with its own governance, logging, and documentation, while the low-risk component is managed more lightly. This requires a deliberate design choice and clear functional boundaries between the two components—something best documented in the technical documentation that the AI Act already requires for high-risk systems.
