{"id":31848,"date":"2026-07-01T08:00:00","date_gmt":"2026-07-01T06:00:00","guid":{"rendered":"https:\/\/pegamento.nl\/niet-gecategoriseerd\/which-customer-service-applications-are-considered-high-risk-ai\/"},"modified":"2026-07-01T10:01:02","modified_gmt":"2026-07-01T08:01:02","slug":"which-customer-service-applications-are-considered-high-risk-ai","status":"publish","type":"post","link":"https:\/\/pegamento.nl\/en\/contact-center\/which-customer-service-applications-are-considered-high-risk-ai\/","title":{"rendered":"Which customer service applications are considered high-risk AI?"},"content":{"rendered":"<p>AI applications in customer service are considered high-risk when they are used to make decisions that have a significant impact on fundamental rights, access to essential services, or people\u2019s legal status. Examples include automated systems that assess creditworthiness, handle emergency calls, or create customer profiles based on personal characteristics. The <a href=\"https:\/\/pegamento.nl\/en\/ai-powered-intelligence\/\">EU AI Act<\/a> imposes specific obligations in this regard that are becoming increasingly relevant for customer service organizations. This article answers the most frequently asked questions about high-risk AI in the context of customer service.   <\/p>\n<h2>What officially classifies an AI system as \u201chigh-risk\u201d under the AI Act?<\/h2>\n<p>An AI system is officially considered high-risk under the EU AI Act if it falls within one of the eight domains listed in Annex III of the regulation, or if it constitutes a safety component of a product that is already subject to EU harmonization legislation and requires a third-party conformity assessment. Profiling of natural persons is always high-risk, without exception. <\/p>\n<p>The eight areas listed in Annex III are specifically defined in the law. These include biometrics, critical infrastructure, education and vocational training, employment and human resources management, access to essential services (such as creditworthiness, insurance, and emergency calls), law enforcement, migration and border control, and the administration of justice and democratic processes. <\/p>\n<p>There is one exception to the Annex III classification: a system that performs only a narrow procedural or preparatory task and does not pose a significant risk to fundamental rights may fall outside the high-risk category. However, the provider must provide substantiated documentation to support this. As soon as a system profiles individuals, that possibility is completely ruled out.  <\/p>\n<h2>Which specific customer service AI applications are considered high-risk?<\/h2>\n<p>In the context of customer service, AI applications are considered high-risk when they support or make decisions regarding access to essential services, create personal profiles, or handle emergency calls. Not every chatbot or routing tool is automatically high-risk, but the line is closer to everyday practice than many organizations realize. <\/p>\n<p>Specific examples of customer service AI that are classified as high-risk:<\/p>\n<ul>\n<li><strong>Automated credit assessment:<\/strong> AI that determines whether a customer is eligible for a payment plan, credit, or financial product is explicitly covered by Annex III.<\/li>\n<li><strong>Insurance-related decisions:<\/strong> Systems that analyze customer data to assess insurance applications or calculate premiums are high-risk.<\/li>\n<li><strong>Handling of emergency calls:<\/strong> AI that routes, prioritizes, or assesses emergency calls falls under the category of access to essential services.<\/li>\n<li><strong>Customer profiling for personalized offers:<\/strong> As soon as a system analyzes customer behavior to create individual profiles, it is, by definition, high-risk due to the profiling provision.<\/li>\n<li><strong>AI in Recruitment Through Customer Service:<\/strong> Systems that evaluate applicants or employees based on customer service interactions fall under the categories of employment and human resources management.<\/li>\n<\/ul>\n<p>A standard FAQ chatbot that answers general questions without creating user profiles generally falls outside the high-risk category. But as soon as such a chatbot combines customer data to make personalized decisions, its classification changes. <\/p>\n<h2>What is the difference between high-risk AI and prohibited AI in customer service?<\/h2>\n<p>Prohibited AI applications are completely banned under the AI Act and may not be deployed, regardless of the context or precautionary measures. High-risk AI is not prohibited, but may only be deployed if strict compliance requirements are met. The distinction is fundamental: prohibited AI is a hard line, while high-risk AI is a regulated category.  <\/p>\n<p>In the context of customer service, prohibited uses include: AI systems that use subliminal techniques to manipulate customer behavior without their knowledge; systems that exploit the vulnerabilities of specific groups (such as the elderly or people with disabilities); and real-time biometric identification in public spaces for law enforcement purposes.<\/p>\n<p>High-risk AI in customer service is permitted, but requires, among other things, a conformity assessment, technical documentation, human oversight, and registration in the EU database. The bans have been in effect since February 2, 2025. Anyone still using prohibited AI applications now risks fines of up to 35 million euros or 7% of global annual revenue.  <\/p>\n<h2>What obligations apply to organizations that use high-risk AI in customer service?<\/h2>\n<p>Organizations that use high-risk AI in customer service must comply with a comprehensive set of obligations, depending on their role in the chain. The AI Act distinguishes between providers (who develop the system), deployers (who implement the system), and importers or distributors. Each role entails its own set of responsibilities.  <\/p>\n<h3>Requirements for Providers of High-Risk AI<\/h3>\n<p>Suppliers must conduct a conformity assessment, prepare and maintain technical documentation, affix the CE marking, and register the system in the EU database. In addition, they are required to establish a quality management system and ensure that human supervision is technically feasible. <\/p>\n<h3>Requirements for Deployers of High-Risk AI<\/h3>\n<p>Organizations that deploy high-risk AI in their customer service (deployers) must follow the provider\u2019s instructions, ensure human oversight, document data management practices, and conduct a fundamental rights impact assessment for certain applications. Important note: a deployer may inadvertently become a provider if they substantially modify the system or alter its intended purpose in such a way that it becomes high-risk. In that case, all provider obligations apply.  <\/p>\n<p>In practical terms, this means that customer service organizations must already begin compiling a registry of all AI systems and determine their role in them. Special attention must be paid to circumstances that could inadvertently turn a party into a provider. <\/p>\n<h2>When will the high-risk AI regulations take effect for customer service organizations?<\/h2>\n<p>The compliance requirements for high-risk AI systems covered by Annex III will become enforceable on August 2, 2026. That is the key date for most customer service AI applications. The prohibitions and the AI literacy requirement have been in effect since February 2, 2025.  <\/p>\n<p>The complete timeline is as follows:<\/p>\n<ol>\n<li><strong>February 2, 2025:<\/strong> Prohibited AI practices (Article 5) and the requirement for AI literacy (Article 4) take effect. <\/li>\n<li><strong>August 2, 2025:<\/strong> Requirements for GPAI models, notified bodies, governance structures, and penalty provisions take effect.<\/li>\n<li><strong>August 2, 2026:<\/strong> Compliance requirements for high-risk Annex III systems become enforceable. This is the most relevant date for customer service AI. <\/li>\n<li><strong>August 2, 2027:<\/strong> Requirements for high-risk AI used as a safety component in regulated products (Annex I) take effect. GPAI models that were already on the market before August 2025 must also comply by that date. <\/li>\n<\/ol>\n<p>So by 2026, there will be no time left to sit back and relax. Organizations that have not yet taken stock of their AI systems run the risk of being ill-prepared when enforcement begins. <\/p>\n<h2>How do you determine whether your current customer service AI tools are high-risk?<\/h2>\n<p>To determine whether your customer service AI tools are high-risk, ask three key questions for each system: Does the system create profiles of individuals? Does it influence decisions regarding access to essential services? And does its use fall under one of the eight Annex III domains? If you answer \u201cyes\u201d to any of these questions, the system is likely high-risk.   <\/p>\n<p>A practical approach to assessing this:<\/p>\n<ul>\n<li><strong>Take inventory of all AI systems:<\/strong> Create a registry of every system that uses AI, including chatbots, routing tools, speech recognition, and analytics software.<\/li>\n<li><strong>Determine your role:<\/strong> Are you a supplier, deployer, importer, or distributor? This determines which obligations apply to you. <\/li>\n<li><strong>Analyze the intended purpose:<\/strong> What does the system actually do? Does it evaluate customers, create profiles, or make decisions that affect customers? <\/li>\n<li><strong>Check for profiling:<\/strong> Any system that creates individual customer profiles based on behavior, preferences, or personal characteristics is automatically considered high-risk.<\/li>\n<li><strong>Request documentation from suppliers:<\/strong> Providers of high-risk AI are required to provide documentation regarding capabilities, limitations, and compliance status.<\/li>\n<\/ul>\n<p>Are you unsure about the classification of a specific system? The law allows providers to classify an Annex III system outside the high-risk category if it performs only a narrow procedural task and does not pose a significant risk to fundamental rights. However, this requires substantiated documentation and does not apply to systems that perform profiling.  <\/p>\n<h2>How Pegamento Helps Ensure AI Compliance in Customer Service<\/h2>\n<p>Navigating the AI Act is a significant challenge for many customer service organizations, especially when existing systems have been in operation for years and the question is how they align with the new regulations. We help organizations map out their AI landscape and set up responsible, compliant customer service solutions. <\/p>\n<p>What we specifically offer:<\/p>\n<ul>\n<li><strong>AI Systems Inventory:<\/strong> We help you create a comprehensive overview of all AI applications in your customer service environment and determine the risk classification for each system.<\/li>\n<li><strong>Agentic AI for Customer Service:<\/strong> Our <a href=\"https:\/\/pegamento.nl\/en\/agentic-ai-for-customer-service\/\">Agentic AI solutions for customer service<\/a> are designed with human oversight as a core principle. Agentic AI represents the evolution from task-oriented bots to self-thinking assistants that not only follow instructions but also take the initiative and act independently, within the parameters you set. <\/li>\n<li><strong>Everything under one roof:<\/strong> From development and implementation to management and support. No complex supplier structures\u2014just a single point of contact for the complete package. <\/li>\n<li><strong>Customized solutions using standard building blocks:<\/strong> No costly custom work, but a smart combination of proven modules that fit your specific situation and compliance requirements.<\/li>\n<li><strong>ISO 27001 certified:<\/strong> Information security is our top priority, complemented by ISO 9001 and ISO 26000 certifications that ensure quality and social responsibility.<\/li>\n<\/ul>\n<p>Would you like to know how your current customer service AI tools measure up against the high-risk criteria of the AI Act? <a href=\"https:\/\/pegamento.nl\/en\/contact-2\/\">Contact us<\/a>, and we\u2019d be happy to work with you to develop an approach that suits your organization.<\/p>\n<div class=\"wp-block-seoaic-faq-block\">\n    <h2 class=\"seoaic-faq-section-title\">Frequently Asked Questions<\/h2>\n            <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Wat gebeurt er als mijn organisatie per ongeluk aanbieder wordt van hoog-risico AI?            <\/h3>\n            <p class=\"seoaic-answer\">\n                Als je als deployer een hoog-risico AI-systeem substantieel wijzigt of het beoogde doel aanpast waardoor het hoog-risico wordt, val je automatisch onder de aanbiedersverplichtingen. Dit betekent dat je alsnog een conformiteitsbeoordeling moet uitvoeren, technische documentatie moet opstellen en het systeem moet registreren in de EU-databank. Het is daarom essentieel om bij elke aanpassing van een AI-systeem vooraf te toetsen of die wijziging je rol in de keten verandert.            <\/p>\n        <\/div>\n                <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Hoe voer ik een grondrechteneffectbeoordeling uit voor mijn klantenservice AI?            <\/h3>\n            <p class=\"seoaic-answer\">\n                Een grondrechteneffectbeoordeling (Fundamental Rights Impact Assessment) breng je stap voor stap in kaart: identificeer welke grondrechten het systeem mogelijk raakt, analyseer de risico&#8217;s per betrokken groep (zoals kwetsbare klanten), en beschrijf welke maatregelen je neemt om die risico&#8217;s te mitigeren. Deployers van hoog-risico AI zijn verplicht deze beoordeling uit te voeren v\u00f3\u00f3r ingebruikname. Raadpleeg hiervoor ook de richtlijnen van de Europese AI Office, die praktische sjablonen en guidance publiceren.            <\/p>\n        <\/div>\n                <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Wat betekent de AI-geletterdheidsplicht concreet voor mijn klantenserviceteam?            <\/h3>\n            <p class=\"seoaic-answer\">\n                Sinds 2 februari 2025 zijn organisaties verplicht ervoor te zorgen dat medewerkers die met AI-systemen werken voldoende kennis hebben van de werking, beperkingen en risico&#8217;s van die systemen. Voor klantenserviceteams betekent dit in de praktijk dat agents die werken met AI-ondersteunde tools getraind moeten worden in het herkennen van AI-gegenereerde output, het uitoefenen van menselijk toezicht en het signaleren van afwijkend systeemgedrag. Leg deze trainingen en hun inhoud vast in documentatie als bewijs van naleving.            <\/p>\n        <\/div>\n                <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Kan een standaard CRM-systeem met AI-functies ook als hoog-risico worden aangemerkt?            <\/h3>\n            <p class=\"seoaic-answer\">\n                Ja, dat is mogelijk. Als een CRM-systeem AI-functies bevat die klantprofielen aanmaken op basis van gedrag of persoonskenmerken, of als het aanbevelingen doet die invloed hebben op toegang tot diensten, kan het onder de hoog-risico-categorie vallen \u2014 ook al is het primair een CRM. De classificatie hangt niet af van het type software, maar van wat het systeem concreet doet. Vraag bij je CRM-leverancier expliciet na welke AI-functies actief zijn en of zij conformiteitsdocumentatie kunnen aanleveren.            <\/p>\n        <\/div>\n                <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Welke documentatie moet ik nu al aanleggen, ook al is de deadline pas in augustus 2026?            <\/h3>\n            <p class=\"seoaic-answer\">\n                Begin direct met een AI-register waarin je elk systeem vastlegt met het beoogde doel, de gebruikte data, je rol in de keten (aanbieder of deployer) en een voorlopige risicoclassificatie. Leg daarnaast de instructies en documentatie van je AI-leveranciers vast, en documenteer hoe menselijk toezicht is ingericht. Hoe eerder je dit register opbouwt, hoe minder werk je hebt naarmate de deadline nadert \u2014 en hoe sterker je positie is als toezichthouders vroegtijdig vragen stellen.            <\/p>\n        <\/div>\n                <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Wat zijn de meest voorkomende fouten die organisaties maken bij het inschatten van hun AI-risicoclassificatie?            <\/h3>\n            <p class=\"seoaic-answer\">\n                De meest gemaakte fout is aannemen dat een systeem veilig is omdat het &#8216;slechts ondersteunend&#8217; werkt of geen eindbesluiten neemt. Onder de AI Act is ook beslissingsondersteuning hoog-risico als het de uitkomst significant be\u00efnvloedt. Een tweede veelgemaakte fout is het vergeten van indirecte profilering: systemen die op het oog alleen segmenteren of personaliseren, maken in de praktijk toch individuele profielen aan. Controleer altijd op basis van wat het systeem feitelijk doet, niet op basis van hoe de leverancier het positioneert.            <\/p>\n        <\/div>\n                <div class=\"seoaic-faq-item\">\n            <h3 class=\"seoaic-question\">\n                Hoe ga ik om met hoog-risico AI van een externe leverancier die nog geen conformiteitsdocumentatie heeft?            <\/h3>\n            <p class=\"seoaic-answer\">\n                Als je leverancier geen conformiteitsdocumentatie kan aanleveren, is dat een serieus risicosignaal. Als deployer ben je medeverantwoordelijk voor naleving en kun je niet volstaan met de verklaring dat &#8216;de leverancier het regelt&#8217;. Stel je leverancier schriftelijk in gebreke en vraag om een concreet tijdpad voor conformiteit. Overweeg in de tussentijd het gebruik van het systeem te beperken tot niet-hoog-risico toepassingen, of start een selectietraject voor een alternatief dat aantoonbaar compliant is v\u00f3\u00f3r augustus 2026.            <\/p>\n        <\/div>\n        <\/div>\n","protected":false},"excerpt":{"rendered":"<p>Credit scoring, customer profiling, emergency calls: Find out which customer service AI applications are considered high-risk under the EU AI Act.<\/p>\n","protected":false},"author":2,"featured_media":31849,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[500],"tags":[],"class_list":["post-31848","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-contact-center"],"_links":{"self":[{"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/posts\/31848","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/comments?post=31848"}],"version-history":[{"count":2,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/posts\/31848\/revisions"}],"predecessor-version":[{"id":31851,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/posts\/31848\/revisions\/31851"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/media\/31849"}],"wp:attachment":[{"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/media?parent=31848"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/categories?post=31848"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pegamento.nl\/en\/wp-json\/wp\/v2\/tags?post=31848"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}