The AI Center of Excellence is the organizational structure that enterprises create when they are serious about AI at scale. It is also the organizational structure that, in most enterprises, becomes the single greatest impediment to AI at scale within eighteen months of its creation.
The failure pattern is consistent. The CoE starts as a small team of senior practitioners with a mandate to accelerate AI adoption. Within a year, it has accumulated approval requirements, review processes, platform mandates, and governance checkpoints that mean any new AI project must pass through the CoE before it can proceed. The team that was supposed to accelerate AI adoption has become the organization's AI traffic controller. The backlog grows. Business units begin building AI capabilities outside the CoE structure to avoid the delays. The CoE responds with stronger governance requirements to bring shadow AI under control. The cycle accelerates in the wrong direction.
This pattern is not inevitable. It is the result of a specific organizational design choice, made at CoE formation, about what the CoE's primary function is. Getting that choice right at formation is far easier than correcting it after the CoE has accumulated the organizational gravity that comes with approval authority.
The Two Fundamental CoE Models
Every AI CoE design sits somewhere on a spectrum between two extreme models. Understanding these extremes clarifies why so many organizations end up in the bottleneck failure pattern and what the alternative looks like in practice.
The Gate
The CoE owns the AI platform, approves all AI projects, employs all AI practitioners, and is accountable for AI governance. Business units submit requests. The CoE reviews, approves, and assigns resources. Every AI initiative must pass through the gate. This model provides consistency and control. It also creates a single point of failure for organizational AI velocity.
The Platform
The CoE owns standards, tools, and training. Business units own execution. The CoE provides the platform, the guardrails, and the expertise that business units need to deploy AI safely and effectively. Business units employ embedded AI practitioners who report into the business unit and access CoE resources and standards. Speed of deployment lives with the business unit. Quality and compliance sit on the CoE platform.
The federated enablement model is harder to build and requires more upfront investment in tooling, documentation, and training. It is also the only model that scales. An organization with twenty active AI programs cannot staff a centralized review function that keeps pace with twenty programs while maintaining review quality. The math does not work. The federated model moves the constraint from the CoE's review capacity to the business units' implementation capacity, which distributes the scaling problem across the organization rather than concentrating it in one team.
The Five Core Functions of a Functioning CoE
Regardless of where a CoE sits on the centralized-to-federated spectrum, there are five functions that a CoE must perform to justify its existence. These are not the functions that most CoEs spend their time on, which is part of the problem.
Standards Development
Creating and maintaining the technical standards, development guidelines, and deployment criteria that define what good AI looks like across the organization. This is a production function, not an approval function.
Platform and Tooling
Owning and operating the shared infrastructure, model registries, data pipelines, and MLOps tooling that reduce the marginal cost of each new AI deployment. The more this platform does, the faster each individual team moves.
Practitioner Development
Training, certification, and community of practice management for AI practitioners embedded in business units. The CoE does not employ all AI practitioners. It develops all of them.
Risk and Compliance Advisory
Providing expert guidance on AI risk, governance requirements, and compliance to business units before they build, not after. The CoE is a consulting resource for compliance questions, not an approval body for compliance sign-off.
Strategic Portfolio View
Maintaining visibility into AI programs across the organization to identify duplication, share learnings, and inform the C-suite on AI portfolio performance. Not to control the portfolio, but to inform it.
The function that is conspicuously absent from this list is project approval. In a well-designed CoE, projects do not require CoE approval to proceed. They require compliance with CoE standards, which is a different thing. The difference is that compliance can be demonstrated by the project team without requiring CoE review time. Approval requires review time, which creates the queue, which creates the bottleneck.
Getting Governance Right Without Creating Gates
The most common objection to the federated model is governance. If the CoE does not approve AI projects, how does the organization ensure that AI systems are safe, compliant, and aligned with enterprise standards? This is a legitimate concern. The answer is that governance does not require approval queues. It requires standards that are clear enough to be self-assessed and infrastructure that makes compliance the path of least resistance.
The governance design that works at scale has three components. The first is a tiered risk framework that classifies AI systems by their risk profile and applies proportionate governance requirements. Low-risk AI systems, such as internal productivity tools and content generation assistants, have lightweight requirements that any competent AI practitioner can satisfy without CoE involvement. High-risk AI systems, such as those that make decisions affecting customers, employees, or regulated activities, have substantive requirements that typically benefit from CoE guidance and may require independent review. The CoE designs the framework. Business units self-classify and comply.
The second component is a compliance-by-design platform. When the shared infrastructure that business units use to build and deploy AI automatically enforces data governance requirements, model documentation standards, and monitoring obligations, compliance is not an approval process. It is a property of the platform. The CoE invests in making the right thing the easy thing rather than in reviewing whether teams are doing the right thing.
The third component is post-deployment audit rather than pre-deployment approval. A sample of AI deployments is reviewed after deployment against a defined audit checklist. Teams that are found out of compliance receive remediation requirements. Teams with clean audits build a track record that reduces future oversight requirements. This creates the right incentive structure: teams are accountable for compliance and develop genuine competence in meeting standards, rather than becoming skilled at passing approval reviews.
For the detailed governance framework design, see the AI governance advisory service and the enterprise AI governance guide.
The Four Ways CoEs Fail
Mandate Scope Creep
The CoE is asked to review a project for technical quality. It adds a risk review. Then a data governance review. Then a vendor approval requirement. Within two years, every project requires five CoE sign-offs that collectively take longer than the project itself. Mandate scope creep is almost always well-intentioned and always destructive.
Talent Accumulation Without Distribution
The CoE hires the best AI practitioners in the organization and keeps them. Business units cannot build internal capability because the talent is centralized. The CoE becomes the only place where competent AI work happens, which concentrates the bottleneck further rather than distributing capacity.
Platform Without Adoption
The CoE builds a technically excellent shared platform that business units do not adopt because it is harder to use than the commercial tools they already have access to. The platform investment is stranded. Business units continue building on uncoordinated commercial tools. The CoE's influence over AI deployment is zero despite significant infrastructure investment.
Metrics Misalignment
The CoE is measured on projects reviewed, standards published, and governance checklists completed. None of these metrics capture the CoE's actual purpose, which is accelerating AI value delivery across the enterprise. When the metrics do not measure what matters, the CoE optimizes for the metrics and produces the wrong outcomes.
Measuring a CoE That Is Working
The right metrics for an AI CoE are outcomes metrics for the business units it serves, not activity metrics for the CoE itself. The question is not "how many projects did the CoE review this quarter?" It is "what is the average time from AI project start to production deployment across the enterprise, and is it improving?" Not "how many governance standards did we publish?" but "what fraction of AI projects are meeting their stated business objectives within twelve months of deployment?"
CoEs that are genuinely enabling AI deployment show improvement in these outcomes metrics over time. CoEs that have become bottlenecks show stagnation or deterioration in the same metrics while showing impressive activity metrics. When a CoE presents its quarterly review to the leadership team, the first chart should be deployment velocity across the enterprise. If it is a chart of CoE activities instead, the CoE has the wrong success definition.
See the building an AI organization guide for the full organizational design framework and the AI Center of Excellence advisory service for how we help enterprises design CoEs that accelerate rather than impede.
Getting Formation Right
The most important CoE design decisions are made at formation. Once the CoE has established an approval-based operating model, restructuring it requires overcoming the organizational gravity of every team that has adapted their workflow to the approval process and every risk function that relies on the CoE as a control point. Reform is possible. Formation is easier.
The three formation decisions that determine whether a CoE accelerates or impedes are: the mandate definition (enable or control), the staffing model (talent accumulation or talent distribution), and the governance design (approval queues or compliance infrastructure). Organizations that get all three right at formation build CoEs that accelerate AI deployment and remain effective as AI programs scale. Organizations that get any one of these wrong spend the next two to three years managing the consequences.
For related guidance on the organizational structure that supports effective CoE operation, see the AI team structure guide and the enterprise AI strategy overview.