The real operational risk isn't a hallucinating chatbot; it's the sprawl around it. Teams are standing up their own AI pipelines against unreviewed databases, invisible to whoever tracks what's actually running. This "shadow AI" pattern mirrors the DevOps-versus-platform engineering turf war that plagued software delivery for years, except now the stakes involve model costs, data governance, and regulatory compliance.

What Is LLMOps and Why Does It Matter?

Large language model operations, or LLMOps, is the set of practices, tools, and workflows used to develop, deploy, and manage LLMs throughout their production lifecycle. Unlike traditional machine learning operations, LLMOps addresses a fundamentally different problem: LLMs cost significantly more to fine-tune and serve, and their output is far harder to evaluate than a simple accuracy score. An LLM has to be secure, trustworthy, and aligned with organizational values, not just accurate.

The LLMOps lifecycle spans multiple stages, from data preparation and prompt engineering through fine-tuning open foundation models with libraries like Hugging Face Transformers, to model versioning, inference serving on GPU-backed endpoints, and ongoing monitoring with human feedback. Each stage requires infrastructure, access controls, and governance.

How Are Organizations Currently Managing LLMOps?

The problem is organizational fragmentation. Software delivery has split into three competing camps: DevOps teams handling deployment tickets, platform engineering teams building self-service golden paths, and MLOps teams who built an entirely parallel stack because traditional DevOps tooling never anticipated data versioning or drift monitoring. Add LLMOps to the mix, and companies risk a third parallel stack for prompts, vector stores, and retrieval-augmented generation (RAG) pipelines, invisible to whoever is supposed to be governing the rest.

The unhelpful question is "who owns the pipeline?" The useful one is "who owns which layer, and is anyone actually coordinating across them?" Platform engineering is infrastructure-centric, while MLOps is model-centric. These are different disciplines solving different problems, with platform engineering serving as the backbone the others run on top of.

Steps to Implement Governed LLMOps Through Platform Engineering

• Governed APIs, Not Ad Hoc Scripts: Fine-tuning jobs, prompt deployments, and inference endpoints should be requestable through the same self-service interface developers already use for everything else, rather than built in isolation.
• Policy Enforcement at Request Time: Cost limits, data residency rules, and model access controls should be checked before a job kicks off, not discovered after the cloud bill arrives or a compliance audit begins.
• Human Approval Where Warranted: Not every prompt change needs sign-off, but a model touching customer personally identifiable information or making autonomous decisions probably does, with clear audit trails documenting who approved what and why.
• Audit Trails for Accountability: Regulators and security teams need to know what changed about the model, the prompt, or the data, and who or what approved it, creating a complete record of the system's evolution.

The teams getting this right aren't picking a side in the DevOps-versus-platform-versus-MLOps argument. Rather, they're treating the entire pipeline, LLM workloads included, like a product: versioned, observable, cost-aware, and built with feedback loops rather than handed down as a fixed set of rules.

Why Does Shadow AI Pose Such a Serious Risk?

The bigger operational risk isn't a hallucinating chatbot; it's the sprawl around it. A team standing up its own RAG pipeline against an unreviewed vector store, invisible to whoever tracks what's actually running, creates the same shadow IT problem that platform engineering was built to prevent, just with prompts instead of Jenkinsfiles. This "shadow AI" pattern emerges because a capability gets built outside the platform when the platform wasn't ready, and it never gets folded back in.

The fix isn't slowing teams down. Instead, it's making the platform say yes fast, with governance built in. LLMOps doesn't need its own kingdom; it needs a well-run platform willing to let it in. The CNCF TAG App Delivery's Platforms Whitepaper describes a three-tier split: Products at the top, Platforms in the middle as the thinnest reasonable integration layer, and Capability Providers at the bottom. Model fine-tuning jobs, vector databases, prompt registries, and inference endpoints are functionally just another platform capability, needing the same API, versioning, and clear ownership as anything else.

"LLMOps isn't a rebranding of MLOps; it's MLOps under real strain: bigger models, higher costs, fuzzier evaluation, and a production surface that didn't exist a few product cycles ago," noted Daniel Bryant, a platform engineering expert at Syntasso.

Daniel Bryant, Platform Engineering Expert at Syntasso

The tooling already exists across the CNCF ecosystem to solve this problem. Backstage surfaces golden paths at the product layer, Crossplane composes infrastructure at the bottom layer, and orchestration frameworks such as Kratix, KusionStack, and KubeVela sit in the middle, letting teams expose an LLM pipeline through the same self-service interface as everything else. Skip that integration, and the DevOps-versus-MLOps turf war resurfaces one layer higher, with LLMOps caught in the middle.

The conversation around LLMOps governance is actively happening inside the CNCF community. TAG App Delivery's Platforms Working Group is where much of the cross-project thinking on golden paths and platform governance is already taking place, and it remains open to anyone who wants to help shape how organizations will manage AI pipelines at scale.