AI agents query data constantly and across multiple tools simultaneously, making the choice between open and proprietary data architectures far more consequential than it was for human analysts. When organizations deploy agentic AI, the metadata layer sitting between raw storage and query engines becomes a strategic decision that can either unlock flexibility or lock in vendor dependency.

What's the Difference Between Open and Proprietary Data Layers?

A data lakehouse stores raw data in cloud object storage like Amazon S3 or Google Cloud Storage. On its own, that storage has no concept of transactions, schema, or version history. A table format and catalog layer sit above the files and add the structure that engines and applications rely on, including ACID transactions (atomicity, consistency, isolation, and durability), schema evolution, time travel, and effective query planning.

There are two ways to implement this layer. Open table formats, such as Apache Iceberg and Delta Lake, are open source and allow any compliant engine to read and write data, typically through an open catalog such as a REST catalog. Proprietary metadata layers, built into platforms like a single-vendor warehouse, can only be accessed by other engines through vendor-provided bridges, if at all.

The distinction matters more for agentic AI than for traditional analytics. While human data analysts may run scheduled dashboards and occasional ad hoc queries using one or two tools, AI agents query continuously and unpredictably, across many different frameworks and tools at once.

Why Does Vendor Lock-In Matter More for AI Agents?

The more open and standards-based the table format and catalog are, the less custom integration work is required every time a new agent framework, model provider, or internal tool is adopted. Standardizing on Iceberg as the table format, accessed through a REST catalog, doesn't preclude using proprietary-platform compute engines where they add value. Most major vendors, including Snowflake and Databricks, now expose Iceberg-compatible REST catalog APIs.

The key is to confirm that any vendor under consideration allows Iceberg tables to be written and read natively, not just imported or exported, and that an agent-facing access path, such as Model Context Protocol (MCP), is available or on a committed roadmap.

How to Evaluate Data Platforms for Agentic AI Workloads

• Multi-engine compatibility: Can every tool or AI agent you use actually read the same data? If you bring in a new AI tool next year, can the solution read your data on day one, or do you need the vendor's permission and custom engineering first?
• Modern connection standards: Can AI agents connect the way modern AI tools expect to connect? If the vendor doesn't support MCP, every new AI project may need custom, one-off engineering work to get the agent talking to your data.
• Granular permissions: Can you limit exactly what each AI agent is allowed to see? An agent built for customer support should not be able to see payroll or financial data, even by accident. Check how granular a vendor's permissions are for AI agents specifically, and whether access can be limited to exactly what's needed and automatically expire.
• Accumulated knowledge: Can the system get smarter over time, or does it forget everything? As AI agents use your data, they can learn useful things along the way, such as which reports are actually used or how data relates across departments, which could help the next agent work faster.
• Performance under load: Will performance hold up once AI agents are barraging the database with questions? A human analyst might run a handful of reports a day. AI agents can generate hundreds or thousands of small requests in rapid succession, which can slow a system down in ways that aren't obvious from a normal demo.

Open Architecture Doesn't Mean Weak Governance

One common misconception is that choosing an open architecture means giving up governance. An open table format can coexist with strong identity controls, catalog-level permissions, lineage, monitoring, encryption, and policy enforcement. The challenge is that those capabilities may come from several components rather than a single vendor platform.

That creates a different responsibility for the CTO: defining which capabilities should remain portable and which are worth tying to a specific platform. A useful architecture might keep data in an open table format and object storage while using proprietary engines for workloads where they provide a meaningful advantage. Another organization may prefer a more integrated platform because operational simplicity matters more than portability.

How Open and Proprietary Approaches Compare

When comparing Iceberg to proprietary metadata layers, several factors stand out:

• Format ownership: Apache Iceberg is open source via the Apache Software Foundation and implemented by many engines, while proprietary metadata layers are owned and controlled by a single vendor.
• Engine access: Any compliant engine such as Spark, Trino, Flink, DuckDB, Snowflake, or Databricks can read and write the same Iceberg table, whereas proprietary layers typically limit access to the vendor's own engine or offer read-only access for others.
• Catalog interoperability: REST Catalog standards enable multi-engine governance on one copy of data, while proprietary catalogs depend on vendor-provided bridges or exports.
• Data portability: Open formats carry low lock-in risk because data stays in Parquet or ORC format in your object storage and metadata is swappable, whereas proprietary layers carry higher risk because migrating off the platform often requires re-ingesting or exporting data.
• AI and agent tool ecosystem: Open formats have broad and growing MCP and agent tool support because many vendors build against the same open standard, while proprietary options have narrower agent tooling limited to what that vendor ships or partners on.
• Pace of innovation: Community-driven open formats ship features such as Iceberg v3 across many vendors simultaneously, while vendor-driven proprietary layers arrive on the vendor's roadmap and timeline.

The right choice depends on the organization's workload, governance requirements, skills, and tolerance for platform dependency. In a recent Cisco and Omdia study titled "The Race to Agentic AI: Why Infrastructure Will Make or Break Workforce Transformation," 87% of surveyed executives said their strategic priorities have been fundamentally reshaped by agentic AI. On average, they estimate that 55% of their workforce will regularly work alongside agents within the next 24 months. The question for CTOs is whether your data architecture is ready.