State of agents on the internet, layer by layer

We just crossed a strange line: bots and agents now make up more than half of all internet traffic, around 53% in the 2026 Thales and Imperva report. The web officially has more machine visitors than human ones.

So I wanted to write a deepdive on the state of agents on the internet, layer by layer.

Basics first. Right now two protocols shape how an agent "browses" the web.

1. MCP (Model Context Protocol) is how an agent plugs into a tool. It lets the agent take one concrete action: read this file, call that endpoint, run this query. But MCP has a catch: on its own it does not really solve trust. Anyone can walk into your house and use your stuff. That gap is a big part of why A2A showed up.

2. A2A (Agent2Agent): built by Google, announced on April 9, 2025, then donated to the Linux Foundation the same year. The idea is simple: one agent hands a whole job to another agent and walks away, without needing to know how that other agent works inside. Each agent publishes an Agent Card, a small JSON (JavaScript Object Notation) file placed at a standard path that says who it is, what it can do, and how to authenticate. Another agent reads that card, hands over the task, tracks the lifecycle, and takes the result.

One year in, 150+ organizations have joined, A2A runs on the major cloud platforms, and it has real production use in enterprises. Read that word enterprise carefully. Almost all A2A traffic today runs inside a single company, between agents that already share one owner. The romantic version, an open internet where a stranger agent finds and hires another stranger agent, does not exist at meaningful scale yet. What actually crosses company lines in volume right now is commerce and payments, not pure A2A.

So the real question is not whether agents talk to each other. The question is which layer of the stack is already built, who owns it, and where real money is flowing.

Layer 1: Discovery (the agent finds you)

llms.txt is a text file that tells AI what your site contains, the same idea as robots.txt for search crawlers like Google. From Casey Burridge's research on HTTP Archive data in June 2026: only 5.61% of the top 10,000 websites have a valid llms.txt. That is up about 5.4x in twelve months, but it is growth off a very low base, and there is still no clear proof that the big AI systems actually read the file.

Shopify quietly pushed llms.txt to nearly all of its stores across April and May 2026. That is why Shopify accounts for about 78% of all llms.txt adoption in the top 10,000 sites: not because 78% of Shopify stores opted in, but because a single release switched this technical layer on for millions of merchants at once.

This is the pattern that runs through the whole map. Platforms will automate the plumbing for their entire customer base with one deploy, and when they do, any standalone product selling that exact plumbing loses its reason to exist overnight. Microsoft's NLWeb sits in this layer too, along with the official MCP Registry that went to preview in September 2025.

Layer 2: Identity and trust (proving an agent is who it claims)

Before you let an agent act, you need to know it is not a hostile bot in a costume. The answer taking shape is Web Bot Auth, a cryptographic signature standard that lets an agent sign its requests so a server can verify it. It is being standardized at the IETF (Internet Engineering Task Force), it has its own working group, and Cloudflare is leading it. This is the heaviest kind of infrastructure, and it belongs to companies that already sit in front of a huge slice of internet traffic.

Layer 3: Auth (letting an agent act on your behalf)

Different problem from Layer 2. Here the agent has to do something as you, with your permissions, without you handing over your raw password. Arcade raised $60M in June 2026 to build exactly this secure action layer for agents. WorkOS, Stytch and Auth0 have all moved in. This layer is crowded and well funded, which means the market already believes it is real.

Layer 4: Payments (letting an agent pay)

The protocol here is x402, a way to pay over HTTP 402, the Payment Required response code. That code sat basically unused in the web for decades.

Coinbase started it in May 2025, and the x402 Foundation went operational in July 2026 under the Linux Foundation, co founded with Cloudflare. In that same month Visa, Mastercard and Ripple joined, and Cloudflare and AWS (Amazon Web Services) embedded x402 support right at the edge.

In a recent 30 day window, x402 saw about 75 million transactions moving a combined $24M, an average of roughly 32 cents per transaction. The rails are being built by giants, but the real money is not flowing much yet. Alongside this, Cloudflare turned on Pay Per Crawl on July 1, 2026, forcing AI crawlers to pay to read content, again over HTTP 402, and paired it with a GoDaddy partnership on an open agentic web announced back in April.

Layer 5: Commerce (letting an agent buy)

This is an active war, not settled yet. There are 3 main players:

1. ACP (Agentic Commerce Protocol) from OpenAI and Stripe, once the method behind Instant Checkout in ChatGPT. The consumer product was reportedly wound down in early 2026, while the protocol lives on as infrastructure.

2. UCP (Universal Commerce Protocol), a coalition play, launched in January 2026 by Google, Shopify, Etsy, Walmart, Target and more than twenty other partners.

3. AP2 (Agent Payments Protocol), which started at Google in 2025 and was then donated to the FIDO Alliance in April 2026. Shopify is live with self serve commerce, and merchants generally have to pick a side or eat the cost of supporting several standards at once.

Small side note: back when I was building the Intern product at my old company, I wanted to integrate AP2 but never finished it ><

Layer 6: Converter

Speakeasy Gram: hosted platform, OpenAPI spec (a standard API specification) into an MCP server. Stainless: generates SDKs and MCP servers from OpenAPI. Mintlify: a button that turns docs into an MCP server, literally 1 click. Postman also generates MCP servers. FastMCP: open source, a few lines of code. Cloudflare: a 1 click template to deploy an MCP server to Workers.

Layer 7: Catalog and integration (selling access to a library that already exists)

Instead of generating one server, you sell access to a huge pre built library of integrations. Zapier MCP has around 9,000 apps. Composio, Pipedream and Klavis all play the same game. The moat here is the catalog, and a catalog takes years to build.

Layer 8: Observability and analytics (Google Analytics for agents)

When traffic shifts from humans clicking a UI (User Interface) to agents acting through Claude, ChatGPT and Cursor, your old analytics go blind. Armature, a Y Combinator company, rebuilds the session: it replays what the user asked, what the agent thought, and every call it made, detects intent, groups sessions into use cases ranked by volume and success rate, spots where the agent fails, and scores whether the user actually got what they needed. Pricing is a free tier of 1,000 sessions a month, then $50 per additional 1,000 sessions. This is analytics for agent traffic.

Layer 9: Standards and audit (scoring agent readiness)

Even this meta layer already has someone in it. AgentReady.org is an open specification under an MIT license that defines agent readiness across five groups: discoverability, content, capabilities, identity and access, and commerce. It ships with a free scanner called Deep Scan. So even the act of scoring how agent ready a site is has already been turned into open source with a free tool attached.

So across all 9 layers, every one already has someone in it, mostly platform giants or heavily funded startups. But that is not a reason to quit, because the openings are not in building yet another way to make a site agent ready. They are in the layers where agent behavior is already happening but nobody has solved it yet. For example: an agent has no cookies, no fingerprint, no browsing history, so how do you even market to it? An agent has no eyes to judge whether your banner looks good or bad, so how does it know?

Maybe the moat now is not making things people want, but making things agents want.