A little more than one request per second. Every second, through the night, over the weekend, on public holidays. That is what roughly 100,000 requests a day looks like when they all come from one client, and that is what we found one morning in the logs of a data service we help run for a client.
My first instinct was to block it. It was the wrong instinct, and I had it anyway.
The client was an AI agent. It used a documented endpoint, asked for data that exists to be used, and broke no rule. Every single request was legitimate. The behaviour as a whole was absurd. It took me a few days to accept that the absurd part was ours.
Every request correct, the behaviour wrong
During my PhD I worked on automated program repair, teaching machines to fix bugs. The field has a word for a patch that passes every test and is still wrong: plausible. Plausible patches are why you cannot judge a fix by its test results alone. The tests encode what their author remembered to check. Nothing more.
The agent was the same problem turned inside out. Each request passed every check our interface had. The interface just had very few checks, because it encoded what we had remembered to expect from a consumer, and we had only ever expected people.
Look at what the agent had to work with. It could ask for the current values. It had no good way to ask what had changed since its last visit, nothing clearly told it when the data refreshes, and nothing marked which values are final and which may still move. So it did the one reasonable thing left to a careful consumer without a clock. It kept asking.
Try the arithmetic as a thought experiment. Suppose the underlying data changes ten times a day. Then 99,990 of those 100,000 answers were identical to the one before. The agent was not really reading our data. It was reading our silence about time, over and over, at a little over one hertz.
Boredom was our rate limiter
The uncomfortable part is that people do exactly the same. Anyone who has refreshed an exam results page, a flight status or a bank transfer knows the gesture. We poll too. We just get bored after the fifth refresh, make a coffee and come back in an hour.
For thirty years a good part of the web has quietly relied on that. Boredom limited request rates. Sleep gave us night-time maintenance windows. Caching worked because humans mostly look at the same few popular pages. Reading the documentation once, or phoning the person who owns the data, settled questions no API ever answered. Nobody designed any of this. It came free with human consumers, and we built on it without writing it down.
Agents remove all of it at once. They do not get bored, they do not sleep, and they are evenly curious about everything. The Wikimedia Foundation found this out the expensive way. By April 2025 its bandwidth for multimedia downloads had grown 50% since the start of 2024, and bots produced at least 65% of its most expensive traffic while accounting for about 35% of page views. Bots don't stick to the popular pages, so they miss the caches and land on the core data centre. Cloudflare's 2025 review counted more than fifteen-fold growth, in a single year, in crawling done by AI assistants on behalf of a user. That is exactly our agent's category.
So when I read BARC's spotlight report from June 2026, A Data Marketplace Is What Your Agents Need, the number that stayed with me was an old one. In a 2024 BARC survey, 73% of respondents said relevant data is hard to find. That was a human problem long before agents arrived. Analysts paid for it in lost afternoons and phone calls, and nobody ever sent an invoice. Agents did not create the mess. They are the first consumers whose confusion shows up on the bill.
A disclosure, because it matters: the report was sponsored by Entropy Data, a company that sells data marketplaces. That doesn't make it wrong. It means I read it twice, and I disagree with it in a couple of places below.
A warehouse with the door open
Every town in Vojvodina has a pijaca, an open-air market. I keep coming back to it as the most useful picture of what was missing.
Our service, like most data services, works like a warehouse with the door left open. Anyone can walk in, take what they want and leave. In the morning the owner sees footprints and has to guess who came, what they took and why.
A market works differently, and the difference is not the produce. Every stall has a person behind it, someone you can find again tomorrow. The goods carry a price and an origin. There is a scale, and somebody from the municipality checks the scale. There are rules that apply to every stall, from hygiene to where you may leave the cart. And there are opening hours, painted on the gate.
Nobody knocks on that gate at three in the morning to ask whether the tomatoes have arrived. Not because market customers are unusually polite, but because the hours are posted.
Translate the market to data and you get, almost item by item, what the data mesh community calls a data product and what vendors call a marketplace: a named owner for each dataset, a description of what it means and how good it is, a promise about freshness, rules that hold for everyone, and something that checks each transaction against them.
This changed how I look at the service from our story. I no longer think of it as an API with a traffic problem. It is a market that never put up its stalls. The goods are there, the sellers are there, and the customers have clearly arrived. What's missing is the sign on the gate, the labels and the inspector. That is a much smaller step than "build a data marketplace" sounds, and I suspect a lot of public data services are standing in the same place without knowing it.
Twenty years of documentation for nobody
The data world calls the label a data contract. In practice it is a YAML file, these days usually written in the Open Data Contract Standard (ODCS) from the Bitol project under the Linux Foundation. It says what a dataset is, who owns it, how fresh it is, which quality checks it must pass and what you may use it for. A simplified excerpt would have fixed most of our week:
# simplified, ODCS-style
id: daily-aggregates
status: active
description:
purpose: Final aggregated values, refreshed once per business day.
limitations: Values before 10:00 CET are provisional.
slaProperties:
- property: frequency
value: 1
unit: d
- property: timeOfAvailability
value: "10:00 CET"An agent that reads "once a day, at ten, earlier values are provisional" has no reason to ask a hundred thousand times. It asks once, after ten. A sentence of metadata is the cheapest rate limiter I know.
Here is the part I have not seen discussed much. For twenty years most of us wrote documentation that nobody read. People skimmed it, guessed, and then emailed the one colleague who knew. That colleague was the real documentation. Now our documentation has a reader who reads every word, takes it literally and cannot email anyone.
That is a gift, and it is also a threat. A missing freshness field made our agent poll. Expensive, but every answer it used was current. A wrong freshness field is worse. If the contract says ten o'clock and the data actually lands at half past eleven, a well-behaved agent asks once at ten, gets yesterday's numbers and uses them with complete confidence until tomorrow. Missing metadata costs you money. Wrong metadata quietly costs your consumers the truth.
So the contract has to be tested like code, in the pipeline, every time the data moves. Open tools such as the Data Contract CLI do exactly that. An untested contract is a lie with a schema.
Purpose is a claim, not a fact
Most writing on this topic, the BARC report included, makes a fair point about access. It should not be a badge you receive once and keep forever. Approvals pile up, data drifts into uses nobody approved, and with agents the drift is fast and invisible. The usual remedy is purpose-based access: every request states why it wants the data, and that purpose is checked against the contract. Entropy Data's MCP server, for example, refuses to run a query without one.
I agree with the direction. I don't believe the declaration alone.
We have seen this film before. Cookie banners were meant to capture informed consent and became a click everyone makes without reading. Declared purpose will go the same way. Within a year agents will learn which word opens which door, and every request will say "research". A purpose field that only checks spelling is theatre.
What an agent says is a claim. What it does is evidence. An agent that declares "monthly reporting" and asks every second is not doing monthly reporting. It is monitoring something, quite possibly to feed a commercial product. The control worth building compares the declared purpose with the observed pattern and acts on the gap.
In our case we could not even get that far. We knew where the requests came from and how often. We did not know who was behind them, what they were building, or whether they would have happily switched to something cheaper had anyone been able to suggest it. The only tool left was the blunt one, a rate limit. It works, but it treats a possibly valuable consumer as an attacker, and it teaches the next agent nothing.
Trust is not enough. Markets need prices
BARC has a line I like: trust is the market mechanism that is missing in many failed marketplace implementations. It's true, but it's half the story. Plenty of internal data marketplaces of the last decade died with decent metadata. Nothing in them cost the consumer anything, and nothing in them earned the owner anything. A market without prices is a catalogue with a nicer front end.
Agents are bringing prices back, whether anyone planned it or not. Each of those 100,000 daily requests cost something in compute, bandwidth, logs and on-call attention, and all of it was absorbed by the side serving the data. On the open web the imbalance is spectacular: across 2025, Cloudflare measured Anthropic's crawlers making somewhere between 25,000 and 100,000 requests for every visit they sent back. In July 2025 Cloudflare launched pay per crawl, built on HTTP status 402, Payment Required. That code had been "reserved for future use" since 1997. It took almost thirty years and a few billion crawler requests for somebody to need it. In Europe, the Data Act has applied since September 2025 and treats business-to-business data sharing as something done on contract terms, with reasonable compensation where the sharing is mandatory.
This leaves a question I don't have a clean answer to. Much public data should stay free, and I would argue for that loudly. But somebody pays for every request. When a publicly funded service answers an agent that is building a commercial product, the taxpayer is subsidising someone's agent at one request per second. Maybe that's fine. Maybe it's exactly what open data is for. But it should be a decision, not something nobody noticed.
Ostrom would have built a marketplace
The usual frame here is Garrett Hardin's tragedy of the commons. Shared pasture, every herder rationally adds one more cow, the pasture dies. One agent polling once a second is fine. A thousand agents doing it is an outage, and each of them is behaving rationally.
Elinor Ostrom received the 2009 Nobel Prize in economics largely for showing that Hardin's ending is not inevitable. She studied real commons, from Swiss alpine pastures to irrigation systems in Spain and the Philippines, found communities that had kept shared resources alive for centuries, and distilled what they had in common. Read her design principles today and they sound like a requirements document. Clear boundaries: consumers have names. Rules that fit local conditions: a contract per data product, not one policy for everything. Monitoring that somebody actually looks at. Graduated sanctions instead of instant bans. Cheap ways to resolve conflicts, which in practice means an owner who answers. And nested layers of rules: organisation-wide policies, with each domain's contracts inside them.
I find this more convincing than any vendor architecture diagram, mostly because it isn't about software. The marketplace is not the platform. It is the set of rules that lets a shared resource survive heavy use, and the platform only helps enforce them.
A plug is not a policy
A word on the Model Context Protocol, since every conversation about agents and data ends up there. MCP makes it trivially easy to give an agent a tool that talks straight to a database. Done that way, it's the open warehouse door again, with better signage. Done properly, MCP is the gate of the market. Simon Harrer of Entropy Data describes the marketplace as a layer that every agent call passes through, the query included, so contract and policy are checked on the way in. The protocol is the same in both cases. The difference is whether the governance lives behind it or in a prompt.
And a word against my own trade. Be careful with anyone who says step one is a platform, vendors included, and us included. Most of what our incident needed fits in a YAML file, three HTTP headers and one conversation about who owns what.
What I would do on Monday
- Count your machines. Find out what share of your traffic is automated and which ten clients produce most of it. We were surprised. Most teams are.
- Post the opening hours. Say when the data refreshes and when it becomes final. Offer a change feed or a daily snapshot, and support
ETagandIf-Modified-Since, so that "nothing changed" costs a 304 and almost nothing else. - Write one contract, and test it. Take the most requested dataset, describe it in ODCS and run its checks in the pipeline. An untested contract is worse than none.
- Give heavy consumers a name. A free key is enough. Ask for a purpose, then compare it with what the key actually does.
- Replace the ban with a ladder. When someone asks too often, answer with a
429, aRetry-Afterand a link to the bulk file. Tiers come before blocks. - Decide who pays. Before the cloud bill decides for you.
The agent never broke a rule. It just never got bored, and so it found every place where we had relied on boredom instead of design. At the pijaca nobody knocks on the gate at three in the morning. The hours are painted on it.
Sources
- Florian Bigelmaier, BARC: BARC Spotlight: A Data Marketplace Is What Your Agents Need, June 2026, sponsored by Entropy Data
- Jochen Christ: From Data Catalog to Data Marketplace
- Simon Harrer: A Data Marketplace is What Your Agent Needs, DATA Festival 2026
- Entropy Data: MCP: Connect AI Clients with Business Data
- Bitol: Open Data Contract Standard; Data Contract CLI
- Wikimedia Foundation: How crawlers impact the operations of the Wikimedia projects, April 2025
- Cloudflare: Radar 2025 Year in Review; Introducing pay per crawl
- European Commission: Data Act
- Elinor Ostrom: Governing the Commons, Cambridge University Press, 1990
MALI Labs is a software and AI engineering company based in Ada, Vojvodina, Serbia. We help teams make their data usable for AI agents without losing control of it: data contracts, governed MCP interfaces and agent-ready APIs. If your service has started seeing more machines than people, talk to us. The first proof of concept, one week on your own data, is free.
