Robots.txt and trade publishers
Helen King put together a useful summary of robots.txt policies across major academic publishers, as compared with the news media—as a snapshot, some 53% of academic sites blocked at least one AI bot, compared to 86% of news sites. I was curious to see how this compared with trade publishing and asked an AI agent to look at the top 20 trade publishers: you can download the results here. Across 27 UK and US domains, only 19% blocked at least one bot.
This is five minutes of raw research, and the agent’s quick analysis makes a good point that there are some rational reasons for a trade house to approach this differently. Incidentally, guess which one of the Big Five gets a gold star for being the only publisher to specify Universal Commerce Protocol (UCP) and Model Context Protocol (MCP) endpoints for agentic browsers and shopping.