Server rendered HTML
Many agents run no JavaScript. What only comes into being in the browser does not exist for them.
AI agents
Who will be researching on your website.
A growing share of requests comes neither from people nor from classic crawlers, but from agents looking something up on behalf of a person.
Categories
Three kinds of visit.
The classic crawler indexes regularly and broadly. It collects for later and takes no interest in any single question.
The training collector gathers content for building models. Its visit takes effect months later and its impact can barely be measured.
The agent on assignment fetches exactly one page right now, because somebody wants to know something right now. Its visit is the most valuable and the least patient: anything slow to load, or appearing only after scripts run, drops out of the answer.
In robots.txt those three can be treated separately, and that is exactly what we recommend. A blanket rule treats the visit that brings you customers the same as the one that trains on you.
Requirements
What an agent needs.
Speed
Time limits are tight. A slow page gets dropped rather than waited for.
Unambiguous details
Price, availability, opening hours. An agent extracts, it does not interpret charitably.
No barriers
Bot defences, consent banners and login walls end the visit. Often unnoticed.
Practice
What can be tested.
The simplest test is a fetch without a browser. If the essential details are missing from your page's raw text, they are missing for the agent too.
The second test runs in the server log. The known identifiers of the AI providers turn up there, and their ratio to human requests is a number almost nobody looks at.
The third test is the answer itself: ask questions, look at the sources, check whether the cited detail is right. That is the work we do systematically in the audit.
Common questions
About AI agents.
How do I recognise agents in the server log?
By the identifier in the user agent. The large providers publish theirs, and they distinguish between training, search index and retrieval on assignment.
Should AI crawlers be blocked?
Decide separately. Blocking training collectors is a legitimate choice. Blocking retrieval on assignment costs you mentions in front of people who are about to buy.
Do the providers honour robots.txt?
The large ones publish their identifiers and state that they do. It cannot be verified fully, which is why sensitive content belongs behind a login rather than behind a request.
Why is JavaScript a problem?
Because many agents fetch only the raw HTML. A price appearing after load is not there for them. Server side rendering solves it.
Do agent requests distort analytics?
Yes, in both directions. They produce requests without people and at the same time mentions without requests. Both belong separated in the evaluation.
How fast does a page have to be?
No provider states concrete limits. In practice: what is fast for people is fast enough here. What takes seconds is at risk.
What about consent banners?
A banner releasing content only after a click blocks agents completely. The content has to sit in the HTML, and the banner must not replace it.
Can I serve agents deliberately?
Yes, with clean markup, structured data and a Markdown version per page. We ship the latter on every page of this website.
Is the share of these requests growing?
In what we observe yes, with large differences by sector. There are no reliable general figures, and your own logs are the better source.
Do we need our own interface for agents?
In most cases no. A clean website is the interface. A dedicated one gets interesting only when agents are meant to act with you rather than read.
Next step
Let us find out where you stand in AI answers.
One conversation, 30 minutes, no sales pressure. You leave with an honest read on whether this is worth the effort for you.
