NodionBot

NodionBot is the web retrieval of the Nodion AI Gateway. It loads a page when a Nodion customer calls the web_fetch tool, or asks the web_search tool to load the content of its search results. Every fetch is triggered by a customer request in real time. NodionBot does not crawl the web on its own and does not build an index, and Nodion does not use the pages it fetches to train its own AI models.

How to recognize NodionBot

NodionBot sends this user agent with every request, including the request for your robots.txt:

NodionBot/1.0 (+https://nodion.com/bot)

It never presents itself as a browser or as another bot. If your site answers with 401, 403, 429 or a bot challenge, NodionBot does not retry under a different identity and does not try to get around the block. The customer gets your answer instead of the page: its status code and the content you returned, converted to text and possibly shortened, unless a tdm-reservation applies.

Control access with robots.txt

NodionBot reads /robots.txt on each host before it fetches a page there, and again for every host a redirect leads to. It follows the Robots Exclusion Protocol (RFC 9309). A group for NodionBot takes precedence; without one, the * group applies. To block NodionBot from your whole site:

User-agent: NodionBot
Disallow: /

To keep it out of part of the site only:

User-agent: NodionBot
Disallow: /account/
Disallow: /drafts/

NodionBot only requests robots.txt when a customer request needs it, and at most once every 24 hours per origin (scheme, host and port) while the file can be read, so changes can take that long to apply. A missing robots.txt (404), or another 4xx answer except 429, means no restrictions, and that result is kept for 24 hours as well. If robots.txt cannot be retrieved (a 5xx or 429 answer, a network error or a timeout), NodionBot keeps using the last result it received successfully for up to 30 days and asks again at most every 10 minutes; without such a result, it does not fetch from your site and asks again at most every 5 minutes. A request for exactly /robots.txt (without a query) skips the robots.txt and Content-Signal check, as RFC 9309 requires; a domain block and tdm-reservation still apply.

Content-Signal: ai-input

NodionBot fetches pages as input for AI answers, which is what the ai-input signal of the Content Signals Policy describes. If the robots.txt group that applies to NodionBot sets ai-input=no, NodionBot does not fetch the page, even where the path is otherwise allowed:

User-agent: *
Content-Signal: search=yes, ai-input=no
Allow: /

A signal can also be limited to a path, for example Content-Signal: /docs/ ai-input=yes. The most specific matching path wins.

tdm-reservation

If a page is served with the response header tdm-reservation: 1, or contains <meta name="tdm-reservation" content="1"> in its head, NodionBot does not pass its content on to the customer. A meta element takes precedence over the header.

Block your whole domain

If you prefer not to rely on robots.txt, you can ask us to block your domain for NodionBot. A blocked domain includes all of its subdomains. Write to team@nodion.com and name the domain. We may ask you to show that you control it, for example by sending the request from an address at that domain or by adding a DNS record we give you.

Contact

Questions about NodionBot, or a request from it that looks wrong? Contact us at team@nodion.com. Please include the time of the request, the URL and, if you have it, your server log line.