TidywrightBot

• It runs only when a person asks for an audit. There is no scheduled crawl and no discovery queue. One request from a visitor means one site is looked at.
• One audit is at most 30 requests to your site. That is the whole promise, worst case, and it is a ceiling the code enforces by stopping rather than a number we aim at. It went up from an earlier, lower figure because an audit can now include a second pass in a real browser on top of the plain one, and we would rather publish a ceiling that covers both than one we might cross. Where it goes:
• The usual audit is one pass without a browser, and it is at most 10 requests. It fetches the page itself, /robots.txt, /sitemap.xml and at most one resource the page names, which is the image it nominates for sharing. Redirects and a second try at /robots.txt are requests too, and they are counted. So is robots.txt of any other address a redirect leads to, which we read before following it. On our own test page the whole pass stays at half of that ceiling.
• When the page arrived fine but cannot be read without JavaScript, we open a browser once, and that pass is at most 20 requests. The browser loads the page and the files the page needs to run, and nothing else: no images, no fonts, no video, and nothing at all from any other host, including your CDN. It never goes to the network itself: every request it wants is sent by us, the same way as the plain pass. Crossing that ceiling cancels the render: we would rather deliver a thinner report than spend more of your server than we said we would.
• We never open a browser after your site refused us: not after a 401, 403 or 429, a server error, a challenge page or a failed connection. A refusal is an answer.
• The worst case is both passes on one audit, which is the browser pass after the plain one. Even then the two together cannot cross the 30 requests above, because both draw from one budget for the audit and the audit stops when it is spent.
• A site audit reads up to 10 pages of one site, and it is at most 50 requests to your site. It starts with your home page, read exactly as described above, and the home page is the only page for which we may open a browser. The other pages are the addresses your sitemap lists, in the order it lists them; when your sitemap is an index of sitemaps, up to four of the sitemaps it names are read, in the order it lists them. Then, if the sitemap does not give 10 pages, the links on your home page, in the order they appear, and links on those pages are not followed. We take only addresses on the same host as your home page, without a query string, and never files such as PDFs or images. Each of those pages is one plain request, plus its redirects, and every one of them is counted. Every address is checked against your robots.txt before it is sent, and waits the same gap as every other request. The 50 requests are the whole site audit, the home page included, and when they are spent the audit stops asking.
• It never walks a site beyond those 10 pages, in either pass, and never sends more than 1 request per second to your site within one audit. If your robots.txt asks for a longer Crawl-delay, we wait that long instead. Every request our browser makes waits for that gap as well.
• The report goes to the person who asked for the audit, and, if they asked through an agency's audit form, to that agency too. Anyone can ask for an audit of any public address, so that person is not necessarily the owner of the site.
• To draft suggested fixes, part of the page goes to the API of Anthropic, the language model provider our code uses by default: its address, title, description, headings, site name and the first 1,500 characters of its visible text. Anthropic's published terms say it does not use API inputs to train its models by default, and deletes them within 30 days unless they are flagged under its usage policy or must be kept by law. We do not sell what the audit reads.

Every request we send carries this User-Agent, unchanged, in the plain pass and in the browser alike, on every plan and every route:

Mozilla/5.0 (compatible; TidywrightBot/1.0; +https://tidywright.com/bot)

Every request we send is signed, in the plain pass and in the browser, redirects included: each carries an RFC 9421 HTTP Message Signature (Web Bot Auth), made for the host it goes to. You can verify us cryptographically rather than trusting a string anyone could copy. Our public key is published as a JSON Web Key Set at:

https://tidywright.com/.well-known/http-message-signatures-directory

Every address we own is listed here and in JSON at https://tidywright.com/bot/ips.json. Each one has reverse DNS that resolves into our domain and forward-confirms back to the same address. If a request claims to be us and its address is not on this list, it is not us.

Every request an audit sends, in the plain pass and in the browser, leaves from the fetcher's address above, over IPv4.

Put this in your robots.txt:

User-agent: TidywrightBot
Disallow: /

This is exactly what we do with that file:

• We read /robots.txt of every address we are about to ask, once per audit: yours, and again at any other scheme, host or port a redirect leads to, before we follow it. We keep no copy from one audit to the next.
• We follow the group that names TidywrightBot. If no group names us, we follow the group for all robots, User-agent: *.
• Before every request, the page, each redirect, the sitemap, the share image and everything the browser asks for, we check its exact path against those rules. The longest matching rule wins, and Allow wins a tie.
• If the rules say no to the page, we send nothing more: no page, no sitemap, no image, no browser, and the report says that robots.txt stopped the audit. If they say no only to the sitemap or the image, we skip that one request.
• If robots.txt answers with a 4xx status other than 429, or with a web page instead of a robots file, we treat that as having no file and go ahead.
• If it times out, the connection fails or it answers with a 5xx status, we try once more a second later. If we still cannot read it, for any reason other than the ones above, we treat everything at that address as disallowed. For your page's own address that means we send nothing more.
• We honour Crawl-delay: the gap between our requests to your host is the longer of one second and your Crawl-delay. If it asks for more than ten seconds, we do not audit the page at all, and the report says why.

If you would rather block us at the edge, a rule on the published addresses, on the User-Agent above or on our signature stops every request we send, in both passes. When you refuse us, we do not try again in any other way.

This is a policy, not a preference, and it is the reason you can treat the identity above as reliable:

• We do not forge TLS or HTTP/2 fingerprints.
• We do not use stealth patches to hide that a page was opened by automation.
• We do not solve CAPTCHAs, and we do not pay anyone to solve them.
• We do not use residential or rotating proxies. Our addresses are ours and published.
• We do not change identity after being blocked. A block is an answer.

When a page will not render without JavaScript, we open it in a browser, Chromium, to run its scripts. The browser cannot reach the network by itself: every request it makes is sent by us, from our address, under the name and the signature above. We say so here rather than hiding it.

Questions, allowlisting, or a request that we stop: bot@tidywright.com. Abuse or anything that looks like us behaving badly: abuse@tidywright.com. Both are read by a person.

Operator: Tidywright. Declared category: SEO.