AuditZapBot: our website crawler

AuditZap checks websites for problems such as broken links, slow pages and missing tags. AuditZapBot is the crawler that does the checking.

It crawls a site when a customer asks for an audit of it. Each audit fetches the site's pages once, and a customer can choose a weekly re-audit of their sites.

• Audits use this user agent: Mozilla/5.0 (compatible; AuditZapBot/1.0; +https://auditzap.io)
• Our free tools use this one: Mozilla/5.0 (compatible; AuditZapToolBot/1.0; +https://auditzap.io)

The addresses we connect from:

• 178.104.89.143
• 2a01:4f8:1c19:b804::1
• 139.59.163.105

Allowing all of them covers every case. The first has the reverse DNS name crawler.auditzap.io.

Core Web Vitals are measured by Google PageSpeed Insights, which fetches the page from Google, not from these addresses. Our security page has more on how the crawler behaves.

• The HTML pages of your site, found by following links from the address it was given and from your sitemap.
• Your robots.txt, sitemap.xml and llms.txt files.
• The images, scripts, styles and fonts a browser loads for the pages it opens in a browser, because it measures those pages the way a visitor's browser sees them.
• At most 250 pages per audit, and fewer on smaller plans.

It opens pages in a browser one at a time. After the first few, it fetches the rest as plain requests, at most four at a time. If your robots.txt sets a Crawl-delay, it sends one request at a time and waits that long between them, up to 2 seconds. If your server starts answering with "too many requests" or "forbidden", it slows down, and then stops early.

If a security service shows our crawler a challenge page or blocks it, the audit cannot see your site. Tell your host, firewall or CDN to allow every address above, or the user agent.

Cloudflare

Open your site in the Cloudflare dashboard and go to Security > WAF > Custom rules. Create a rule with this expression:

ip.src in {178.104.89.143 2a01:4f8:1c19:b804::1 139.59.163.105}

Choose the action Skip, and tick managed challenges and Super Bot Fight Mode in the list of things to skip. Matching on the user agent containing "AuditZapBot" also works, but anyone can send that user agent, so the addresses are the safer match.

A simpler alternative is Security > WAF > Tools > IP Access Rules: add each address with the action Allow.

Agency customers can also create a crawler access key in AuditZap. Our crawler then sends it in the x-auditzap-key header, to your site only and over https only, and a Cloudflare rule that matches that header lets every audit through.

20i

Add each address to the site's allowed IP addresses, or ask 20i support to allow them.

Add these lines to your robots.txt:

User-agent: AuditZapBot
Disallow: /

AuditZapBot follows the rules in any robots.txt group addressed to it by name, including Allow and Disallow rules for single paths. If the page an audit starts from is blocked, the audit stops and says so.

It does not follow the rules under User-agent: *. Those are usually written for search engines, and an audit is asked for by the site's owner, who wants the whole site checked. To keep AuditZapBot out of all or part of your site, name it as above. These rules apply to audits. The free tools fetch at most 10 pages and do not read robots.txt rules.

Questions about the crawler, or want it to stop visiting your site? Email support@auditzap.io or use our contact form.