About the VeloCMS Site Audit crawler

Who we are

VeloCMS is a blogging and website platform. Our free Site Audit tool lets anyone submit a URL and get a report on technical SEO, performance, platform detection, and migration readiness. When someone submits your URL, we fetch it — and, for a deeper scan, a bounded number of additional pages we discover from your sitemap and internal links.

How to identify our requests

Every request we send carries this exact User-Agent string:

The URL in that string always points back to this page.

What we fetch, and how

A single-page check fetches only the one URL a visitor submits — no login, no crawling.

A deeper scan, available once a site owner verifies ownership, may also fetch pages discovered from your robots.txt sitemap directives and the links on pages we've already fetched, up to a link-depth of 3 from the submitted page.

• Per-page limits — 10-second timeout, 5 MB response cap, at most 3 redirects, standard 80/443 ports only.
• Pacing — at most 4 requests in flight at once, with a 400 ms delay between requests to the same host (roughly 2.5 requests per second to your server).
• Scope — a single-page check reads at most 100 pages within about 2 minutes; a verified deeper scan reads at most 250 pages within about 7 minutes.
• Frequency — we scan a given hostname at most once every 24 hours.
• Retention — we never store the HTML we fetch. Only a small set of extracted signals (word counts, headings, links, and similar) survive past the request.
• Safety — every request, including redirects, is checked so we never fetch a private or internal network address.

robots.txt compliance

Beyond the single page a visitor submits, every additional page we discover is checked against your robots.txt before we fetch it. We look for a group addressed to our token, VeloCMS-SiteAudit, and fall back to the wildcard (*) group when you have none naming us specifically. Any page your robots.txt disallows for us is discovered but never fetched.

How to stop us

Add the following to your robots.txt to block every page beyond the one a visitor explicitly submits:

User-agent: VeloCMS-SiteAudit
Disallow: /

This stops us from following your sitemap or internal links to discover more of your site. It does not stop a single explicit request — if someone submits one specific page from your site, we fetch that one page, the same way a person visiting it in a browser would. We don't offer a separate opt-out mechanism, such as a meta tag, today.

Contact

Questions, or think we're behaving badly on your site? Email us: [email protected]