For site owners
DataCrawlyBot
The crawler behind datacrawly.io. It reads public pages the way a search engine does, identifies itself on every request, obeys robots.txt and never runs your JavaScript. Here is exactly what it does and how to keep it out.
How to recognise it
Every request carries this User-Agent header. Search for it in your server logs or firewall.
Mozilla/5.0 (compatible; DataCrawlyBot/1.0; +https://datacrawly.io/bot)
We do not yet publish a fixed IP range; the User-Agent is the identification. If you see this string used in a way that contradicts this page, email us with the log line and we will check.
What it collects
DataCrawlyBot maps the public web: which technologies a site runs and in what versions, where it is hosted, what kind of business it is, the contact channels the business publishes on its own pages, and whether an observed version matches a publicly disclosed vulnerability. That index powers technology profiles, security notices and the search on datacrawly.io.
What we keep, for how long, and how to object is in the Privacy Policy. Privacy Policy
How it behaves
These limits are built into the crawler, not configured per customer.
- robots.txt
- Fetched for every origin before anything else and cached. Crawl-delay and Request-rate are honoured. A 401 or 403 on robots.txt means the whole site is off limits; a 429 or a server error means we wait.
- Rate
- At least one second between requests to the same origin, longer if your robots.txt says so. One connection at a time per origin. 429 and server errors back off exponentially and stop after three attempts.
- Scope
- The home page, the sitemap and a handful of detail pages such as about and contact: up to 30 URLs per host, two link levels deep. Public pages only, never behind a login.
- Size
- HTML, XML and plain text only, up to 4 MiB per response. It does not download images, video, PDF, fonts, scripts or stylesheets.
- Never
- No JavaScript execution, no cookies, no form submission, no logins, no CAPTCHA solving, no probing or exploitation of vulnerabilities. Requests to private or local network addresses are refused before they are made.
Why it never appears in your analytics
DataCrawlyBot fetches raw HTML and stops. It does not run scripts, so Google Analytics, Tag Manager, ad pixels and session-recording tools never fire. It sends and accepts no cookies, so it cannot create a session. Only your web server sees it, and there the User-Agent above is enough to filter it out of any report.
How to slow it down or block it
Add a group for DataCrawlyBot to your robots.txt. The crawler picks it up on its next visit to the site, usually within a day, and stops collecting from the pages you exclude.
Block the crawler from the whole site
User-agent: DataCrawlyBot Disallow: /
Keep it out of a section and slow it down
User-agent: DataCrawlyBot Disallow: /private/ Crawl-delay: 10
You can also ask us by email to remove a domain from the index and hide its public profile. Blocking the crawler does not by itself delete what was already collected; the email does.
Security notices
When an observed version matches a public advisory we may send one email to the address your site publishes for that purpose: the Contact line of /.well-known/security.txt (RFC 9116) or a role mailbox such as security@ printed on your pages. At most one notice per domain in 90 days, never a request for money or credentials, always with a report page and a one-click unsubscribe. The clearest way to tell us where to write is a security.txt file.
Contact
Questions about the crawler, removal requests, or a log line that looks wrong:
[email protected]