Documentation

How the index is built, how to read it, and how to get it out as JSON.

Developers

API quick start

A key, a first call, similar sites, search, and the same API over MCP for agents.

API reference

Every endpoint with its request, response shape, errors and cost in credits.

The crawler

What DataCrawlyBot does on a site, its user agent, robots.txt, and how to block it.

MethodologySoon

How we crawl, how technologies and versions are detected, how often, and how sure we are.

Reading a reportSoon

How to read a report: match levels, coverage, and what “not determined” means.

For site ownersSoon

Why we saw your site, what we keep, how to exclude a domain, responsible disclosure.