API quick start

From an empty terminal to a site profile in JSON. Every call below goes to https://api.datacrawly.io/v1; replace the sample key with your own. The API reference lists every endpoint and every field.

1. Create an API key

Keys are issued in your account: open Account → API keys, create one and name it after the thing that will hold it. The secret is shown once, on the screen where it is created, and never again. Put it wherever your other secrets live before closing that dialog. Lost it? Revoke the key and create another.

The key authenticates your organization, not your browser. Keep it on a server; anything shipped to a browser or a mobile app is public.

Open your account

2. Read a site profile

Authentication is a bearer token in the Authorization header on every request. The shortest useful call asks for one site:

curl https://api.datacrawly.io/v1/sites/brightsmile-dental.example \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"

A 200 is the profile: every technology with the versions seen and how sure we are of each (exact, likely, weak), the tracking IDs with how many other sites share them, the contacts the site publishes, the domains it links to, the pages checked and a timeline of what changed and when:

{
  "domain": "brightsmile-dental.example",
  "title": "Brightsmile Dental, Private Dental Clinic",
  "description": "Brightsmile Dental: private dental care, implants and orthodontics. Book online.",
  "lang": "en",
  "alive": true,
  "firstSeenAt": "2026-09-24T10:17:41.188Z",
  "lastCheckedAt": "2026-09-24T10:17:44.355Z",
  "pagesChecked": 2,
  "technologies": [
    { "slug": "wordpress", "label": "WordPress", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "7.1.2", "confidence": "exact" }] },
    { "slug": "wp-theme:astra", "label": "wp-theme:astra", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "4.13.12", "confidence": "weak" }] },
    { "slug": "wp-plugin:elementor-pro", "label": "wp-plugin:elementor-pro", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "4.2.2", "confidence": "likely" }] },
    { "slug": "wp-plugin:jetpack", "label": "wp-plugin:jetpack", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "16.1.3", "confidence": "likely" }] },
    { "slug": "jquery", "label": "jQuery", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "3.7.1", "confidence": "likely" }] },
    { "slug": "gtm", "label": "Google Tag Manager", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [] }
  ],
  "contacts": {
    "emails": ["[email protected]"],
    "phones": ["+44 20 0000 0000"],
    "socials": [
      { "platform": "facebook", "url": "https://facebook.com/brightsmile.example" },
      { "platform": "instagram", "url": "https://instagram.com/brightsmile.example" }
    ]
  },
  "tracking": [
    { "type": "ga4_measurement_id", "id": "G-EXAMPLE123", "sharedWith": 0 },
    { "type": "gtm_container_id", "id": "GTM-EXAMPLE", "sharedWith": 0 },
    { "type": "google_ads_conversion_id", "id": "AW-000000000", "sharedWith": 0 }
  ],
  "links": ["booking-partner.example", "studio-web.example"],
  "pages": [
    { "url": "https://brightsmile-dental.example/contact.html", "lastCheckedAt": "2026-09-24T10:17:44.355Z", "lastStatus": 200 },
    { "url": "https://brightsmile-dental.example/", "lastCheckedAt": "2026-09-24T10:17:40.547Z", "lastStatus": 200 }
  ],
  "timeline": [
    { "at": "2026-09-24T10:17:44.355Z", "change": "added", "kind": "email", "key": "", "value": "[email protected]" },
    { "at": "2026-09-24T10:17:44.355Z", "change": "added", "kind": "technology", "key": "", "value": "google-maps" }
  ]
}

Abridged: the live response for this domain carries 22 technologies and 48 timeline rows. A 404 means the domain is not in the index yet; a 401 means the header never arrived or the key is wrong.

3. Find similar sites

Sites that share a tracking ID with this one come first, then the ones with the most technologies in common:

curl "https://api.datacrawly.io/v1/sites/brightsmile-dental.example/similar?limit=10" \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"

{
  "items": [
    {
      "domain": "northgate-dental.example",
      "title": "Northgate Dental: Cosmetic Dentistry",
      "lang": "en",
      "alive": true,
      "technologies": [
        { "slug": "elementor", "label": "Elementor", "version": "4.3.1" },
        { "slug": "jquery", "label": "jQuery", "version": "3.7.1" },
        { "slug": "wordpress", "label": "WordPress", "version": "7.1.2" }
      ],
      "technologyCount": 19,
      "emails": 1,
      "phones": 2,
      "socials": 3,
      "tracking": 3,
      "firstSeenAt": "2026-09-24T10:17:44.339Z",
      "lastCheckedAt": "2026-09-24T10:17:50.136Z"
    }
  ]
}

Each row is the compact shape of a profile: up to eight technologies, and counts for emails, phones, socials and tracking IDs instead of the values. Read the profile of any row you care about.

4. Search the index

Search takes the same filters as the list builder on the site: technologies the site must have, may have or must not have, languages, what it has to publish (email, phone, social, tracking) and how recently it was checked. Every filter is optional:

curl -X POST https://api.datacrawly.io/v1/sites/search \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8" \
  -H "Content-Type: application/json" \
  -d '{
    "techAll": ["wordpress", "wp-plugin:elementor-pro"],
    "lang": ["en"],
    "has": ["email", "tracking"],
    "checkedWithinDays": 30,
    "sort": "technologies",
    "pageSize": 50
  }'

{
  "items": [ /* SiteRow, as in /similar */ ],
  "total": 173,
  "page": 1,
  "pageSize": 50,
  "facets": {
    "tech": [ { "value": "jquery", "label": "jQuery", "count": 171 }, /* … */ ],
    "lang": [ { "value": "en", "label": "English", "count": 173 } ]
  },
  "tookMs": 37
}

The result is one page of rows, the total, and two facets (technology and language) counted over the whole result, so a filter panel can be built from a single call. pageSize goes up to 100. The slugs for the technology filters come from GET /sites/catalog.

Use it from an AI agent

The same API, spoken over MCP instead of over HTTP, so an agent can look sites up in a conversation. The key is what authenticates it, exactly as above: an agent is a program holding a credential, not a session that signs in. Revoke the key the moment you are done; revocation stops it mid-conversation.

Claude Code, from the terminal

One command and the agent has the tools. Note the scope: without --scope user the server is registered for the directory you run it in, and an agent started elsewhere will not see it.

claude mcp add --scope user --transport http datacrawly https://api.datacrawly.io/v1/mcp \
  --header "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"

Claude Desktop, Cursor, and anything else that reads a config file

Most clients keep MCP servers in a JSON file: Claude Desktop in claude_desktop_config.json, Cursor in .cursor/mcp.json. Add this entry, with your own key in place of the sample:

{
  "mcpServers": {
    "datacrawly": {
      "type": "http",
      "url": "https://api.datacrawly.io/v1/mcp",
      "headers": {
        "Authorization": "Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"
      }
    }
  }
}

Restart the client after saving. The agent then lists four tools: get_site, find_similar, search_sites and get_catalog. Ask it what it knows about a domain and it will call them.

An agent without MCP support

Any agent that reads OpenAPI can use the API without MCP. Hand it the specification and your key, and it builds the calls itself:

https://api.datacrawly.io/v1/openapi.json

Where the key ends up

Both methods write the key in clear text into a configuration file on the machine the agent runs on: ~/.claude.json for the command above, and whichever file your client keeps its servers in. Treat that file the way you treat the key itself, and remove the server when you are finished: claude mcp remove datacrawly deletes the entry and the key with it.

Things worth knowing before the first bill

Domains are normalized

Pass a bare hostname. Scheme, path, port and a leading www. are stripped, the rest is lower-cased and an IDN is converted to punycode, so example.com, https://www.example.com/ and EXAMPLE.COM are the same profile.

Switch on code, never on message

Every error body is a code and a message. The code is the stable half of the contract; the message is written for the human reading a log and can be reworded in any release. Four are worth handling by name: not_found (the domain is not indexed yet), insufficient_credits (top up in your account), rate_limited (temporary; sleep on Retry-After) and unauthorized (the key is wrong, expired or revoked, and no retry will help).

What a call costs

Calls draw on the same credit balance as the rest of the product. A profile is one credit; similar sites and search cost one credit per site returned; the catalog is free. Reading the same domain again within 30 days is free, so polling a profile does not burn the balance.

Obey Retry-After, not your own timer

Sixty requests a minute per key. Past that you get a 429 with Retry-After, and that number is the one to sleep on; a fixed delay of your own either wastes time or walks straight back into the limit.