API reference

Paths are relative to the base URL below. Every response is JSON; dates are ISO 8601 in UTC and counts are plain numbers. If this is your first call, start with the quick start.

https://api.datacrawly.io/v1

Authentication

A bearer token in the Authorization header on every request. Keys are created in your account and belong to the organization, not the person; revoking one takes effect on the next request.

Authorization: Bearer <your key>

Errors

Errors are JSON with a stable code and a human message. Branch on the code; the message can change wording in any release.

{
  "code": "not_found",
  "message": "Domain is not in the index yet"
}

Status	Code	When
400	validation_failed	A parameter or body field is missing or of the wrong shape. The message names it.
401	unauthorized	No Authorization header, or a key that is wrong, expired or revoked.
402	insufficient_credits	The organization's balance cannot cover the call. Nothing was charged.
404	not_found	The domain is not in the index. Look it up on the site to request a crawl, or try later.
429	rate_limited	Over 60 requests a minute for this key. Retry-After says how long to wait.
503	service_unavailable	Temporary. The same request is worth sending again.

Credits and limits

Calls are metered in credits from the same balance as the rest of the product; the cost of each endpoint is listed next to it. A domain read in the last 30 days is free to read again. The rate limit is 60 requests a minute per key.

Endpoints

Site profile

GET/sites/{domain}

Everything the index holds about one site: technologies with versions, tracking IDs, contacts, outbound links, the pages checked and the change timeline.

Parameters: domain: a bare hostname, normalized as described in the quick start.
Cost: 1 credit; free again for 30 days.
Response: SiteProfile, or 404 not_found

Request

curl https://api.datacrawly.io/v1/sites/thedentalarchitect.com \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"

Response

{
  "domain": "thedentalarchitect.com",
  "title": "The Dental Architect, Private Dentist, Leeds City Centre",
  "description": "The Dental Architect, Leeds: Yorkshire's largest Invisalign provider & Dentist of the Year 2025. Private dental care. Book now.",
  "lang": "en",
  "alive": true,
  "firstSeenAt": "2026-09-24T10:17:41.188Z",
  "lastCheckedAt": "2026-09-24T10:17:44.355Z",
  "pagesChecked": 2,
  "technologies": [
    { "slug": "wordpress", "label": "WordPress", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "7.1.2", "confidence": "exact" }] },
    { "slug": "wp-theme:astra", "label": "wp-theme:astra", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "4.13.12", "confidence": "weak" }] },
    { "slug": "wp-plugin:elementor-pro", "label": "wp-plugin:elementor-pro", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "4.2.2", "confidence": "likely" }] },
    { "slug": "wp-plugin:jetpack", "label": "wp-plugin:jetpack", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "16.1.3", "confidence": "likely" }] },
    { "slug": "jquery", "label": "jQuery", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [{ "version": "3.7.1", "confidence": "likely" }] },
    { "slug": "gtm", "label": "Google Tag Manager", "firstSeenAt": "2026-09-24T10:17:40.547Z",
      "versions": [] }
  ],
  "contacts": {
    "emails": ["[email protected]", "[email protected]"],
    "phones": ["01138684324"],
    "socials": [
      { "platform": "facebook", "url": "https://facebook.com/thedentalarchitectdentist" },
      { "platform": "instagram", "url": "https://instagram.com/thedentalarchitect" }
    ]
  },
  "tracking": [
    { "type": "ga4_measurement_id", "id": "G-NHLQXYXMZX", "sharedWith": 0 },
    { "type": "gtm_container_id", "id": "GTM-MB9GLVZV", "sharedWith": 0 },
    { "type": "google_ads_conversion_id", "id": "AW-16486622207", "sharedWith": 0 }
  ],
  "links": ["cqc.org.uk", "expertmediatech.co.uk", "portal.dental", "scoreapp.com"],
  "pages": [
    { "url": "https://thedentalarchitect.com/contact.html", "lastCheckedAt": "2026-09-24T10:17:44.355Z", "lastStatus": 200 },
    { "url": "https://thedentalarchitect.com/", "lastCheckedAt": "2026-09-24T10:17:40.547Z", "lastStatus": 200 }
  ],
  "timeline": [
    { "at": "2026-09-24T10:17:44.355Z", "change": "added", "kind": "email", "key": "", "value": "[email protected]" },
    { "at": "2026-09-24T10:17:44.355Z", "change": "added", "kind": "technology", "key": "", "value": "google-maps" }
  ]
}

Similar sites

GET/sites/{domain}/similar

Sites that share a tracking ID with this one first, then the ones with the most technologies in common.

Parameters: limit: 1 to 50, default 10.
Cost: 1 credit per site returned.
Response: { items: SiteRow[] }

Request

curl "https://api.datacrawly.io/v1/sites/thedentalarchitect.com/similar?limit=10" \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"

Response

{
  "items": [
    {
      "domain": "vicidental.com",
      "title": "VICI Dental: Cosmetic Dentistry & Smile Makeovers",
      "lang": "en",
      "alive": true,
      "technologies": [
        { "slug": "elementor", "label": "Elementor", "version": "4.3.1" },
        { "slug": "jquery", "label": "jQuery", "version": "3.7.1" },
        { "slug": "wordpress", "label": "WordPress", "version": "7.1.2" }
      ],
      "technologyCount": 19,
      "emails": 1,
      "phones": 2,
      "socials": 3,
      "tracking": 3,
      "firstSeenAt": "2026-09-24T10:17:44.339Z",
      "lastCheckedAt": "2026-09-24T10:17:50.136Z"
    }
  ]
}

Search

POST/sites/search

A filtered, sorted page of sites, with technology and language facets counted over the whole result.

Parameters: All body fields are optional. techAll, techAny and techNone take technology slugs from the catalog; has takes email, phone, social and tracking and requires all of them; sort is lastChecked, firstSeen, domain or technologies; pageSize is at most 100, default 50.
Cost: 1 credit per site on the page.
Response: SiteListResult

Body

type SiteQuery = {
  q?: string;                     // substring of the domain or title
  techAll?: string[];             // has every one of these
  techAny?: string[];             // has at least one
  techNone?: string[];            // has none
  lang?: string[];                // any of
  has?: ('email' | 'phone' | 'social' | 'tracking')[];   // all of
  checkedWithinDays?: number;
  sort?: 'lastChecked' | 'firstSeen' | 'domain' | 'technologies';
  sortDir?: 'asc' | 'desc';       // domain ascending, the rest descending
  page?: number;                  // from 1
  pageSize?: number;              // up to 100, default 50
};

Request

curl -X POST https://api.datacrawly.io/v1/sites/search \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8" \
  -H "Content-Type: application/json" \
  -d '{
    "techAll": ["wordpress", "wp-plugin:elementor-pro"],
    "lang": ["en"],
    "has": ["email", "tracking"],
    "checkedWithinDays": 30,
    "sort": "technologies",
    "pageSize": 50
  }'

Response

{
  "items": [ /* SiteRow, as in /similar */ ],
  "total": 173,
  "page": 1,
  "pageSize": 50,
  "facets": {
    "tech": [ { "value": "jquery", "label": "jQuery", "count": 171 }, /* … */ ],
    "lang": [ { "value": "en", "label": "English", "count": 173 } ]
  },
  "tookMs": 37
}

Catalog

GET/sites/catalog

Every technology and language in the index with its site count: the vocabulary for the search filters.

Parameters: None.
Cost: Free.
Response: SiteCatalog

Request

curl https://api.datacrawly.io/v1/sites/catalog \
  -H "Authorization: Bearer dck_4kQ2mP7xR9tA9f1c0a4e2b6d83e5a7c1f0b9d2e4a6c8"

Types

The shapes above, as TypeScript. Dates are ISO strings; a null means the index has not seen the value.

type TechRef = { slug: string; label: string; version: string | null };

type SiteRow = {
  domain: string;
  title: string | null;
  lang: string | null;
  alive: boolean;                 // an open page answered 200
  technologies: TechRef[];        // up to 8, the ones with a version first
  technologyCount: number;
  emails: number; phones: number; socials: number; tracking: number;
  firstSeenAt: string;            // ISO 8601, UTC
  lastCheckedAt: string | null;
};

type Facet = { value: string; label: string; count: number };

type SiteListResult = {
  items: SiteRow[];
  total: number;
  page: number;
  pageSize: number;
  facets: { tech: Facet[]; lang: Facet[] };   // over the whole result, up to 60 each
  tookMs: number;
};

type SiteCatalog = {
  total: number;
  technologies: Facet[];          // every technology, by count
  languages: Facet[];
};

type SiteProfile = {
  domain: string;
  title: string | null;
  description: string | null;
  lang: string | null;
  alive: boolean;
  firstSeenAt: string;
  lastCheckedAt: string | null;
  pagesChecked: number;
  technologies: {
    slug: string; label: string; firstSeenAt: string;
    versions: { version: string; confidence: 'exact' | 'likely' | 'weak' | null }[];
  }[];
  contacts: {
    emails: string[]; phones: string[];
    socials: { platform: string; url: string }[];
  };
  tracking: { type: string; id: string; sharedWith: number }[];  // sharedWith: other sites with the same id
  links: string[];                // outbound domains
  pages: { url: string; lastCheckedAt: string; lastStatus: number }[];   // up to 50, newest first
  timeline: {
    at: string; change: 'added' | 'removed';
    kind: string; key: string; value: string;
  }[];                            // up to 200, newest first
};

OpenAPI

The same contract as an OpenAPI 3 document, for code generators and agents:

https://api.datacrawly.io/v1/openapi.json