API v1 REST, JSON, Bearer keys

Create AI datasets from code

Everything the dashboard does is available over HTTP: turn a website or a file into a dataset, wait for it, download the files, publish it, and search the public catalog.

Authentication

Keys and limits

Create a key in the dashboard under Settings, API keys. Send it in the Authorization header. The catalog also works without a key, at a lower limit.

Create a key

Open Settings and press Create key. The key starts with dsdb_ and is shown once. We store only its hash.

Send it as a Bearer token

Authorization: Bearer dsdb_... on every request. Keep the key on your server; do not put it in browser code.

Regenerate when needed

Regenerating gives a key a new secret and switches the old one off at once. You can hold up to 5 active keys and revoke any of them.

CallerRequests per minutePages per websiteProjects
No key (catalog only)30 per IP——
Free plan12030020
Pro plan (planned)6005,000200
Scale plan (planned)3,00050,0002,000

Every response carries X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. Up to 3 projects can run at the same time per account.

Endpoints

Projects and account

A project is one website crawl or one uploaded file. Paths are relative to https://api.datasetsdb.com/v1.

MethodPathWhat it doesKey
GET/meYour account, plan and limits.required
GET/projectsList your projects.required
POST/projectsCreate a dataset from a website. JSON body: url, max_pages, max_depth, llm_qa, name.required
POST/projects/uploadCreate a dataset from a file. Multipart form: file, name.required
GET/projects/{id}Status, statistics, AI-readiness audit and the list of export files.required
POST/projects/{id}/cancelStop a queued or running project.required
POST/projects/{id}/rerunRun the project again; current results are replaced.required
DELETE/projects/{id}Delete the project and all its data.required
GET/projects/{id}/recordsRecords: kind (qa, chunk, row), q, flag, min_conf, limit, offset.required
GET/projects/{id}/pagesCrawled pages with their scores.required
GET/projects/{id}/export/{file}Download an export. min_conf, mask_pii; all.zip returns every file.required
POST/projects/{id}/publishPublish in the catalog. JSON body: description, license, task, tags, handle, slug, rights.required
POST/projects/{id}/unpublishRemove the dataset from the catalog.required
POST/projects/{id}/verify-robotsCheck the Dataset line in robots.txt of the source site.required
Endpoints

Public catalog

Read-only access to published datasets. Open to any origin, so it can be called from a browser.

MethodPathWhat it doesKey
GET/datasetsSearch public datasets. Filters: q, task, license, size, lang, kind, sort, limit, offset.optional
GET/datasets/{owner}/{name}Dataset card: description, license, tags, row count, files, robots.txt verification.optional
GET/datasets/{owner}/{name}/rowsRows of a dataset. subset is qa, chunk or row; q, limit up to 100, offset.optional
GET/datasets/{owner}/{name}/files/{file}Download a file, for example qa.jsonl. Personal data is masked.optional
GET/suggestSearch suggestions for a prefix q: datasets, tags and tasks.optional

Catalog from the command line

# No key needed for the catalog
curl "https://api.datasetsdb.com/v1/datasets?q=seo&sort=rows&limit=5"
curl "https://api.datasetsdb.com/v1/datasets/semalt/datasetsdb-com"
curl "https://api.datasetsdb.com/v1/datasets/semalt/datasetsdb-com/rows?subset=qa&limit=3"

Errors

Errors are JSON with one field, detail. Status codes: 400 bad input, 401 missing or invalid key, 402 plan limit, 404 not found, 409 wrong state, 413 file too large, 429 rate limit.

HTTP/1.1 429 Too Many Requests
Retry-After: 37
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 0

{"detail": "Rate limit exceeded. Slow down and retry later."}
Example

A full run in Python

Create a project, wait for it and read the result. A crawl of 100 pages usually takes a few minutes.

import os, time, requests

API = "https://api.datasetsdb.com/v1"
H = {"Authorization": "Bearer " + os.environ["DSDB_KEY"]}

project = requests.post(API + "/projects", headers=H,
                        json={"url": "https://example.com", "max_pages": 100}).json()

while True:
    p = requests.get(f"{API}/projects/{project['id']}", headers=H).json()
    if p["status"] in ("done", "failed", "canceled"):
        break
    time.sleep(5)

print(p["status"], p["audit"].get("score"), p["counts"])
qa = requests.get(f"{API}/projects/{project['id']}/export/qa.jsonl",
                  headers=H, params={"min_conf": 0.8}).text

Start with a free key

The Free plan includes the API: 120 requests per minute and 300 pages per website.