Create a key
Open Settings and press Create key. The key starts with dsdb_ and is shown once. We store only its hash.
Everything the dashboard does is available over HTTP: turn a website or a file into a dataset, wait for it, download the files, publish it, and search the public catalog.
# 1. Create a dataset from a website
curl -X POST https://api.datasetsdb.com/v1/projects \
-H "Authorization: Bearer $DSDB_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "max_pages": 100}'
# -> {"id": 42, "status": "queued", ...}
# 2. Poll until status is "done"
curl https://api.datasetsdb.com/v1/projects/42 -H "Authorization: Bearer $DSDB_KEY"
# 3. Download the Q&A pairs with confidence 0.8 and higher
curl -L "https://api.datasetsdb.com/v1/projects/42/export/qa.jsonl?min_conf=0.8" \
-H "Authorization: Bearer $DSDB_KEY" -o qa.jsonl
Create a key in the dashboard under Settings, API keys. Send it in the Authorization header. The catalog also works without a key, at a lower limit.
Open Settings and press Create key. The key starts with dsdb_ and is shown once. We store only its hash.
Authorization: Bearer dsdb_... on every request. Keep the key on your server; do not put it in browser code.
Regenerating gives a key a new secret and switches the old one off at once. You can hold up to 5 active keys and revoke any of them.
| Caller | Requests per minute | Pages per website | Projects |
|---|---|---|---|
| No key (catalog only) | 30 per IP | — | — |
| Free plan | 120 | 300 | 20 |
| Pro plan (planned) | 600 | 5,000 | 200 |
| Scale plan (planned) | 3,000 | 50,000 | 2,000 |
Every response carries X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. Up to 3 projects can run at the same time per account.
A project is one website crawl or one uploaded file. Paths are relative to https://api.datasetsdb.com/v1.
| Method | Path | What it does | Key |
|---|---|---|---|
| GET | /me | Your account, plan and limits. | required |
| GET | /projects | List your projects. | required |
| POST | /projects | Create a dataset from a website. JSON body: url, max_pages, max_depth, llm_qa, name. | required |
| POST | /projects/upload | Create a dataset from a file. Multipart form: file, name. | required |
| GET | /projects/{id} | Status, statistics, AI-readiness audit and the list of export files. | required |
| POST | /projects/{id}/cancel | Stop a queued or running project. | required |
| POST | /projects/{id}/rerun | Run the project again; current results are replaced. | required |
| DELETE | /projects/{id} | Delete the project and all its data. | required |
| GET | /projects/{id}/records | Records: kind (qa, chunk, row), q, flag, min_conf, limit, offset. | required |
| GET | /projects/{id}/pages | Crawled pages with their scores. | required |
| GET | /projects/{id}/export/{file} | Download an export. min_conf, mask_pii; all.zip returns every file. | required |
| POST | /projects/{id}/publish | Publish in the catalog. JSON body: description, license, task, tags, handle, slug, rights. | required |
| POST | /projects/{id}/unpublish | Remove the dataset from the catalog. | required |
| POST | /projects/{id}/verify-robots | Check the Dataset line in robots.txt of the source site. | required |
Read-only access to published datasets. Open to any origin, so it can be called from a browser.
| Method | Path | What it does | Key |
|---|---|---|---|
| GET | /datasets | Search public datasets. Filters: q, task, license, size, lang, kind, sort, limit, offset. | optional |
| GET | /datasets/{owner}/{name} | Dataset card: description, license, tags, row count, files, robots.txt verification. | optional |
| GET | /datasets/{owner}/{name}/rows | Rows of a dataset. subset is qa, chunk or row; q, limit up to 100, offset. | optional |
| GET | /datasets/{owner}/{name}/files/{file} | Download a file, for example qa.jsonl. Personal data is masked. | optional |
| GET | /suggest | Search suggestions for a prefix q: datasets, tags and tasks. | optional |
# No key needed for the catalog
curl "https://api.datasetsdb.com/v1/datasets?q=seo&sort=rows&limit=5"
curl "https://api.datasetsdb.com/v1/datasets/semalt/datasetsdb-com"
curl "https://api.datasetsdb.com/v1/datasets/semalt/datasetsdb-com/rows?subset=qa&limit=3"Errors are JSON with one field, detail. Status codes: 400 bad input, 401 missing or invalid key, 402 plan limit, 404 not found, 409 wrong state, 413 file too large, 429 rate limit.
HTTP/1.1 429 Too Many Requests
Retry-After: 37
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 0
{"detail": "Rate limit exceeded. Slow down and retry later."}Create a project, wait for it and read the result. A crawl of 100 pages usually takes a few minutes.
import os, time, requests
API = "https://api.datasetsdb.com/v1"
H = {"Authorization": "Bearer " + os.environ["DSDB_KEY"]}
project = requests.post(API + "/projects", headers=H,
json={"url": "https://example.com", "max_pages": 100}).json()
while True:
p = requests.get(f"{API}/projects/{project['id']}", headers=H).json()
if p["status"] in ("done", "failed", "canceled"):
break
time.sleep(5)
print(p["status"], p["audit"].get("score"), p["counts"])
qa = requests.get(f"{API}/projects/{project['id']}/export/qa.jsonl",
headers=H, params={"min_conf": 0.8}).textThe Free plan includes the API: 120 requests per minute and 300 pages per website.