Crawler

DatasetsDBBot

If you found this page in your server logs, this is who we are, what our crawler does on your site and how to stop it.

User agent

DatasetsDBBot/1.0 (+https://datasetsdb.com/bot)
Rate
About 4 parallel requests, with a short pause
Fetches
Public HTML, robots.txt, sitemaps, llms.txt
Runs JavaScript
No

What it does

DatasetsDB turns websites into clean datasets and AI-readiness reports. The crawler does not roam the web on its own. It visits a site only when a signed-in DatasetsDB user asks for that site to be analysed, and it stays on that host.

  • It fetches public HTML pages only, up to the page and depth limits the user set. Other file types and responses larger than 3 MB are skipped.
  • It reads robots.txt, your XML sitemaps and /llms.txt if present.
  • It does not execute JavaScript, submit forms or follow links that require a login. Content behind a login wall is never fetched.
  • It uses about 4 parallel requests with a short pause between them, so a typical crawl adds little load.
  • Every record we produce keeps the URL it came from.

It obeys robots.txt

Before fetching any page, DatasetsDBBot downloads your robots.txt and skips every path it is not allowed to visit. It follows rules for DatasetsDBBot first and falls back to the rules for *.

The Dataset line in robots.txt

When you publish the dataset of your site in our catalog, add one line to robots.txt that points to it:

Dataset: https://datasetsdb.com/datasets/your-name/your-site

We ask every site owner to do this. The line tells crawlers where the machine-readable version of the site lives: question-answer pairs, retrieval chunks and clean page text with a source on every record. It also proves ownership: when DatasetsDBBot finds the line, the dataset page gets a "Verified by site" badge.

Dataset is a convention we propose, not part of the Robots Exclusion Protocol. Crawlers that do not know it skip the line, so it cannot break your existing rules. Put it next to your Sitemap lines, outside any User-agent group. Why and how to add the Dataset line.

How to block it

To block DatasetsDBBot from your whole site, add this to robots.txt at the root of your domain:

User-agent: DatasetsDBBot
Disallow: /

To block only part of the site, list those paths instead:

User-agent: DatasetsDBBot
Disallow: /private/
Disallow: /drafts/

Changes apply to the next crawl that starts after you update the file.

Contact

If you think the crawler is misbehaving, or you want us to stop crawling your site entirely, write to [email protected]. Please include your domain and, if you can, a few log lines with timestamps. We read every message.