robots.txt is the first file a crawler reads
A well-behaved crawler downloads robots.txt before any page. A pointer placed there is found without parsing HTML, and without a new file or URL on your server.
One line tells crawlers where the clean, machine-readable version of your site lives, and proves that you are the one who published it.
# rules for crawlers User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml Dataset: https://datasetsdb.com/datasets/example/example-com
We ask every site owner who publishes a dataset to add the line. Here is what it gives you.
A well-behaved crawler downloads robots.txt before any page. A pointer placed there is found without parsing HTML, and without a new file or URL on your server.
To learn what your site says, a crawler must fetch every page, drop menus and cookie banners, and guess which text answers which question. Your dataset already holds that result, with a source URL on every record.
Before publishing you see every record and can remove any of them. The dataset is the version of your content that you have checked, not whatever a scraper happened to extract.
Anyone can build a dataset from a public website. Only the owner can edit its robots.txt. When DatasetsDBBot finds your line, the dataset page shows a “Verified by site” badge.
The line contains no Allow or Disallow. A crawler that does not know the record skips it, as the Robots Exclusion Protocol expects.
Building a dataset from up to 300 pages, publishing it and verifying it through robots.txt are all part of the Free plan.
Dataset is a convention proposed by DatasetsDB, not part of the Robots Exclusion Protocol.The Sitemap line began the same way: a convention that search engines agreed to read from robots.txt in 2007. A convention only becomes useful when sites adopt it, which is why we ask for the line now. We cannot promise that other crawlers will follow.
Three ways to tell a machine about your site. They do different jobs and work together.
| Sitemap line | llms.txt | Dataset line | |
|---|---|---|---|
| What it points to | A list of page URLs | A short list of key pages | Cleaned content of the pages |
| Where it lives | robots.txt | A new file at /llms.txt | robots.txt |
| Content is already cleaned | No | No | Yes |
| Source URL on every record | No | No | Yes |
| Proves who published it | No | No | Yes |
| Part of a standard | Widely supported | Proposal | Proposal |
| Read by major crawlers | Yes | Rarely | Not yet |
Paste your website address on the sign-up page. A crawl of 300 pages usually takes a few minutes.
Open the project, go to the Publish tab, add a description and a license. The tab then shows your line.
Copy the line into robots.txt of the same site, next to the Sitemap lines, and upload the file.
Press “Check robots.txt now” on the Publish tab. When the line is found, the dataset page gets the badge.
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Dataset: https://datasetsdb.com/datasets/example/example-com
We use it ourselves: see the last line of datasetsdb.com/robots.txt, which points to the dataset of this site.
https address per line. Several Dataset lines are allowed.Dataset and dataset are read the same way.User-agent groups, like Sitemap.Create a dataset from your site, publish it and copy the line from the Publish tab. Free for up to 300 pages.
It is one line in the robots.txt file of a website: the word Dataset, a colon and the address of the public dataset built from that site. It works like the Sitemap line: Sitemap says where the list of pages is, Dataset says where the clean, machine-readable content of those pages is.
No. The Robots Exclusion Protocol, RFC 9309, defines User-agent, Allow and Disallow, and it lets crawlers support extra records such as Sitemap. Dataset is a convention proposed by DatasetsDB. We state this plainly so nobody adds it expecting something it does not do.
We have no evidence that they do. Today the line is read by DatasetsDBBot, which uses it to verify who published a dataset. The Sitemap line started the same way, as a convention that search engines adopted in 2007. We cannot promise the same outcome for Dataset.
It cannot change how your rules are read. A crawler that does not know a record skips it, and the line contains no Allow or Disallow. Place it outside any User-agent group, next to the Sitemap lines, so it cannot be mistaken for part of a group.
They do different jobs and you can use both. llms.txt is a separate file with a short list of important pages. The Dataset line needs no new file on your server: it sits in robots.txt, which every crawler already downloads first, and it points to full structured content with a source URL on every record. Neither is proven to change how assistants answer.
Sitemap points to URLs that a crawler still has to fetch, render and clean one by one. Dataset points to content that is already cleaned: question-answer pairs, retrieval chunks and page text without menus or cookie banners, each with its source URL and a confidence score.
Only someone who controls a website can edit its robots.txt. When DatasetsDBBot finds a Dataset line that points to a dataset page, that page shows a Verified by site badge. Without the line, a dataset built from your site is marked as unverified, so readers can tell an official dataset from one made by a third party.
Yes. Add one Dataset line per dataset, for example one for the main site and one for the documentation. Each line holds one absolute https address.
Nothing. Building a dataset from up to 300 pages, publishing it in the catalog and verifying it through robots.txt are all part of the Free plan.
If you remove the line, the badge disappears at the next check and nothing else changes. If you unpublish the dataset, its page stops being served, so remove the line as well to avoid pointing crawlers at a dead address.
Related: DatasetsDBBot · Export formats · Dataset catalog