robots.txt one line, free, no risk to your rules

Add a Dataset line to your robots.txt

One line tells crawlers where the clean, machine-readable version of your site lives, and proves that you are the one who published it.

  • Free on every plan
  • Cannot change your Allow / Disallow rules
  • Verifies site ownership
Why

Six reasons to add it today

We ask every site owner who publishes a dataset to add the line. Here is what it gives you.

robots.txt is the first file a crawler reads

A well-behaved crawler downloads robots.txt before any page. A pointer placed there is found without parsing HTML, and without a new file or URL on your server.

HTML is expensive and noisy for machines

To learn what your site says, a crawler must fetch every page, drop menus and cookie banners, and guess which text answers which question. Your dataset already holds that result, with a source URL on every record.

You decide which facts represent your site

Before publishing you see every record and can remove any of them. The dataset is the version of your content that you have checked, not whatever a scraper happened to extract.

It proves the dataset is yours

Anyone can build a dataset from a public website. Only the owner can edit its robots.txt. When DatasetsDBBot finds your line, the dataset page shows a “Verified by site” badge.

It cannot break anything

The line contains no Allow or Disallow. A crawler that does not know the record skips it, as the Robots Exclusion Protocol expects.

It is free

Building a dataset from up to 300 pages, publishing it and verifying it through robots.txt are all part of the Free plan.

Straight talk

What the line does not do

  • It is not a standard. Dataset is a convention proposed by DatasetsDB, not part of the Robots Exclusion Protocol.
  • Big crawlers do not read it yet. We have no evidence that Google, OpenAI, Anthropic or Perplexity use it. Today the line is read by DatasetsDBBot.
  • It is not a ranking factor. It does not raise positions and does not guarantee that an assistant will quote your site.

The Sitemap line began the same way: a convention that search engines agreed to read from robots.txt in 2007. A convention only becomes useful when sites adopt it, which is why we ask for the line now. We cannot promise that other crawlers will follow.

Compare

Sitemap, llms.txt and Dataset

Three ways to tell a machine about your site. They do different jobs and work together.

Sitemap linellms.txtDataset line
What it points toA list of page URLsA short list of key pagesCleaned content of the pages
Where it livesrobots.txtA new file at /llms.txtrobots.txt
Content is already cleanedNoNoYes
Source URL on every recordNoNoYes
Proves who published itNoNoYes
Part of a standardWidely supportedProposalProposal
Read by major crawlersYesRarelyNot yet
How

Add the line in four steps

  1. Create a dataset

    Paste your website address on the sign-up page. A crawl of 300 pages usually takes a few minutes.

  2. Publish it

    Open the project, go to the Publish tab, add a description and a license. The tab then shows your line.

  3. Paste the line

    Copy the line into robots.txt of the same site, next to the Sitemap lines, and upload the file.

  4. Get verified

    Press “Check robots.txt now” on the Publish tab. When the line is found, the dataset page gets the badge.

A complete file

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Dataset: https://datasetsdb.com/datasets/example/example-com

We use it ourselves: see the last line of datasetsdb.com/robots.txt, which points to the dataset of this site.

Rules for writing the line

  • One absolute https address per line. Several Dataset lines are allowed.
  • The name is not case sensitive: Dataset and dataset are read the same way.
  • Keep it outside User-agent groups, like Sitemap.
  • The address must be the dataset page built from this same site.

Get your line in a few minutes

Create a dataset from your site, publish it and copy the line from the Publish tab. Free for up to 300 pages.

FAQ

Questions about the Dataset line

What is the Dataset line in robots.txt?

It is one line in the robots.txt file of a website: the word Dataset, a colon and the address of the public dataset built from that site. It works like the Sitemap line: Sitemap says where the list of pages is, Dataset says where the clean, machine-readable content of those pages is.

Is Dataset an official robots.txt directive?

No. The Robots Exclusion Protocol, RFC 9309, defines User-agent, Allow and Disallow, and it lets crawlers support extra records such as Sitemap. Dataset is a convention proposed by DatasetsDB. We state this plainly so nobody adds it expecting something it does not do.

Do Google, OpenAI or Anthropic crawlers read the Dataset line today?

We have no evidence that they do. Today the line is read by DatasetsDBBot, which uses it to verify who published a dataset. The Sitemap line started the same way, as a convention that search engines adopted in 2007. We cannot promise the same outcome for Dataset.

Can the Dataset line break my robots.txt or hurt my rankings?

It cannot change how your rules are read. A crawler that does not know a record skips it, and the line contains no Allow or Disallow. Place it outside any User-agent group, next to the Sitemap lines, so it cannot be mistaken for part of a group.

Which is better for AI crawlers: llms.txt or the Dataset line?

They do different jobs and you can use both. llms.txt is a separate file with a short list of important pages. The Dataset line needs no new file on your server: it sits in robots.txt, which every crawler already downloads first, and it points to full structured content with a source URL on every record. Neither is proven to change how assistants answer.

How is the Dataset line different from the Sitemap line?

Sitemap points to URLs that a crawler still has to fetch, render and clean one by one. Dataset points to content that is already cleaned: question-answer pairs, retrieval chunks and page text without menus or cookie banners, each with its source URL and a confidence score.

How does the Verified by site badge work?

Only someone who controls a website can edit its robots.txt. When DatasetsDBBot finds a Dataset line that points to a dataset page, that page shows a Verified by site badge. Without the line, a dataset built from your site is marked as unverified, so readers can tell an official dataset from one made by a third party.

Can I list more than one dataset?

Yes. Add one Dataset line per dataset, for example one for the main site and one for the documentation. Each line holds one absolute https address.

How much does it cost to add the Dataset line?

Nothing. Building a dataset from up to 300 pages, publishing it in the catalog and verifying it through robots.txt are all part of the Free plan.

What happens if I remove the line or unpublish the dataset?

If you remove the line, the badge disappears at the next check and nothing else changes. If you unpublish the dataset, its page stops being served, so remove the line as well to avoid pointing crawlers at a dead address.

Related: DatasetsDBBot · Export formats · Dataset catalog