Dataset card
Question-answer pairs, retrieval chunks and clean page text built from datasetsdb.com, the website of DatasetsDB itself.
What it covers: what DatasetsDB does and who it is for; how a website or an uploaded file becomes a dataset; the export formats (Q&A in JSON Lines, CSV and Markdown, Alpaca, ChatML, ShareGPT, RAG chunks, pages as Markdown, llms.txt, FAQPage JSON-LD) with their field names; the 11 checks of the AI-readiness audit; plans and limits; the DatasetsDBBot crawler and how to block it; the Dataset line in robots.txt; paid placement of a dataset on partner sites; and the cards of the public datasets in the catalog.
How to use it: load qa.jsonl to test a support assistant or a retrieval pipeline on product documentation, or use chunks.jsonl as a small, fully traceable RAG corpus. Every record carries the URL of the page it came from and a confidence score: 1.0 for FAQ markup, 0.9 for headings phrased as questions, 0.8 for pairs written by a language model whose supporting quote was found on the page.
Limits: the site is small, so the dataset is small. It is in English only. Pairs made from plain topic headings are left out of the public files. The dataset is rebuilt when the site changes, so numbers on this card can differ from an older download.
This dataset is the one referenced by the Dataset line in https://datasetsdb.com/robots.txt.
Summary
- Source website
- https://datasetsdb.com/
- Pages analysed
- 16
- Q&A pairs
- 78
- RAG chunks
- 29
- Words
- 5,946
- Q&A written by LLM, quote-verified
- 47
- AI-readiness score of the site
- 86 / 100
How it was built
Pages were fetched by DatasetsDBBot with robots.txt respected, cleaned of navigation and cookie notices, and split into Q&A pairs and retrieval chunks. Each record keeps its source URL and a confidence score: 1.0 for FAQ markup, 0.9 for question headings, 0.8 for model-written pairs whose supporting quote was found in the page. Low-confidence pairs made from plain topic headings are left out of the public files.
robots.txt
Site owners point crawlers to their dataset with a Dataset line in robots.txt (why):
Dataset: https://datasetsdb.com/datasets/semalt/datasetsdb-com
Verified. This line was found in robots.txt of the source site, so the dataset is published by the site owner.