Robots exclusion standard (Q3700)
Jump to navigation
Jump to search
standard used to advise web crawlers and scrapers not to index a web page or site
Language | Label | Description | Also known as |
---|---|---|---|
English | Robots exclusion standard |
standard used to advise web crawlers and scrapers not to index a web page or site |