Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
[email protected] is safe to use (health: 43/100)
Get this data programmatically — free, no authentication.
curl https://depscope.dev/api/check/conda/r-robotstxtFirst published · 2024-01-21 10:23:44.484000+00:00
Last updated · 2025-09-21 11:42:53.816000+00:00