Jim Calabro
f7e61056f5
* implement sitemap handlers for users * ensure compressed payload is passed through * improved handling * reverse header order * add the sitemap to robots.txt
12 lines
415 B
Plaintext
12 lines
415 B
Plaintext
# Hello Friends!
|
|
# If you are considering bulk or automated crawling, you may want to look in
|
|
# to our protocol (API), including a firehose of updates. See: https://atproto.com/
|
|
|
|
# By default, may crawl anything on this domain. HTTP 429 ("backoff") status
|
|
# codes are used for rate-limiting. Up to a handful concurrent requests should
|
|
# be ok.
|
|
User-Agent: *
|
|
Allow: /
|
|
|
|
Sitemap: https://bsky.app/sitemap/users.xml.gz
|