About the Doorbell crawler

When a business orders a deep scan from Doorbell, we fetch a small set of well-known discovery paths on their site, such as /llms.txt, /robots.txt, /sitemap.xml and files under /.well-known/, plus the homepage. One site, at most two requests per second, once per order.

The crawler's User-Agent names this page. It is based on the open-source agent-discovery-crawler. It respects robots.txt and never submits forms or logs in.

To opt a site out, email crawler-optout@agentsdoorbell.com.