# Digital Junkyard — crawler directives. # # The public storefront is open to every search engine and AI answer engine # (ChatGPT/GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot, Bing). # We *want* to be discoverable in AI search, so the wildcard group welcomes # them all. # # Admin, personalised, and secret-link pages are kept out of crawling here and # are additionally protected by X-Robots-Tag headers and per-page . # # The named groups below the wildcard are commercial SEO/backlink crawlers. # They crawl far harder than search engines, they index us for someone else's # product, and once the sitemaps listed every domain page (~27k URLs) they # became a measurable share of the Netlify bill. It is the same list the edge # Workers refuse with a 403 (infra/cloudflare/traffic-gates.js); this file lets # a polite crawler stop before it gets there. tests/unit/robots-txt.test.js pins # the two together — add a name in traffic-gates.js AND here. User-agent: * Allow: / Disallow: /en/admin.html Disallow: /zh/admin.html Disallow: /en/account.html Disallow: /zh/account.html Disallow: /en/order.html Disallow: /zh/order.html Disallow: /en/private-bundle Disallow: /zh/private-bundle Disallow: /en/private-bundle.html Disallow: /zh/private-bundle.html User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: PetalBot Disallow: / User-agent: Bytespider Disallow: / User-agent: SeekportBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: LinkdexBot Disallow: / User-agent: SISTRIX Disallow: / User-agent: Screaming Frog Disallow: / User-agent: SEOkicks Disallow: / User-agent: spbot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: Timpibot Disallow: / User-agent: ImagesiftBot Disallow: / Sitemap: https://www.digitaljunkyard.com/sitemap.xml