# ====================================================== # 1. ALLOW: AI Search Engines & Live Retrieval # ====================================================== User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # ====================================================== # 2. BLOCK: Pure AI Model Training & Data Scrapers # ====================================================== User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / # ====================================================== # 3. GENERAL SETTINGS # ====================================================== User-agent: * Allow: / User-agent: Yandex Disallow: / Sitemap: https://finewinelibrary.nl/sitemap.xml