# vishwa.ai robots.txt # We want answer engines to crawl and cite us, so AI crawlers are explicitly allowed. # Bing powers Copilot and ChatGPT search, so bingbot is welcome too. User-agent: * Allow: / # --- AI / answer-engine crawlers (explicitly allowed) --- # OpenAI (training, search, and on-demand fetch) User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: anthropic-ai Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Google AI (Gemini / AI Overviews training opt-in) and Apple User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / # Common Crawl (feeds many LLM datasets) User-agent: CCBot Allow: / # LLM-friendly content index # llms.txt: https://www.vishwa.ai/llms.txt # llms-full.txt: https://www.vishwa.ai/llms-full.txt Sitemap: https://www.vishwa.ai/sitemap.xml