# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content Sitemap: https://mostwelcome.pk/sitemap_index.xml Sitemap: https://mostwelcome.pk/sitemap_index.xml # Global base rules User-agent: * Allow: /wp-admin/admin-ajax.php Allow: /*/*.css Allow: /*/*.js Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /readme.html Disallow: /license.txt Disallow: /xmlrpc.php Disallow: /wp-login.php Disallow: /wp-register.php Disallow: */disclaimer/* Disallow: *?attachment_id= Disallow: /*?filtering= # Search Engine Visibility User-agent: Googlebot Allow: / User-agent: bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Sogou Allow: / User-agent: Ecosia Allow: / User-agent: MojeekBot Allow: / User-agent: Brave Allow: / User-agent: coccocbot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Googlebot-Video Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: AdsBot-Google Allow: / User-agent: Mediapartners-Google Allow: / User-agent: BingPreview Allow: / # Blocked Search Engine Overrides User-agent: Slurp Disallow: / User-agent: Yandex Disallow: / User-agent: YandexImages Disallow: / User-agent: Qwantify Disallow: / User-agent: Baiduspider Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: Baiduspider-video Disallow: / User-agent: Baiduspider/2.0 Disallow: / User-agent: Sogou spider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: YoudaoBot Disallow: / User-agent: youdao Disallow: / User-agent: yodao Disallow: / User-agent: SeznamBot Disallow: / User-agent: Yeti Disallow: / User-agent: Exabot Disallow: / # AI Bot Restrictions User-agent: OAI-SearchBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: PetalBot Disallow: / User-agent: CCBot Disallow: / User-agent: AI2Bot Disallow: / User-agent: Diffbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: TimpiBot Disallow: / User-agent: omgili Disallow: / # AI Bot Allow Overrides User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Google-CloudVertexBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: MistralAI-User Allow: / User-agent: Meta-ExternalAgent Allow: / # SEO Tool Restrictions User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: rogerbot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / # SEO Tool Allow Overrides User-agent: AhrefsBot Allow: / User-agent: SemrushBot Allow: / User-agent: SemrushBot-SA Allow: / User-agent: SemrushBot-BA Allow: / User-agent: SemrushBot-SI Allow: / User-agent: SemrushBot-SWA Allow: / User-agent: SemrushBot-CT Allow: / User-agent: SemrushBot-BM Allow: / # Bot & Scraper Protection (Full AI-Curated List) User-agent: python-requests Disallow: / User-agent: curl Disallow: / User-agent: wget Disallow: / User-agent: libwww-perl Disallow: / User-agent: Go-http-client Disallow: / User-agent: Apache-HttpClient Disallow: / User-agent: Scrapy Disallow: / User-agent: Java Disallow: / User-agent: GiftGhostBot Disallow: / User-agent: Seznam Disallow: / User-agent: PaperLiBot Disallow: / User-agent: Genieo Disallow: / User-agent: Dataprovider/6.101 Disallow: / User-agent: DataproviderSiteExplorer Disallow: / User-agent: Dataprovider/6.92 Disallow: / User-agent: Dataprovider Site Explorer Disallow: / User-agent: Dazoobot/1.0 Disallow: / User-agent: Diffbot Disallow: / User-agent: Diffbot/0.1 Disallow: / User-agent: DomainStatsBot/1.0 Disallow: / User-agent: dubaiindex Disallow: / User-agent: eCommerceBot Disallow: / User-agent: ExpertSearchSpider Disallow: / User-agent: Feedbin Disallow: / User-agent: Fetch/2.0a Disallow: / User-agent: FFbot/1.0 Disallow: / User-agent: focusbot/1.1 Disallow: / User-agent: HuaweiSymantecSpider Disallow: / User-agent: HuaweiSymantecSpider/1.0 Disallow: / User-agent: JobdiggerSpider Disallow: / User-agent: LemurWebCrawler Disallow: / User-agent: LipperheyLinkExplorer Disallow: / User-agent: LSSRocketCrawler/1.0 Disallow: / User-agent: LYT.SRv1.5 Disallow: / User-agent: MiaDev/0.0.1 Disallow: / User-agent: Najdi.si/3.1 Disallow: / User-agent: BountiiBot Disallow: / User-agent: Experibot_v1 Disallow: / User-agent: bixocrawler Disallow: / User-agent: bixocrawler TestCrawler Disallow: / User-agent: bixo Disallow: / User-agent: bixolabs/1.0 Disallow: / User-agent: Crawler4j Disallow: / User-agent: Crowsnest/0.5 Disallow: / User-agent: CukBot Disallow: / User-agent: DBLBot/1.0 Disallow: / User-agent: Digg Deeper/v1 Disallow: / User-agent: discobot/1.0 Disallow: / User-agent: discobot/1.1 Disallow: / User-agent: discobot/2.0 Disallow: / User-agent: discoverybot/2.0 Disallow: / User-agent: Dlvr.it/1.0 Disallow: / User-agent: drupact/0.7 Disallow: / User-agent: Ezooms/1.0 Disallow: / User-agent: fastbot crawler beta 2.0 Disallow: / User-agent: fastbot crawler beta 4.0 Disallow: / User-agent: feedly social Disallow: / User-agent: Feedly/1.0 Disallow: / User-agent: FeedlyBot/1.0 Disallow: / User-agent: Feedspot Disallow: / User-agent: Feedspotbot/1.0 Disallow: / User-agent: Clickagy Intelligence Bot v2 Disallow: / User-agent: classbot Disallow: / User-agent: CISPA Vulnerability Notification Disallow: / User-agent: CirrusExplorer/1.1 Disallow: / User-agent: Checksem/Nutch-1.10 Disallow: / User-agent: CatchBot/5.0 Disallow: / User-agent: CatchBot/3.0 Disallow: / User-agent: CatchBot/2.0 Disallow: / User-agent: CatchBot/1.0 Disallow: / User-agent: CamontSpider/1.0 Disallow: / User-agent: Buzzbot/1.0 Disallow: / User-agent: Buzzbot Disallow: / User-agent: BusinessSeek.biz_Spider Disallow: / User-agent: BUbiNG Disallow: / User-agent: 008/0.85 Disallow: / User-agent: 008/0.83 Disallow: / User-agent: 008/0.71 Disallow: / User-agent: ^Nail Disallow: / User-agent: FyberSpider/1.3 Disallow: / User-agent: findlinks/1.1.6-beta5 Disallow: / User-agent: g2reader-bot/1.0 Disallow: / User-agent: findlinks/1.1.6-beta6 Disallow: / User-agent: findlinks/2.0 Disallow: / User-agent: findlinks/2.0.1 Disallow: / User-agent: findlinks/2.0.2 Disallow: / User-agent: findlinks/2.0.4 Disallow: / User-agent: findlinks/2.0.5 Disallow: / User-agent: findlinks/2.0.9 Disallow: / User-agent: findlinks/2.1 Disallow: / User-agent: findlinks/2.1.5 Disallow: / User-agent: findlinks/2.1.3 Disallow: / User-agent: findlinks/2.2 Disallow: / User-agent: findlinks/2.5 Disallow: / User-agent: findlinks/2.6 Disallow: / User-agent: findlinks/1.0 Disallow: / User-agent: findlinks/1.1.3-beta8 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findlinks/1.1.4-beta7 Disallow: / User-agent: findlinks/1.1.6-beta1 Disallow: / User-agent: findlinks/1.1.6-beta1 Yacy Disallow: / User-agent: findlinks/1.1.6-beta2 Disallow: / User-agent: findlinks/1.1.6-beta3 Disallow: / User-agent: findlinks/1.1.6-beta4 Disallow: / User-agent: Crawlera/1.10.2 Disallow: / User-agent: SiteExplorer Disallow: / User-agent: spbot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: linkdexbot Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: Moreover Disallow: / User-agent: sentibot Disallow: / User-agent: Aboundexbot Disallow: / User-agent: proximic Disallow: / User-agent: oBot Disallow: / User-agent: meanpathbot Disallow: / User-agent: Nutch Disallow: / User-agent: TurnitinBot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: ZmEu Disallow: / User-agent: grapeshot Disallow: / # Block Archive Services User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Wayback Disallow: / User-agent: Wayback Machine Disallow: / User-agent: webrecorder Disallow: / User-agent: ArchiveTeam Disallow: / User-agent: ArchiveBot Disallow: / User-agent: SavePageNow Disallow: / User-agent: ArchiveBox Disallow: / # Block Author Archives User-agent: * Disallow: /author/ Disallow: /?author= Disallow: *?replytocom= Disallow: *?replytocom # E-commerce Basic Cleanup User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /wishlist/ Disallow: /compare/ Disallow: /account/ Disallow: /order-received/ Disallow: /order-pay/ Disallow: /view-order/ # E-commerce Advanced Cleanup User-agent: * Disallow: /*?add-to-cart=* Disallow: /*?add_to_cart=* Disallow: /*?added-to-cart=* Disallow: /*?orderby=* Disallow: /*?order=* Disallow: /*?filter_* Disallow: /*?filtering=* Disallow: /*?min_price=* Disallow: /*?max_price=* Disallow: /*?price_range=* Disallow: /*?pa_* Disallow: /*?product_cat=* Disallow: /*?product_tag=* Disallow: /product-category/*?* Disallow: /product-tag/*?* Disallow: /shop/*?* Disallow: /store/*?* Disallow: /*?wc-ajax=* Disallow: /*?remove_item=* Disallow: /*?removed_item=* Disallow: /*?undo_item=* # Allow CSS/JS User-agent: * Allow: /*.css$ User-agent: * Allow: /*.js$ # Allow Images User-agent: * Allow: /*.jpg$ User-agent: * Allow: /*.jpeg$ User-agent: * Allow: /*.png$ User-agent: * Allow: /*.gif$ User-agent: * Allow: /*.webp$ User-agent: * Allow: /*.bmp$ User-agent: * Allow: /*.ico$ User-agent: * Allow: /*.svg$ # Block Search & Pagination User-agent: * Disallow: /?s= Disallow: /*?s= Disallow: /search/ Disallow: /page/ Disallow: /*?p= Disallow: /*?paged= Disallow: /*?page= # Block common crawl trap parameters User-agent: * Disallow: /*?p=* Disallow: /*&p=* Disallow: /*?preview=* User-agent: * Allow: /ads.txt User-agent: * Allow: /app-ads.txt Crawl-delay: 5 # Custom Rules Disallow: /cgi-bin Disallow: /*.inc$ Disallow: /*.gz$ Disallow: /*.cgi$ # Generated by Better Robots.txt # https://better-robots.com # Crawl governance based on the Interpretive Governance framework # Reference: https://interpretive-governance.org/