User-agent: AwarioBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: Bytespider Disallow: / User-agent: Claudebot Disallow: / User-agent: DotBot Disallow: / User-agent: GPTBot/1.2 Disallow: / User-agent: MJ12bot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Proton/1.0 Disallow: / User-agent: Scrapy/2.11.2 Disallow: / User-agent: Semrushbot Disallow: / User-agent: Trident/5.0 Disallow: / User-agent: YandexBot Disallow: / User-agent: bytedance Disallow: / User-agent: scrapy.org Disallow: / User-agent: * Crawl-delay: 5.00 Disallow: /*.exe$ Disallow: /*.gif$ Disallow: /*.Gif$ Disallow: /*.GIF$ Disallow: /*.gz$ Disallow: /*.jpg$ Disallow: /*.Jpg$ Disallow: /*.JPG$ Disallow: /*.jpeg$ Disallow: /*.Jpeg$ Disallow: /*.JPEG$ Disallow: /*.pdf$ Disallow: /*.Pdf$ Disallow: /*.PDF$ Disallow: /*.png$ Disallow: /*.zip$ Disallow: /*.Zip$ Disallow: /*.ZIP$