User-Agent: * Disallow: /messe/ Disallow: /excel_export/ Disallow: /jsiwyche3h19a6/ Disallow: /lagerdata/ Disallow: /EASTIN/ Disallow: /excel_export/ Disallow: /slwjgrndwef534/ Disallow: /api/ Disallow: /r6x.asp Disallow: /en/r6x.asp Disallow: /r18x.asp Disallow: /en/r18x.asp User-Agent: Mozilla/5.0+(Linux;+Android+5.0)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Mobile+Safari/537.36+(compatible;+Bytespider;+spider-feedback@bytedance.com) Disallow: / User-Agent: Amazonbot Disallow: / User-Agent: Mozilla/5.0+(Linux;+Android+7.0;)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Mobile+Safari/537.36+(compatible;+PetalBot;+https://webmaster.petalsearch.com/site/petalbot) Disallow: / User-Agent: Scrapy/2.11.2+(+https://scrapy.org) Disallow: / User-agent: YandexBot Disallow: / User-agent: Mozilla/5.0+(compatible;+Thinkbot/0.5.8;++In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_please_block_its_IP_address._Thank_you.) Disallow: / User-Agent: * Crawl-delay: 30 User-Agent: BLEXBot Crawl-delay: 30 User-agent: 008 Crawl-delay: 30