http://www.seekbot.net/bot.html Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2 TMCrawler / http://help.goo.ne.jp/door/crawler.html ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) / Yahoo! Slurp http://search.msn.com/msnbot.htm msnbots have changed their name recently msnbot/1.0 (+http://search.msn.com/msnbot.htm) Crawl-delay: 20 http://search.msn.com/msnbot.htm msnbots have changed their name recently msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) Crawl-delay: 20 http://www.google.com/ another name for the same google thing ;) Googlebot/2.1 http://www.google.com/ adsense and more Mediapartners-Google/2.1 http://www.bredband.net/ a swedish provider WebCrawler_1.1 internet@bredband.net / http://mywebsite.com my-heritrix-crawler(+http://mywebsite.com) / FAST Crawler (crawler at fast.no) / wordpress stats WordPress/2.1.2 Technoratibot/0.7 http://www.google.com/support/webmasters/bin/topic.py?topic=8843 the one and only ... webs most used search engine Googlebot http://www.google.com/ google image search Googlebot-Image http://www.google.com/ googles adwords campain Mediapartners-Google* http://www.seekport.de/help/webmaster_robot.html young european search engine seekbot http://help.yahoo.com/help/us/ysearch/slurp/index.html inktomi's "slurp" can read a minimum delay between hits Slurp Crawl-delay: 20 http://www.msn.com/ msn also accepts a delay MSNBot Crawl-delay: 20 http://irl.cs.tamu.edu/crawler/ IRL-crawler is a Texas A+M research project IRLbot / Crawl-delay: 20 UbiCrawler / DOC / Zao / Firefly/1.0 / WebReaper / sitecheck.internetseer.com / Zealbot / MSIECrawler / SiteSnagger / WebStripper / WebCopier / Fetch / Offline Explorer / Teleport / linko / HTTrack / Microsoft.URL.Control / Xenu / larbin / libwww / ZyBORG / Download Ninja / http://www.almaden.ibm.com/cs/crawler / thesubot / boitho.com-dc / HenryTheMiragoRobot / wget / k2spider / NPBot / A-Online Search / ATN_Worldwide / Arachnophilia / FAST / Scooter / Crawler V 0.2.1 admin@crawler.de / CoolBot / Deweb/1.01 / grabber / Eule-Robot / Arachnoidea / ArchitextSpider / KIT-Fireball/2.0 / IBM_Planetwide / InfoSeek Sidewinder/0.9 / LinkWalker / Lycos_Spider_(T-Rex) / WebMoose / Tarantula / Gulliver/1.3 / fido/1.0 / Harvest/1.4.pl2 / MegaSheep / Ultraseek / whatUseek_winona / Teoma / Gigabot / Scrubby / Robozilla / Nutch / baiduspider / yahoo-mmcrawler / yahoo-blogs/v3.9 / ActiveAgent / Alexibot / AskJeeves / BackDoorBot / BackDoorBot 1.0 / BackWeb / BecomeBot / Black Hole / BlackWidow / BlowFish / BlowFish 1.0 / BotRightHere / Bullseye / Cegbfeieh / CherryPicker /1.0 / CherryPicker 1.0 / CherryPickerElite 1.0 / CherryPickerSE 1.0 / ChinaClaw / Collector / Copernic / Copier / Crescent Internet ToolPak HTTPOLE Control v.1.0 / DISCo / DISCo Pump / DISCo Pump 3.1 / Download Demon / Download Wonder / Downloader / Drip / EirGrabber / EmailCollector 1.0 / EmailWolf 1.00 / Enterprise_Search / Enterprise_Search/1.0 / Express WebPictures / EyeNetIE / FileHound / FlashGet / FreeFind / GetRight / GetSmart / GetWeb! / Go!Zilla / Go-Ahead-Got-It / GrabNet / Grabber / Grafula / HLoader / HMView / Harvest / Harvest 1.5 / Hatena Antenna / Image Stripper / Image Sucker / Indy Library / InterGET / Internet Ninja / Iria / Iron33 / JOC / JOC Web Spider / Jeeves / JetCar / Jetbot / Jetbot/1.0 / JustView / Keyword Density / LinkScan / MIDown tool / Mag-Net / Magnet / Mass Downloader / Memo / Mirror / Mozilla / Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) / Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) / Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) / Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) / Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) / Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) / Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AIRF) / Navroad / NearSite / Net Vampire / NetSpider / NetZIP / Ninja / Octopus / Offline Navigator / OmniExplorer_Bot / Openfind data gatherer / PageGrabber / Papa Foto / ProPowerBot / Pump / Radiation / Radiation Retriever / ReGet / RealDownload / Reaper / Recorder / Roverbot / Siphon / SmartDownload / Snake / SpaceBison / Stanford / Stanford Comp Sci / Sucker / SuperBot / SuperHTTP / Surfbot / Szukacz / Teleport Pro / Teleport Pro/1.29.1590 / Teleport Pro/1.29.1616 / Teleport Pro/1.29.1632 / Teleport Pro/1.29.1718 / TeleportPro / TightTwatBot / Titan / Vacuum / VoidEYE / WWW-Collector / WWWOFFLE / WX_mail / Web Sucker / WebBandit 2.1 / WebBandit 3.50 / WebCapture 2.0 / WebCopier v.2.2 / WebCopier v3.2a / WebEMailExtrac. / WebEMailExtractor 1.0B / WebFetch / WebGo IS / WebLeacher / WebVac / WebWhacker / WebZIP/4.21 / WebZIP/5.0 / WebZip / WebmasterWorld / WebmasterWorld Extractor / Website / Website eXtractor / Webster / Whacker / WhoWhere / Widow / Xaldon / Xaldon/WebSpider / Xenu\'s / Xenu\'s Link Sleuth 1.1c / aconon Index / autoemailspider / b2w / b2w 0.1 / dloader(naverrobot)/1.0 / dumbot / eCatch / emailcollector / es / gotit / lftp / libWeb / likse / mozilla / mozilla/3 / mozilla/4 / mozilla/5 / naver / pavuk / pcBrowser / scooter / sootle / tAkeOut / webbandit 4.00.0 / CherryPicker / EmailCollector / EmailSiphon / WebBandit / EmailWolf / ExtractorPro / CopyRightCheck / Crescent / ProWebWalker / CheeseBot / LNSpiderguy / way back machine ia_archiver / ia_archiver/1.6 / TeleportPro / MIIxpc / Telesoft / Website Quester / moget/2.1 / WebZip/4.0 / WebSauger / NetAnts / Mister PiX / WebAuto / TheNomad / grub-client / grub / looksmart / WebZip / larbin / b2w/0.1 / psbot / Python-urllib / NetMechanic / URL_Spider_Pro / WWW-Collector-E / RMA / libWeb/clsHTTP / asterias / httplib / turingos / spanner / InfoNaviRobot / Harvest/1.5 / Bullseye/1.0 / Mozilla/4.0 (compatible; BullsEye; Windows 95) / CherryPickerSE/1.0 / CherryPickerElite/1.0 / WebBandit/3.50 / NICErsPRO / Microsoft URL Control - 5.01.4511 / DittoSpyder / Foobot / WebmasterWorldForumBot / SpankBot / BotALot / lwp-trivial/1.34 / lwp-trivial / BunnySlippers / Microsoft URL Control - 6.00.8169 / URLy Warning / Wget/1.6 / Wget/1.5.3 / Wget / cosmos / moget / hloader / humanlinks / LinkextractorPro / Mata Hari / LexiBot / Web Image Collector / The Intraformant / True_Robot/1.0 / True_Robot / BlowFish/1.0 / JennyBot / MIIxpc/4.2 / BuiltBotTough / ProPowerBot/2.14 / BackDoorBot/1.0 / toCrawl/UrlDispatcher / WebEnhancer / suzuran / VCI WebViewer VCI WebViewer Win32 / VCI / Szukacz/1.4 / QueryN Metasearch / Openfind data gathere / Openfind / Xenu's Link Sleuth 1.1c / Xenu's / Zeus / RepoMonkey Bait Tackle/v1.01 / RepoMonkey / Microsoft URL Control / Openbot / URL Control / Zeus Link Scout / Zeus 32297 Webster Pro V2.9 Win32 / Webster Pro / EroCrawler / LinkScan/8.1a Unix / Keyword Density/0.9 / Kenjin Spider / Iron33/1.0.2 / Bookmark search tool / GetRight/4.2 / FairAd Client / Gaisbot / Aqua_Products / Radiation Retriever 1.1 / Flaming AttackBot / Oracle Ultra Search / PerMan / searchpreview / http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 wildcard for the rest of the obeyers. you don't need to add the adjacent following bots ... i did it for the sake of knowledge ;) * /