CrawlTrack, the spiders and crawlers tracking script.
protection
CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.


diable

CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.








Crawlers and spiders who can visit your website

Here is the list of crawlers used by CrawlTrack; that list is updated regularly. As you will see, it's already a very long list but there is still a lot of spiders crawling on the web not yet listed here.



Crawler User agent Owner
192.com 192.comAgent
4anything 4anything.com LinkChecker v2.0
80legs Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/200
8032620
A-Online A-Online Search
ABCdatos ABCdatos BotLink/5.xx.xxx#BBL
AOL Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)
ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/)
ASPseek ASPseek/1.2.xx
ASPSeek/1.2.x
ASPSeek/1.2.xxpre
ASPseek/1.2.9d
ASPSeek/1.2.5
ASPSeek/1.2.xa
AVSearch AVSearch-1.0(peter.turney@nrc.ca)
AbachoBot AbachoBOT (Mozilla compatible)
AbachoBOT
Abby Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
Aberja Checkoma Aberja Checkoma
Abot abot/0.1 (abot; http://www.abot.com; abot@abot.com)
About About/0.1libwww-perl/5.47
AboutUsBot Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
Accelobot Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com)
Accoona accoona
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Acoi AcoiRobot
Acoon Robot Acoon Robot v1.50.001
Acoon-Robot Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com)
Acorn Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o
rg)
Activtourist Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com)
AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html)
Aesop AESOP_com_SpiderMan
Agada Mozilla/4.0 (agadine3.0) www.agada.de
agadine/1.x.x (+http://www.agada.de)
Mozilla/4.0 (agadine3.0)
AgentName AgentName/0.1 libwww-perl/5.48
Aibot AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , C
hina)
Aicrawler Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Aipbot aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)
Alacra PortalBSpider/2.0 (spider@portalb.com)
Aladin.de Aladin/3.324
Aleksika Danmark Aleksika Spider/1.0 (+http://www.aleksika.com/)
Alexa ia_archiver
AlkalineBOT AlkalineBOT/1.3
AlkalineBOT/1.4 (1.4.0326.0 RTM)
Allesklar.de Allesklar/0.1 libwww-perl/5.46
Allrati Allrati/1.1 (+)
Almaden http://www.almaden.ibm.com/cs/crawler
http://www.almaden.ibm.com/cs/crawler [hc5]
Altavista Scooter2_Mercator_x-x.0
Scooter-W3.1.2
Scooter-ARS-1.1-ih
Scooter-3.0.HD
Scooter-3.0.FS
Scooter-3.0.VNS
Scooter-3.0QI
Scooter-3.2.BT
Scooter-3.2
Scooter-3.0.EU
Scooter/2.0 G.R.A.B. X2.0
Scooter/2.0 G.R.A.B V1.0
Scooter/1.0
Scooter/1.0 scooter@pa.dec.com
Scooter/1.1 (custom)
Scooter/2.0 G.R.A.B. V1.1.0
Scooter-3.2.DIL
Scooter-3.2.EX
Scooter-3.3dev
Scooter/3.3.vscooter
Scooter-ARS-1.1
Scooter_bh0-3.0.3
scooter-venus-3.0.vns
Scooter_trk3-3.0.3
Scooter/3.3_SF
Scooter/3.3.QA.pczukor
Scooter-3.2.NIV
Scooter-3.2.JT
Scooter-3.2.SF0
Scooter-3.2.snippet
Scooter/3.3
Scooter-W3-1.0
Amfibibot Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)
Amidalla libwww-perl/5.65
Annomille AnnoMille spider 0.1 alpha - http://www.annomille.it
AnsearchBot Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/)
AnswerBus AnswerBus (http://www.answerbus.com/)
Answerchase PROve AnswerBot 4.0
Antibot antibot-V1.3.3.1/debian-linux-sarge
Any Search Info Mozilla/4.0 (Sleek Spider/1.2)
Anzwers Australia AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine)
Apexoo Spider Apexoo Spider 1.0
ApocalXExplorerBot ApocalXExplorerBot
Aport Aport
Appie appie 1.1 (www.walhello.com)
appie 1.1 (www.walhello.com)
Arabulbot Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html)
ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby
.com;)
Arachnoidea Arachnoidea (arachnoidea@euroseek.com)
ArchitextSpider ArchitextSpider
Archive.org_bot Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht
tp://www.archive-it.org)
Arexera X-Crawler
TECOMAC-Crawler/0.x
Arianna www.arianna.it
Arikus_Spider Arikus_Spider
Asahina Asahina-Antenna/1.x
Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)
Ask 24x Info ask.24x.info
Ask Jeeves/Teoma Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr
awling.html)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_cra
wling.html)
Mozilla/2.0 (compatible; Ask Jeeves)
Asked asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info)
Asterias asterias/2.0
Asterias Crawler Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revis
ion: 3.11
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revisi
on: 3.10
Astrafind! Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)
Atlocal AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)
Attentio Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@att
entio.com)
Augurnet Swiss augurfind
augurnfind V-1.x
Axada axadine/ (Axadine Crawler; http://www.axada.de/; )
Axandra Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322)
Axmo AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine
.
Ay-Up FastBug http://www.ay-up.com
BE Internet Search Engine Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com)
Ba.be Mozilla/4.72 [en] (BACS http://www.ba.be)
BaBoom Web Portal BaboomBot/1.x.x (+http://www.baboom.us)
BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html)
Baiduspider BaiduImagespider+(+http://www.baidu.jp/search/s308.html)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.ht
ml)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
BaiDuSpider
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Balihoo Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots
meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a
spx; tgautier at balihoo dot com)
BanBot BanBots/1.2 (spider@banbots.com)
BeamMachine BeamMachine/0.5 (dead link remover of www.beammachine.net)
Beauty (Cosmoty) beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)
BebopBot BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html )
BecomeBot Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become
.com/site_owners.html)
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.
com/site_owners.html)
BecomeJPBot Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom
e.co.jp/site_owners.html)
BeijingCrawler BeijingCrawler
BigClique BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com)
Biglotron BIGLOTRON (Beta 2;GNU/Linux)
Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
BilBasen FAST Enterprise Crawler 6 used by BilBasen ApS (michael@bilinfo.dk)
BilgiBetaBot BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html
; nutch-agent@lucene.apache.org)
BilgiBot BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)
BinGet BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/)
Bingbot Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Bisnisseek Custom Spider www.bisnisseek.com /1.0 Bisnisseek
Bitacle Robot Bitacle bot/1.1
Bitacle Robot (V:1.0;) (http://www.bitacle.com)
Bitlybot bitlybot
Blaiz Enterprises Blaiz-Bee/1.0 (+http://www.blaiz.net)
Blaiz-Bee Blaiz-Bee/2.00.5502 (+http://www.blaiz.net)
Blaiz-Bee/2.00.5622 ( http://www.blaiz.net)
Blitzsuche Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T)
BlitzBOT@tricus.net
BlitzBOT@tricus.net (Mozilla compatible)
BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers
)
BlogScope Mozilla/5.0 (compatible; BlogScope/1.0; +http://www.blogscope.net/; U of Toronto
)
BlogSearch BlogSearch/1.x +http://www.icerocket.com/
BlogSearch/1.0 +http://www.icerocket.com/
BlogWatcher blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/)
Blogbot Naamah 1.0.1/Blogbot (http://blogbot.de/)
Naamah 1.0a/Blogbot (http://blogbot.de/)
Blogdex BlogBot/1.x
Blogdimension BlogBot Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com)
Bloglines Bloglines Title Fetch/1.0 (http://www.bloglines.com)
Bloglines/3.1 (http://www.bloglines.com)
Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com)
BlogzIce BlogzIce/1.0 +http://www.icerocket.com/
BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)
BnF bot Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/)
Boitho boitho.com-dc
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
boitho.com-robot/1.x (http://www.boitho.com/bot.html)
boitho.com-robot/1.x
Bot bot/1.0 (bot; http://; bot@bot.bot)
BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu)
Botmobi Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi
nd.mobi/bot.html abuse@mtld.mobi)
Brandwatch magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
BravoBrian bSTOP BravoBrian SpiderEngine MarcoPolo
BStop.BravoBrian.it Agent Detector
BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
Bruinbot BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)
Btbot BTbot/0.x (+http://www.btbot.com/btbot.html)
BuildCMS crawler BuildCMS crawler (http://www.buildcms.com/crawler)
BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html)
BullsEye/Intelliseek BullsEye
BurstFindCrawler BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawl
er@burstfind.com)
Buscaplus Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/)
Butterfly Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) G
ecko/2009032608 Firefox/3.0.8
CCBot CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
CEA larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)
CMP libwww-perl/5.41
CUPS PrivacyFinder Cache Bot v1.0
Camcrawler Camcrawler (+http://www.camdiscover.com/crawler.html)
CamontSpider CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
CanadianContent Search RoboCrawl (www.canadiancontent.net)
RoboCrawl (http://www.canadiancontent.net)
Carleson carleson/1.0
Catall Spider Catall Spider
Catall-Spider Catall-Spider/3.3.3(www.Catall.de)
CazoodleBot CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood
lebot; cazoodlebot@cazoodle.com)
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.co
m)
Ccubee ccubee/x.0
Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect
ion.com/bot.html )
Charlotte Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)
Chen Li Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; chenlibiti@163.com)
Christcentral ChristCRAWLER 2.0
Mozilla/4.0 (compatible; ChristCrawler.com, ChristCrawler@ChristCENTRAL.com)
CipinetBot CipinetBot/1.0 (http://www.cipinet.com/bot.html) Cipinet
CipinetBot (http://www.cipinet.com/bot.html)
Cityreview Robot Cityreview Robot (+http://www.cityreview.org/crawler/)
CjLogbot Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot)
Claymont Search Claymont.com
CloakDetect CloakDetect/0.9 (+http://fulltext.seznam.cz/)
Clushbot Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)
Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)
Clushbot/2.x (+http://www.clush.com/bot.html)
Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html)
Cnet robot Mozilla/4.6 [en] (http://www.cnet.com/)
CoBITSProbe CoBITSProbe
Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
oBot ((compatible;Win32))
Combine Combine/3 http://combine.it.lth.se/
Combine/2.0
Combine/2.0 http://combine.it.lth.se/
Cometrics-bot cometrics-bot, http://www.cometrics.de
Cometsystems Crawler (cometsearch@cometsystems.com)
Crawler (cometsearch@cometsystems.com)
Comperio FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no)
Compete.com larbin_2.2.0 (crawl@compete.com)
Computerorgs htdig/3.1.6 (http://computerorgs.com)
Comrite Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen
e.apache.org)
ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl)
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
Converas RetrievalWare Convera Internet Spider V6.x
ConveraCrawler/0.2
ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)
CrawlConvera0.1 (CrawlConvera@yahoo.com)
CoolBot CoolBot
Copyright sheriff copyright sheriff (+http://www.copyrightsheriff.com/)
Cortina Vision Research Lab image spider at vision.ece.ucsb.edu
CougarSearch CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml)
Cowbot Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
CrawlerBoy CrawlerBoy Pinpoint.com
Crawling jpeg Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp)
Crawllybot Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com)
Croccrawler CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
CsCrawler Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo
glespam/crawler.html RPT-HTTPClient/0.3-3
Csci_b659/0.13 csci_b659/0.13
Cuasar Cuasarbot/0.9b http://www.cuasar.com/spider_beta/
CurryGuide CurryGuide SiteScan 1.1
CyberAlerts Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)
Cydral CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com)
CydralSpider CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com)
CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com)
DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net
/aboutkr.html)
DAUM Web Robot Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi
cations Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu
nications Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm
unications Corp., Korea)
DNS-Digger Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
DailyOrbit Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
DataFountains DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents
.shtml)
DataFountains/DMOZ Downloader
DataSpear Spider Bot DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.co
m)
DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html
; dssb@dataspear.com)
DataparkSearch DataparkSearch/4.xx (http://www.dataparksearch.org/)
DataparkSearch/4.47 (+http://dataparksearch.org/bot)
DaviesBot DaviesBot/1.7 (www.wholeweb.net)
Daypop daypopbot/0.x
DbDig dbDig(http://www.prairielandconsulting.com)
De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com)
DeepIndexer DeepIndexer.ca
Deepak-USC/ISI deepak-USC/ISI
Deepindex Deepindex V2
DeepIndex (www.en.deepindex.com)
DeepIndex
Denmex Websearch Denmex websearch (http://search.denmex.com)
DepSpid Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net)
Dev-spider2 dev-spider2.searchpsider.com/1.3b
DiaGem Japan DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)
Die Kraehe -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de
Diggit Digger/1.0 JDK/1.3.0rc3
Direct Hit Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)
Disco-crawl disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@
discoveryengine.com)
disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@disc
overyengine.com)
Ditto DittoSpyder
DoCoMo DoCoMo/1.0/Nxxxi/c10
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.go
o.ne.jp/door/crawler.html)
DoCoMo/1.0/Nxxxi/c10/TB
DoCoMo/2.0 P900iV(c100;TB;W24H11)
Dodgebot dodgebot/experimental
DotBot DotBot/1.0.1
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne
tdotcom.org)
Doubanbot Doubanbot/1.0 (bot@douban.com http://www.douban.com)
Download-Tipp Download-Tipp Linkcheck (http://download-tipp.de/)
EyeCatcher (Download-tipp.de)/1.0
Drecombot Drecombot/1.0 (http://career.drecom.jp/bot.html)
DtSearchSpider dtSearchSpider
DuckDuckBot DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Dumbot Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)
Dumbot(version 0.1 beta - dumbfind.com)
Dumbot(version 0.1 beta)
E-SocietyRobot e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
E-StyleISP eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)
EApolloBot eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at
global-opto dot com)
EMPAS_ROBOT EMPAS_ROBOT
ESISmartSpider ESISmartSpider
Earthcom Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info)
EARTHCOM.info/1.xbeta [www.earthcom.info]
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
EARTHCOM.info/1.x [www.earthcom.info]
EasyDL EasyDL/3.xx http://keywen.com/Encyclopedia/Bot
EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
EasyDL/3.xx
Echo.com Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5
Echo.fr EchO!/2.0
Egothor Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/)
Egotobot EgotoBot/4.8 (+http://www.egoto.com/about.htm)
Elfbot elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)
Elsop LinkScan/11.0beta2 Unix
LinkScan/x.x Unix
LinkScan/9.0g Unix
EmeraldShield.com Web Spider EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx)
Enfish Tracker Enfish Tracker
Enoola enoola (http://www.enoola.com)
Enterprise Search Search/1.0 (http://www.innerprise.net/es-spider.asp)
Enterprise_Search/1.0.xxx
Enterprise_Search/1.0
ES.NET_Crawler/2.0 (http://search.innerprise.net/)
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
Entireweb Speedy_Spider (http://www.entireweb.com)
Speedy Spider (Beta/x.x; speedy@entireweb.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
WorldLight
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)
envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
EroCrawler EroCrawler
Eruvo-bot eruvo-bot 4.8.1 (http://www.eruvo.com)
EuripBot EuripBot/0.4 (+http://www.eurip.com) PreCheck
EuripBot/0.4 (+http://www.eurip.com) GetFile
EuripBot/0.5 (+http://www.eurip.com) PreCheck
EuripBot/0.2 (+http://www.eurip.com) GetRobots
Euro-spider Euro-Spider Shopping 1.0
Evaal Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com)
EvaalSE EvaalSE - bot@evaal.com
Eventax eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de)
Everest-Vulcan Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra
wlerhelp)
Evrinid-RT Mozilla/5.0 (compatible; Evrinid-RT; +http://www.evri.com/evrinid)
Exabot Exalead NG/MimeLive Client (convert/http/0.120)
Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)
NG/2.0
Exabot/2.0
ExaBotTest/3.0
Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko)
Exabot-Test/1.0
Exabot/3.0
ExaBotTest/2.0
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Exabot-Images Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
NG/4.0.1229
Exabot-Images/1.0
ExactSEEK eseek-larbin_2.6.2 (crawler@exactseek.com)
ExactSeek Crawler/0.1
exactseek-crawler-2.63 (crawler@exactseek.com)
exactseek.com
exactseek-pagereaper-2.63 (crawler@exactseek.com)
ExactSeek_Spider ExactSeek_Spider
Excalibur Excalibur Internet Spider V6.5.4
Execrawl Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com)
FAST-WebCrawler FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/
faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch
/faqfastwebcrawler.html)
FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastw
ebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support
/crawler.asp)
FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net)
FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net)
FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/fa
qfastwebsearch/faqfastwebcrawler.html)
fastlwspider/1.0
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST Enterprise Crawler/6.4.18 (crawler@fast.no)
FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FDSE Mozilla/4.0 (compatible; FDSE robot)
FaXobot Faxobot/1.0
Factbot Factbot 1.09 (see http://www.factbites.com/webmasters.php)
factbot : http://www.factbites.com/robots
FairShare Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairsh
are.cc)
Fast Search PycURL
Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de)
FauBot Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
Favo.eu crawler favo.eu crawler/0.6 (http://www.favo.eu)
Feed24 Feed24.com
FeedChecker FeedChecker/0.01
Feedfetcher-Google Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
Feedfinder feedfinder/1.371 Python-urllib/1.17 +http://www.aaronsw.com/2002/feedfinder/
Felix Felix - Mixcat Crawler (+http://mixcat.com)
Filangy Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi
langy-agent@filangy.com)
FindLinks findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
http://wortschatz.uni-leipzig.de/findlinks/
findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Findexa Crawler Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
FineBot FineBot
Firefly Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
Firefly/1.0
FirstGov FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov
Firstsbot firstsbot
Flapbot Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor perio
d com)
Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.
flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan
dindustries.com/flatlandbot.php; jason@flatlandindustries.com)
flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu
stries.com/flatlandbot.php; jason@flatlandindustries.com)
FlickBot FlickBot 2.0 RPT-HTTPClient/0.3-3
Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se
archhippo.com)
FnooleBot FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html)
Folkd.com Spider Folkd.com Spider/0.1 beta 1 (www.folkd.com)
ForAll.pl-Crawler ForAll.pl-Crawler/1.0
Francis Francis/1.0 (francis@neomo.de http://www.neomo.de/)
FreshNotes crawler FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com FreshNotes
FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com
Freshmeat freshmeat.net URL validator/1.1
FuchsBot FuchsBot +http://www.fuchsbot.tld
FurlBot Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn
.furlbot@looksmart.net)
FuseBulb FuseBulb.Com
FyberSpider FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
FyberSpider (+http://www.fybersearch.com/fyberspider.php)
GAIS Robot GAIS Robot/1.0B2
GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.co
m)
GPU p2p crawler Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin
e.php)
GSiteCrawler GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/)
Gaaz gazz/x.x (gazz@nttrd.com)
Gaisbot Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot Mozilla/4.0 (compatible; www.galaxy.com)
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Gamekitbot gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/)
GammaSpider GammaSpider/1.0
GenieKnows Mozilla/5.0 (wgao@genieknows.com)
larbin_2.6.3 (wgao@genieknows.com)
Mozilla/5.0 wgao@genieknows.com
geniebot wgao@genieknows.com
GeonaBot GeonaBot 1.x; http://www.geona.com/
Georgia Institute of Technology larbin_2.6.2 (listonATccDOTgatechDOTedu)
Geourl Mozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot)
GigaBaz Brainbot MicroBaz
gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/)
Gigabot Gigabot/3.0 (http://www.gigablast.com/spider.html)
Gigabot/2.0
Gigabot/2.0/gigablast.com/spider.html
Gigabot/2.0; http://www.gigablast.com/spider.html
Gigabot/2.0att
Girafabot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf
a dot com; http://www.girafa.com)
GlobalQueue Look.com
GnodSpider GNODSPIDER (www.gnod.net) Gnod
GoForIt GoForIt.com
GOFORITBOT ( http://www.goforit.com/about/ )
Goblin Goblin/0.9 (http://www.goguides.org/)
Goblin/0.9.x (http://www.goguides.org/goblin-info.html)
Gonzo1 gonzo1[P] +http://www.suchen.de/popups/faq.jsp
Gonzo2 gonzo1[P] mailto:crawleradmin.t-info@telekom.de
gonzo2[P] +http://www.suchen.de/faq.html
gonzo2[P] mailto:crawleradmin.t-info@telekom.de
Goo (Japan) Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Google Keyword Tool Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google.com/select
/KeywordToolExternal)
Google-Adsense Mediapartners-Google/2.1
Mediapartners-Google
Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
Google-Image Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
Googlebot-Image/1.0
Google-Sitemaps Google-Sitemaps/1.0
Google-WAP Google WAP Proxy/1.0
Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com
GoogleBot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible;
Googlebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog
lebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0
Googlebot/2.0 beta (googlebot@googlebot.com)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/Test ( http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.google.com/bot.html)
no-user-agent-in-crawltrack-for-that-bot
Googlebot-Mobile Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b
ot.html)
KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google
bot-Mobile/2.1; +http://www.google.com/bot.html)
Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)
GrigorBot GrigorBot 0.8 (http://www.grigor.biz/bot.html)
Gromit Gromit/1.0
Grub-client Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru
b.org)
Gsa-crawler gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com)
gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw)
Gulliver Gulliver/1.2
Gulliver/1.3
GulperBot Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim
/cgi-bin/Link/GulperBot)
Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index)
GurujiBot GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
Harvest-NG Harvest-NG/1.0.2
Hatena Antenna Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)
HatenaScreenshot HatenaScreenshot/1.0 (checker)
HatenaScreenshot/1.0 (checker)
Hbtronix.spider hbtronix.spider.2 -- http://hbtronix.de/spider.php
HeinrichderMiragoRobot HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp)
Helix Helix/1.x (+http://www.sitesearch.ca/helix/)
HenriLeRobotMirago HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp)
HenryTheMiragoRobot HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)
Heritrix Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs
.washington.edu/research/networking/websys/)
archive.org_bot
mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org)
Heritrix L3S Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec
ts/crawling/)
Heritrix/1.4.0 Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net)
Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com)
Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider
Hitwise Spider Hitwise Spider v1.0 http://www.hitwise.com
Holmes holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/x.x
holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/3.11 (http://morfeo.centrum.cz/bot)
holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
HolmesBot HolmesBot (http://holmes.ge)
HomePageSearch HomePageSearch(hpsearch.uni-trier.de)
Homerbot Homerbot: www.homerweb.com
Honda-Search Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda
-search.com)
Hoowwwer HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:
crawler-info<at>hiit.fi)
HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>
hiit.fi)
Htdig htdig/3.1.x (root@localhost)
Htdig/3.1.6 htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)
I1searchbot i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search
.com)
ICC-Crawler ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c
ontact(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con
tact(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a
t)ml(dot)nict(dot)go(dot)jp)
ICCrawler ICCrawler - ICjobs (http://www.icjobs.de/bot.htm)
ICRA_Label_spider ICRA_label_spider/x.0
IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)
IIITBOT IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr a
t iiit dot ac dot in)
INGRID Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://webmaster.
ilse.nl/jsp/webmaster.jsp)
IP2MapBot IP2MapBot/1.1 http://www.ip2map.com
IPiumBot IPiumBot laurion(dot)com
IRLbot IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)
IScraperBot IScraperBot/0.1
IWAgent IWAgent/ 1.0 - www.brandprotect.com
Iaskspider2 iaskspider2 (iask@staff.sina.com.cn)
Ichiro ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/1.0 (ichiro@nttr.co.jp)
ichiro/1.0 (ichiro@nttr.co.jp)
IconSurf IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)
IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)
Icsbot icsbot-0.1
Ideare ideare - SignSite/1.x
IlTrovatore IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator
e.it)
Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo
r more information please visit http://www.ilial.com/crawler; http://www.ilial.c
om/crawler; crawl@ilial.com)
ilial/Nutch-0.9-dev
Ilse Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmeld
en.ilse.nl/?aanmeld_mode=webhints)
Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.
ilse.nl/?aanmeld_mode=webhints)
ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com)
IncyWincy NetResearchServer/x.x(loopimprovements.com/robot.html)
IncyWincy(http://www.loopimprovements.com/robot.html)
IncyWincy/2.1(loopimprovements.com/robot.html)
IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovement
s.com/robot.html)
IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovemen
ts.com/robot.html)
IncyWincy (Look) IncyWincy(http://www.look.com)
IndexTheWeb IndexTheWeb.com Crawler7
Indonesia Interactive Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia
Interactive
Indy Library Mozilla/3.0 (compatible; Indy Library)
InelaBot InelaBot/0.2 (+http://inelegant.org/bot)
Inet Library Inet library
InfoFly InfoFly/1.0 (http://www.versions-project.org/)
InfoLab robot Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/)
InfoSec Search Bot RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht
ml; lhall@telegenetic.net)
Infoseek InfoSeek Sidewinder/0.9
Inria larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr)
Insitor Search robot Insitor.com search and find world wide!
Insitornaut Insitornaut
Internet Ninja Internet Ninja x.0
Internetseer InternetSeer.com
Iprospect Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56)
IpselonBot IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com)
Iseekbot iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot
com)
Ishida Lab larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp)
It-bot IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt
rovatore.it/bot.html; bot@iltrovatore.it)
Jabot Jabot/7.x.x (http://odin.ingrid.org/)
Jabot/6.x (http://odin.ingrid.org/)
Jambot Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot
; crawler@jambot.com)
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
Jayde Crawler Jayde Crawler. http://www.jayde.com
Jeanie jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51)
Jetbot Jetbot/1.0
Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de)
( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesunge
n/search/robot.html )
Jomjaibot Jomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com)
Jongaimpi jongaimpi/2.10 (jonga; http://www.jonga.co.za; info@jonga.co.za)
Jyxobot Jyxobot/x
Jyxobot/1
K2 Spider k2spider
KAIST AITrc Crawler KAIST AITrc Crawler
KFSW-Bot KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de)
KIT_Fireball KIT_Fireball/2.0
KS Crawler KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; crawler@kindsight.net)
KSbot KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte
nt/crawler/index.html; crawleradmin@knowledgestorm.com)
KakleBot KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bo
t.html; support@kakle.com)
KaloogaBot kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; info@kalooga.com)
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw
ler@kalooga.com)
Kasparek Firefox_1.0.6 (kasparek@naparek.cz)
Keegeebot Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html)
Kenjin Spider Kenjin Spider
Kevin Kevin http://dznet.com/kevin/
Kevin http://websitealert.net/kevin/
KicktooBot kicktooBotV1.1 kictooBot@kictoo.com
Kinja-imagebot kinja-imagebot (http://www.kinja.com/)
Kinjabot kinjabot (http://www.kinja.com)
KnowItAll KnowItAll(knowitall@cs.washington.edu)
Knowledge.com Knowledge.com/0.x
Krugle Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info
.html; webcrawler@krugle.com)
Kulokobot kuloko-bot/0.x
kulokobot www.kuloko.com kuloko@backweave.com
Kulturarw kulturarw3/0.1
Kumm KummHttp/1.1 (compatible; KummClient; Linux rulez)
Kyluka crawl Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k
yluka.com)
LECodeChecker LECodeChecker/3.0 libgetdoc/1.0
LNSpiderguy LNSpiderguy
LapozzBot LapozzBot/1.4 ( http://robot.lapozz.com)
LapozzBot/1.5 (+http://robot.lapozz.hu)
LapozzBot/1.4 (+http://robot.lapozz.hu)
Larbin_2.6.3 larbin_2.6.3 marzia.polito@intel.com
larbin_2.6.3 larbin2.6.3@unspecified.mail
Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo
.com; webmaster@lawinfo.com)
Lemur Consulting larbin_2.6.2 (tom@lemurconsulting.com)
Lexibot Mata Hari/2.00
LexiBot/1.00
Liafa larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr)
LibWeb libWeb/clsHTTP -- hiongun@kt.co.kr
LibertyW LibertyW (+http://www.lw01.com)
LibertyW (+http://www.libertyw.eu)
LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co
m)
Linguee Bot Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
LinkWalker LinkWalker
Linknzbot linknzbot
Links2Go Mozilla/3.01 (Compatible; Links2Go Similarity Engine)
Links4US-Crawler Links4US-Crawler, (+http://links4us.com/)
LinksManager.com_bot Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck
er.html)
Llaut Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)
Lmspider lmspider (lmspider@scansoft.com)
LocalBot LocalBot/1.0 ( http://www.localbot.co.uk/)
Lockstep Spider Lockstep Spider/1.0
Look.com NetResearchServer(http://www.look.com)
LookdirBot LookdirBot
Lovel Lovel as 1.0 ( +http://www.everatom.com)
Ltaa_web_crawler larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch)
Luchs.at URL checker luchs.at URL checker
Lycos_Spider Lycos_Spider_(modspider)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_
Spider_Beta2(T-Rex) )
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide
r_(T-Rex) )
Lycos_Spider_(T-Rex)
MJ12bot Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)
MQBOT MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.ui
uc.edu)
MSN Bot msnbot/0.9 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
msnbot/0.3 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
MSNBOT_Mobile MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot)
MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
MaSagool MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp)
Mail.Ru Mail.Ru/1.0
Mainseek_Bot Mozilla/5.0 (compatible;MAINSEEK_BOT)
Mammoth Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
mammoth/1.0 (+http://www.sli-systems.com/)
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
MantraAgent MantraAgent
Mariner Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)
Martini Martini
MARTINI
Marvin Marvin v0.3
Masterseek MasterSeek
Maxbot Spider/maxbot.com admin@maxbot.com
Maxomobot maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html;
maxomobot@maxomo.com)
MediaCrawler MediaCrawler-1.0 (Experimental)
MediaSearch MediaSearch/0.1
Mediater Rechercher libwww/5.3.2
MegaSheep MegaSheep v1.0 (www.searchuk.com internet sheep)
Megaglobe Crawler Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com)
Melbot WebSpider Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.)
Mercator Mercator-1.x
Mercator-2.0
Mercator-Scrub-1.1
Merl.com larbin_2.1.1 larbin2.1.1@somewhere.com
MetaGer_PreChecker MetaGer_PreChecker0.1
Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com)
Metadata Labs MLBot (www.metadatalabs.com/mlbot)
Metaeuro Web Crawler Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta
euro.com; crawler at metaeuro dot com)
Metager-Linkchecker MetaGer-LinkChecker
MetagerBot MetagerBot/0.8-dev (MetagerBot; http://metager.de; )
Metaquerier MQbot http://metaquerier.cs.uiuc.edu/crawler
MQbot metaquerier.cs.uiuc.edu/crawler
Metaspinner Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner
.de/)
Metatagsdir metatagsdir/0.7 (+http://metatagsdir.com/directory/)
Microsoft Small Business Indexer Microsoft Small Business Indexer
Microsoft URL Control Microsoft URL Control - 6.01.9782
Microsoft URL Control - 6.00.8862
Miggibot A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/)
miggibot/2006.01.24
Mirar Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s
ee getmirar.com)
Missigua Locator 1.9 Missigua Locator 1.9
Misterbot Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot
.fr)
Miva Miva (AlgoFeedback@miva.com)
MnoGoSearch MnogoSearch/3.2.xx
Mo College 1.9 Mo College 1.9
Moget moget/x.x (moget@goo.ne.jp)
Mogimogi mogimogi/1.0
Moi-même no-user-agent-in-crawltrack-for-that-bot Moi
MojeekBot Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)
MojeekBot/0.x (archi; http://www.mojeek.com/bot.html)
Monogol http://www.monogol.de
Moreoverbot Moreoverbot/5.00 (+http://www.moreover.com)
Morris Morris - Mixcat Crawler (+http://mixcat.com)
Mowserbot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.
com/bot)
MozDex mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com)
Mozdex Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com)
MultiText MultiText/0.1
Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html)
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
MuscatFerret Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)
Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)
Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)
MusicWalker MusicWalker2.0 (+http://www.somusical.com)
Mxbot Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
My-bytebot my-bytebot/1.0 (+http://spider.my-byte.de/info)
MyFamilyBot Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx
; SearchBot@MyFamilyInc.com)
Mylinea Mylinea.com Crawler 2.0
NASA Search 1.0 NASA Search 1.0
NCSA NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)
NG-SearchBot NG-Search/0.9.8 (http://www.ng-search.com)
NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; )
NII larbin_2.6.2 (hamasaki@grad.nii.ac.jp)
NLCrawler Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.
89 (like Gecko)
NMG Spider NMG Spider/0.3 (szukanko.com)
NTT Directory nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp)
NWSpider NWSpider 0.9
Nabot nabot_1.0
NABOT/5.0
Najdi Mozilla/5.0 (compatible; InterseekWeb/3.x)
Nameprotect NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com)
NPBot-1/2.0
NPBot (http://www.nameprotect.com/botinfo.html)
Nationaldirectory NationalDirectory-WebSpider/1.3
NationalDirectoryAddURL/1.0
NatzanBot NatzanBot, http://www.natzan.com
NaverBot dloader(NaverRobot)/1.0
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)
NaverBot_dloader/1.5
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.
jsp)
NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com)
NavissoBot NavissoBot
NavissoBot/1.7 (+http://navisso.com/)
Nebulla Nebullabot/2.2 (http://bot.nebulla.info)
Nebulla/V1.0 Spider (http://spider.nebulla.de)
Nebullabot/2.2 (http://bot.nebulla.de)
Nelian Pty Ltd - Spider Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com )
NetNose Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience:
http://www.netnose.com)
NetResearchServer NetResearchServer/4.0(loopimprovements.com/robot.html)
NetSprint -- 2.0 NetSprint -- 2.0
NetWhatCrawler NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com
; support@netwhat.com)
NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)
Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de)
Netprospector Netprospector JavaCrawler
Netscape Robozilla/1.0
NextGenSearchBot NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot
)
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB
ot)
NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe
nSearchBot.aspx)
NextopiaBot NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x
NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1
NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob
lems contact: crawler@healthline.com
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe
ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
NjuiceBot Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
Noago Spider Noago Spider
NokodoBot NokodoBot/1.x (+http://nokodo.com/bot.htm)
Norbert Norbert the Spider(Burf.com)
Noxtrumbot noxtrumbot/1.0 (crawler@noxtrum.com)
Noyona noyona_0_1
Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com)
NuSearch Spider nuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible
; MSIE 4.01)
Nutch NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
Nutch
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis
ts.sourceforge.net)
NutchCVS/0.8-dev NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy
crawl@cs.washington.edu)
NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o
rg/nutch/bot.html; ec2test at lucene.com)
OOZBOT OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Obidos-bot obidos-bot (just looking for books.)
Object Sciences Corp. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3
E
Objects Search ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup
port@thesoftwareobjects.com)
Ocelli Ocelli/1.3 (http://www.globalspec.com/Ocelli)
Ocelli/1.x (http://www.globalspec.com/Ocelli)
Octora Octora Beta - www.octora.com
Octora Beta Bot Octora Beta Bot - www.octora.com
OmniExplorer OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler
OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler
OnetSzukaj Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl)
Onet.pl SA, http://szukaj.onet.pl
OpenISearch OpenISearch/1.x (www.openisearch.com)
OpenPortal4U DigOut4U
OpenTaggerBot OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm)
OpenTextSiteCrawler OpenTextSiteCrawler/2.9.2
OpenWebSpider OpenWebSpider/x
Openfind Openfind Robot/1.1A2
Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.ht
ml)
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.
openfind.com.tw/robot.html)
OpidooBot OpidooBOT (larbin2.6.3@unspecified.mail)
Oracle Oracle Ultra Search
OrangeBot-Mobile Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19
.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)
OrangeSpider OrangeSpider
Orbiter Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
Overture-WebCrawler Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor
t/crawler.asp)
OzMonitor Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free)
Ozelot ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat
.de)
PJspider PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)
PWeBot PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php)
Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)
Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)
PageBitesHyperBot PageBitesHyperBot/600 (http://www.pagebites.com/)
Page_verifier page_verifier http://www.securecomputing.com/goto/pv
Pagebull Pagebull http://www.pagebull.com/
Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr)
FAST Enterprise Crawler 6 used by Pages Jaunes (fastadmin@pagesjaunes.fr)
FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com)
PalmeraBot Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi
on 0.001
Pandora Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/
crawl.html)
ParaSite ParaSite/1.0b (http://www.ianett.com/parasite/)
PatchOneSpider Mozilla/5.0 (compatible; PatchOneSpider/1.6; http://www.patchone.se)
Patwebbot Patwebbot (http://www.herz-power.de/technik.html)
Peerbot PEERbot www.peerbot.com
Petitsage www.petitsage.fr site detector 0.4
PicoSearch PicoSearch/1.0
PictureOfInternet PictureOfInternet/3.0 (http://www.malfunction.org/poi; erik@malfunction.org)
Piffany Piffany_Web_Spider_v0.x
Piffany_Web_Scraper_v0.x
Pilot Hitlist Marketwave Hit List
Pingdom Pingdom GIGRIB v1.1 (http://www.pingdom.com)
PipeLiner pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)
Pita Pita
PlantyNet_WebRobot PlantyNet_WebRobot_V1.9 dhkang@plantynet.com
PlantyNet_WebRobot_V1.9 yangsam@plantynet.com
PluckFeedCrawler PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1
subscribers)
Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin
fo.htm)
Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net)
Pogodak Mozilla/5.0 (compatible; pogodak.ba/3.x)
Polybot polybot 1.0 (http://cis.poly.edu/polybot/)
Pompos Pompos/1.3 http://dir.com/pompos.html
Pompos/1.x pompos@iliad.fr
Pompos
Pompos/1.2 http://dir.com/pompos.html
Pompos/1.1 http://dir.com/pompos.html
Pompos/1.x http://dir.com/pompos.html
PopJapanSearch Robot/www.pj-search.com
Popdexter Popdexter/1.0
PrivacyFinder/1.1 PrivacyFinder/1.1
ProWebguide ProWebGuide Link Checker (http://www.prowebguide.com)
Probe! PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi
a.com)
Psbot psbot/0.1 (+http://www.picsearch.com/bot.html)
Psycheclone psycheclone
Pubblisito info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca
Pumpkin blogsearchbot-pumpkin-3
Purebot Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)
Python-urllib Python-urllib/1.16
QPCreep QPCreep Test Rig ( We are not indexing, just testing )
QihooBot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.n
et)
Quantcastbot Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com)
Qube qube/2.0 (+http://qube.qelix.com/v2)
QuepasaCreep QuepasaCreep v0.9.1x
QuepasaCreep ( crawler@quepasacorp.com )
QueryN Metasearch QueryN Metasearch
Qweerybot Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com)
QweeryBot/3.01 ( http://qweerybot.qweery.nl)
QweeryBot/3.02 ( http://qweerybot.qweery.nl)
R6_CommentReader R6_CommentReader_(www.radian6.com/crawler) Radian6
R6_FeedFetcher R6_FeedFetcher_(www.radian6.com/crawler) Radian6
R6_FeedFetcher(www.radian6.com/crawler)
RAMPyBot RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu
cene.apache.org)
RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveram
p.com/bot.html; support@giveRAMP.com)
RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com)
Rambot ramBot xtreme x.x
RankMeter Web Snooper
Rational SiteCheck Rational SiteCheck (Windows NT)
Reaper Reaper/2.0x (+http://www.sitesearch.ca/reaper)
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)
RedBot RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)
RedCarpet/1.2 RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html)
RedCell RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net)
RedKernel RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)
RoboPal RoboPal (http://www.findpal.com/)
Rotondo Rotondo/3.1 libwww/5.3.1
RufusBot RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)
RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html)
S2Bot S2Bot/1.0 (http://search2.net)
SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com)
SBIder SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell
.com/contact-support.html)
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site
sell.com/contact-support.html)
SandCrawler SandCrawler - Compatibility Testing
Savvybot savvybot/0.2
ScanWeb ScanWeb
Schibstedsokbot schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/
/www.schibstedsok.no/bot/)
ScholarUniverse ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho
laruniverse.com)
Scirus-crawler FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com
/srsapp/contactus/
ScollSpider ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php)
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)
ScorpionBot Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots
ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/
ScoutJet ScoutJet
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Scrubby Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Scrubby/2.x (http://www.scrubtheweb.com/)
Sdcresearchlabs-testbot sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache
.org/nutch/bot.html; researchbot@shopping.com)
Search-Engine-Studio Search-Engine-Studio
Search-Info Spider-Sleek/2.0 (+http://search-info.com/linktous.html)
Search.ch search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)
search.ch V1.4
Search17Bot Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
Search4free lwp-trivial/1.34
SearchByUsa SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs
a.com)
SearchEngineWorlds ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
SearchExpress Spider SearchExpress Spider0.99
SearchMonkey Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; http://developer.yahoo.com/sea
rchmonkey/useragent)
SearchScout DiamondBot
ClariaBot/1.0
Diamond/1.0
SearchSight SearchSight/2.0 (http://SearchSight.com/)
SearchSpider Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspi
der.com)
SearchSpider.com/1.1
SearchdayBot SearchdayBot
Searchdnabot searchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at sear
chenginedna dot com)
Searchguild SearchGuild_DMOZ_Experiment (chris@searchguild.com)
Searchit-Now Robot Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)
Searchmee! Spider Searchmee! Spider v0.98a
Seekbot Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
Seeker.lookseek Seeker.lookseek.com
Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/)
Semager/1.0 (http://www.semager.de)
Sensis.com.au Web Crawler Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau)
SeznamBot SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
SeznamBot/1.0 (+http://fulltext.seznam.cz/)
SeznamBot/1.0
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SharewarePlaza Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com)
SheenBot SheenBot/SheenBot-1.0.4 (Sheen web crawler)
Sherlock sherlock/1.0
Shim-Crawler Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra
wl@logos.ic.i.u-tokyo.ac.jp)
Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl
er/; crawler@logos.ic.i.u-tokyo.ac.jp)
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c
rawl@logos.ic.i.u-tokyo.ac.jp)
ShopWiki ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
ShopWiki/1.0 ( +http://www.shopwiki.com/)
Shoula Shoula.com Crawler 2.0
SietsCrawler SietsCrawler/1.1 (+http://www.siets.biz)
Siigle Orumcex Siigle Orumcex v.001 Turkey (http://www.siigle.com)
Silk/1.0 silk/1.0 (+http://www.slider.com/silk.htm)/3.7
silk/1.0
Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html)
Site Server 3.0 Robot Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR
SiteBar SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
SiteBaseBot Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru)
SiteSpider SiteSpider +(http://www.SiteSpider.com/)
SiteTruth SiteTruth.com site rating system
SitiDiBot +SitiDi.net/SitiDiBot/1.0 (+Have Good Day)
Skampy Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en]
Skizzle User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE
.com)
SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
Slider Slider_Search_v1-de
Slurp (Yahoo) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)
Slurp China (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Slurp Inktomi Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp Inktomi (Yahoo) Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071
3 Firefox/2.0.0.5
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch
/slurp)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE
cho/2.0.0.4
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu
rp)
Slurpy Verifier Slurpy Verifier/1.0
SmiffyDCMetaSpider SmiffyDCMetaSpider/1.0
SnapPreview Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
Snapbot Snapbot/1.0
Snoopy Snoopy v1.2
SnykeBot SnykeBot/0.6 (http://www.snyke.com)
SoftHypermarket SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com)
Sogou web spider Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sohu-search sohu-search
Somewhere Mozilla (Mozilla@somewhere.com)
Sosospider Sosospider+(+http://help.soso.com/webspider.htm)
Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
Spbot Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
Speedfind speedfind ramBot xtreme 8.1
Speedy Spider Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Beta/1.0; www.entireweb.com)
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Sphere Scout Sphere Scout&v4.0 - scout at sphere dot com
Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg
)
SpiderMonkey Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml)
SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
Spinn3r Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.
1); http://spinn3r.com/robot) Gecko/20021130
SplatSearch libwww-perl/5.45
Spock Crawler Spock Crawler (http://www.spock.com/crawler)
Sproose sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr
oose.com)
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.c
om)
StackRambler StackRambler/x.x
StackRambler/2.0 (MSIE incompatible)
StackRambler/2.0
Steeler Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Strategic Board Bot Strategic Board Bot (+http://www.strategicboard.com)
Suchbaer suchbaer.de
suchbaer.de (CrawlerAgent v0.103)
Suchpad suchpad/1.0 (+http://www.suchpad.de)
SummizeBot Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com)
SuperSnooper Robot@SuperSnooper.Com
SurveyBot SurveyBot/2.3 (Whois Source)
Susie !Susie (http://www.sync2it.com/susie)
Swooglebot Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html)
SycaBoT SycaBoT/1.0
SycaBoT-Audio SycaBoT-Audio
SycaBoT-Image SycaBoT-Image
SycaBoT-Programme SycaBoT-Programme
SycaBoT-Video SycaBoT-Video
Sycrawl NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr
awl@cs.washington.edu)
Sygol <http://www.sygol.com/> http://www.sygol.com
SygolBot SygolBot http://www.sygol.com
SygolBot http://www.sygol.net
SynooBot SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.
html)
Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html)
Synthesio Crawler Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Syntryx Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler
Szukacz Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl)
Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl)
TAGword Tagword (http://tagword.com/dmoz_survey.php)
TCDBOT TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot
IE)"
Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/)
TailRank Robot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta
ilrank.com/robot) Gecko/20021130
Talkro Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com, Home: http://222.122.15.190/w
ebshot)
TargetSeek Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/
TargetSeek.html)
Tasapspider Tasapspider/0.9 (http://www.tasap.com)
Technoratibot Technoratibot/0.7
Tecomi Bot Tecomi Bot (http://www.tecomi.com/bot.htm)
Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w
ww.netseer.com/crawler.html; crawler@netseer.com)
TelenetDigger TelenetDigger/1.0; htdig/3.1.6 (webdev@staff.telenet.be)
Teoma teomaagent crawler-admin@teoma.com
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
teoma_agent1
Teoma MP
teomaagent1 [crawler-admin@teoma.com]
Teradex Mapper Teradex Mapper; mapper@teradex.com; http://www.teradex.com
TeragramCrawler TeragramCrawler
TeragramWebcrawler TeragramWebcrawler/1.0
TerrawizBot TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)
TheSuBot TheSuBot/0.1 (www.thesubot.de)
Theme Spider Theme Spider (+http://www.themespider.com/spider.html)
Theophrastus Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th
eophrastus.php)
Thumbshots-de-bot thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)
Thunderstones Webinator Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)
TimboBot timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html
TinEye TinEye/1.1 (http://tineye.com/crawler.html)
TivraSpider tivraSpider/1.0 (crawler@tivra.com)
Tkensaku Tkensaku/x.x(http://www.tkensaku.com/q.html)
ToileBot Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1
Topodia Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia
.com/; support@topodia.com)
Toutatis Toutatis x.x (hoppa.com)
Toutatis x.x-x
Toutatis x-xx.x (hoppa.com)
Traazibot traazibot/testengine (+http://www.traazi.de)
Trampelpfad Trampelpfad-Spider
Trampelpfad-Spider-v0.1
TrendTech http://www.trendtech.dk/spider.asp)
Truveo Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/)
Turnitin TurnitinBot
TurnitinBot TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)
Turnpike Emporium Turnpike Emporium LinkChecker/0.1
TutorGigBot Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)
TutorGig/1.5 (+http://www.tutorgig.com/crawler)
TweetmemeBot Mozilla/5.0 (compatible; TweetmemeBot/1.0; +http://www.tweetmeme.com/)
Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0
.6 TweetmemeBot
TwengaBot TwengaBot/1.1 (+http://www.twenga.com/bot.html)
TwengaBot/2.0 (http://www.twenga.com/bot.html)
Twiceler Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Twiceler www.cuil.com/twiceler/robot.html
Twiceler-0.9 http://www.cuil.com/twiceler/robot.html
Twiceler www.cuill.com/twiceler/robot.html
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
TygoBot TygoBot
TygoProwler
T�zilla Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez
illa.de/t_st-odp-entries-agent.html)
UKWizz UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
Mackster( http://www.ukwizz.com )
UN OCHA FAST Enterprise Crawler 6 used by suckling@un.org (UN OCHA)
UOLCrawler UOLCrawler (soscrawler@uol.com.br)
URL-Spider Mozilla/5.0 URL-Spider
URL_Spider_Pro URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp)
URL_Spider_Pro/x.x
URL Spider Pro/x.xx (innerprise.net)
USyd-NLP-Spider USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)
Ultraseek Ultraseek
UnChaosBot <a href="http://www.unchaos.com/"> UnChaos Bot, Hybrid Web Search Engine. </a> (
vadim_gonchar@unchaos.com)
<a href="http://www.unchaos.com/"> UnChaos </a>, From Chaos To Order, Hybrid Web
Search Engine. (vadim_gonchar@unchaos.com)
<b> UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at www.unc
haos.com </b> (info@unchaos.com)
Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt
Unido-bot unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html
Updated updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)
Updated crawler updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
Uptimebot UptimeBot(www.uptimebot.com)
Uptimebot
Urlchecker1.0 Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0)
Urlfan-bot urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
VBSEO Mozilla/4.0 (vBSEO; http://www.vbseo.com)
VIProbot VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; info@vipsolutions.hu)
VMBot VMBot/0.9 (VMBot; http://www.verticalmatch.com; vmbot@tradedot.com)
VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com)
VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiu
c.edu
Vacobot Vacobot; (+http://vaco.ws/bot.html)
Vagabondo Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http
://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w
ebagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt
p://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo/3.0 (webagent at wise-guys dot nl)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://
webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo-WAP Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/
)/1.0 Profile
Vakes Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com)
Verizon fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt
ers@fastsearch.com)
FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad
min@superpages.com)
Vermut Mozilla/5.0 (compatible; vermut +http://vermut.aol.com)
Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com)
mozilla/5.0 (compatible; vermut +http://www.aol.com)
Versus versus 0.2 (+http://versus.integis.ch)
Versus Crawler versus crawler eda.baykan@epfl.ch
VeryGoodSearch VeryGoodSearch.com.DaddyLongLegs
Verzamelgids Verzamelgids/2.2 (http://www.verzamelgids.nl)
verzamelgids.nl - Networking4all Bot/x.x
Vespa Crawler (Yahoo) Vespa Crawler
Visbot VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)
Visbot/1.0 (+http://www.visvo.com/bot.html;bot@visvo.com)
Vlad Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html)
VoilaBot Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor
t.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/
/www.voila.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi
la.com/)
KE_1.0/2.0 libwww/5.2.8
Vonna.com b o t Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t)
Vortex/2.2 Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
Voyager voyager-hc/1.0
voyager/1.0
cfetch/1.0
Vspider vspider/3.x
vspider
W3C-Validator W3C_Validator/1.432.2.10
W3C-checklink W3C-checklink/4.2 [4.20] libwww-perl/5.803
W3SiteSearch Crawler W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de
W8net Mozilla/5.0 usww.com-Spider-for-w8.net
WEP Search WEP Search 00
WIRE WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler)
WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it)
WISEbot WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com)
WWW.fi crawler www.fi crawler, contact crawler@www.fi
WWWeasel Robot WWWeasel Robot v1.00 (http://wwweasel.de)
Wadaino.jp-crawler wadaino.jp-crawler 0.2 (http://wadaino.jp/)
WasaBot Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/)
WasaBot/2.4 (+ http://www.wasalive.com ) Java/1.6.0
WasaBot 0.11 (+http://www.wasalive.com/)
Wavefire Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com)
Waypath Waypath development crawler - info at waypath dot com
Waypath Scout v2.x - info at waypath dot com
WeRelateBot WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer
elate.org)
WeatherBot WeatherBot v1.4 http://www.ezweather.net
WebAlta Crawler WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT
5.1; ru-RU)
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)
WebBOT Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot
/index.html
WebCorp WebCorp/1.0
WebCorp/1.1
WebFindBot WebFindBot(http://www.web-find.com)
WebGobbler webGobbler/1.x.x
WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
WebSearch WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; S
earch@WebSearch.COM.AU)
WebSearchBench WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit�t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WebSearchBench WebCrawler v0.1(Experimental)
WSB, http://websearchbench.cs.uni-dortmund.de
WSB WebCrawler V1.0 (Beta), cl@cs.uni-dortmund.de
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit?t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WebStat WebStat/1.0 (Unix; beta; 20040314)
WebVac WebVac (webmaster@pita.stanford.edu)
WebarooBot WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html)
WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
Webbandit webbandit/4.xx.0
Webbot webbot(+http://webbot.com/bot.htm)
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)
Webclipping Webclipping.com
Webcrawl webcrawl.net
webcrawl.net
Webduniabot Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx)
Webglimpse Webglimpse 2.xx.x (http://webglimpse.net)
Weblog Attitude Diffusion Weblog Attitude Diffusion 1.0
Webmeasurement-bot webmeasurement-bot, http://rvs.informatik.uni-leipzig.de
Webmeasurement-crawler webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/
WebsiteWorth WebsiteWorth v1.0
Webspinne Webspinne/1.0 webmaster@webspinne.de
Websquash Websquash.com (Add url robot)
Webster Webster v0.3 ( http://webster.healeys.net/ )
Webverzeichnis Webverzeichnis.de - Telefon: 01908 / 26005
Whoiam Mozilla/5.0 whoiam [http://www.axxus.de/]
Wikiwix-bot wikiwix-bot-3.0
Willow Willow Internet Crawler by Twotrees V2.1
WinME Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] ?
WinkBot WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB
ot; winkbot@wink.com)
WiseGuys Vagabondo/2.0 MT
Vagabondo/1.x MT (webagent@wise-guys.nl)
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wi
se-guys.nl/)
Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.
nl/)
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)
Worio Mozilla/5.0 (compatible; woriobot support [at] worio [dot] com +http://worio.com
))
Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/)
Worio bot Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)
Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com)
Worio heritrix bot worio heritrix bot (+http://worio.com/)
WorldWideWeb-X/3.1 WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/)
Wotbox Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02
Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)
Wume_crawler wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)
Wwlib/Linux Wwlib/Linux
Wwwster wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de)
XP5 Project XP5 [2.03.07-111203]
Xdefine egothor/3.0a (+http://www.xdefine.org/robot.html)
Xirq xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com)
Xyleme SA France cosmos/0.9_(robot@xyleme.com)
cosmos/0.8_(robot@xyleme.com)
Y!J Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yacy yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/
en)
Yacybot yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/
bot.html
yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net
yacybot (i386 Linux 2.6.31-14-generic-pae; java 1.6.0_15; Europe/fr) http://yacy
.net/bot.html
yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net
Yahoo Japan robot Y!J-SRD/1.0
DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)
Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yahoo Search Marketing crawler Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com)
YahooYSMcm/2.0.0
Yahoo! Mindset Yahoo! Mindset
Yahoo-Blogs Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/
us/ysearch/crawling/crawling-02.html )
Yahoo-MMAudVid Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co
m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo-MMCrawler Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Yahoo-Test/4.0 Yahoo-Test/4.0
Yahoo-VerticalCrawler Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com;
http://www.alltheweb.com/
YahooFeedSeeker YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu
blishers.html)
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c
om/rssguide)
YahooSeeker YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web
master@yahoo-inc.com ) (KHTML, like Gecko)
YahooSeeker/M1A1-R2D2 SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in
c dot com)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile
-search-customer-care AT yahoo-inc dot com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i
nc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http:
//help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo
-inc dot com)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo
.com/help/us/ysearch/crawling/crawling-01.html)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-
01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo.
com/help/us/ysearch/crawling/crawling-01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/
crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust
omer-care AT yahoo-inc dot com)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot
com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch
/crawling/crawling-01.html)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli
ng/crawling-01.html)
Yandex Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)
Yandex/1.01.001 (compatible; Win16; I)
YandexBot Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Yarienavoir yarienavoir.net/0.2
Yellopet spider.yellopet.com - www.yellopet.com
YesupBot Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
Yeti Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti
Yggdrasil yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m
inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres
den dot de)
YodaoBot Nokia3650/1.0 SymbianOS/6.1 Series60/1.2 Profile/MIDP-1.0 Configuration/CLDC-1.0
/ (compatible; YodaoBot-Mobile/1.0; http://www.youdao.com/help/webmaster/spider/
; )
YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide
r/; )
YooW! YooW!/1.8.9 RC1 (+http://www.yoow.eu)
Yoogli yoogliFetchAgent/0.1
Yoono Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/)
Yoono web-crawler yoono/1.0 web-crawler/1.0
YottaCars YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine
YottaShopping YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine
Z-Add Link Checker Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/)
Zao Crawler Zao-Crawler
Zao-Crawler 0.2b
Zao/0.1 (http://www.kototoi.org/zao/)
ZeBot ZeBot_www.ze.bz (ze.bz@hotmail.com)
ZeBot_lseek.net (bot@ze.bz)
Zealbot Mozilla/4.0 (compatible; Zealbot 1.0)
Zearchit Zearchit
Zedzo.digest zedzo.digest/0.1 (http://www.zedzo.com/)
Zerxbot zerxbot/Version 0.6 libwww-perl/5.79
Zetbot DeepIndex ( http://www.zetbot.com )
Zeus Zeus ThemeSite Viewer Webster Pro V2.9 Win32
Zeus xxxxx Webster Pro V2.9 Win32
ZBot/1.00 (icaulfield@zeus.com)
ZipppBot ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net)
ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net)
Zippy Zippy v2.0 - Zippyfinder.com
ZoomSpider ZoomSpider - wrensoft.com
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR
1.1.4322)
Zspider zspider/0.9-dev http://feedback.redkolibri.com/
ZyBorg (LookSmart) ZyBorg
ZyBorg (Wisenut) Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnu
tbot.com)
ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com)
aiHitBot Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
eCairn-Grabber eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15




Webmaster tool: free crawlers and spiders tracking script- SEO script - Outil pour webmaster: script gratuit de statistiques des visites des robots    Mentions légales