French
protection
CrawlTrack, webmaster dashboard.
Web analytic and SEO

CrawlProtect, your website safety.
Protection against hacking, spam and content theft

Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.





diable

Custom Search






Crawlers and spiders who can visit your website

Here is the list of crawlers used by CrawlTrack; that list is updated regularly. As you will see, it's already a very long list but there is still a lot of spiders crawling on the web not yet listed here.



Crawler User agent Owner
192.com 192.comAgent 192.com
4anything 4anything.com LinkChecker v2.0 4anything
80legs Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/200
8032620
80legs
A-Online A-Online Search Aon
ABCdatos ABCdatos BotLink/5.xx.xxx#BBL ABCdatos
AOL Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) AOL
ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) ASAHA
ASPseek ASPseek/1.2.xx ASPseek
ASPSeek/1.2.x
ASPSeek/1.2.xxpre
ASPseek/1.2.9d
ASPSeek/1.2.5
ASPSeek/1.2.xa
AVSearch AVSearch-1.0(peter.turney@nrc.ca) National Research Council Canada
AbachoBot AbachoBOT (Mozilla compatible) Abacho
AbachoBOT
Abby Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html) Ellerdale
Aberja Checkoma Aberja Checkoma Aberja
Abot abot/0.1 (abot; http://www.abot.com; abot@abot.com) Abot.com
About About/0.1libwww-perl/5.47 About
AboutUsBot Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) AboutUs
Accelobot Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com) Accelovation
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com)
Accoona accoona Accoona
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Acoi AcoiRobot Acoi
Acoon Robot Acoon Robot v1.50.001 Acoon
Acoon-Robot Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com) Acoon
Acorn Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o
rg)
Isara
Activtourist Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) Activtourist
AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html) Google
Aesop AESOP_com_SpiderMan Aesop
Agada Mozilla/4.0 (agadine3.0) www.agada.de Agada
agadine/1.x.x (+http://www.agada.de)
Mozilla/4.0 (agadine3.0)
AgentName AgentName/0.1 libwww-perl/5.48 Linkomatic
Aibot AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , C
hina)
21seek
Aicrawler Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) Accoona
Aipbot aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) Aipbot
Alacra PortalBSpider/2.0 (spider@portalb.com) Alacra
Aladin.de Aladin/3.324 Abacho
Aleksika Danmark Aleksika Spider/1.0 (+http://www.aleksika.com/) Aleksika
Alexa ia_archiver Alexa
AlkalineBOT AlkalineBOT/1.3 Vestris
AlkalineBOT/1.4 (1.4.0326.0 RTM)
Allesklar.de Allesklar/0.1 libwww-perl/5.46 Allesklar
Allrati Allrati/1.1 (+) Unknown
Almaden http://www.almaden.ibm.com/cs/crawler IBM
http://www.almaden.ibm.com/cs/crawler [hc5]
Altavista Scooter2_Mercator_x-x.0 Altavista
Scooter-W3.1.2
Scooter-ARS-1.1-ih
Scooter-3.0.HD
Scooter-3.0.FS
Scooter-3.0.VNS
Scooter-3.0QI
Scooter-3.2.BT
Scooter-3.2
Scooter-3.0.EU
Scooter/2.0 G.R.A.B. X2.0
Scooter/2.0 G.R.A.B V1.0
Scooter/1.0
Scooter/1.0 scooter@pa.dec.com
Scooter/1.1 (custom)
Scooter/2.0 G.R.A.B. V1.1.0
Scooter-3.2.DIL
Scooter-3.2.EX
Scooter-3.3dev
Scooter/3.3.vscooter
Scooter-ARS-1.1
Scooter_bh0-3.0.3
scooter-venus-3.0.vns
Scooter_trk3-3.0.3
Scooter/3.3_SF
Scooter/3.3.QA.pczukor
Scooter-3.2.NIV
Scooter-3.2.JT
Scooter-3.2.SF0
Scooter-3.2.snippet
Scooter/3.3
Scooter-W3-1.0
Amfibibot Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) Amfibi
Amidalla libwww-perl/5.65 Amidalla
Annomille AnnoMille spider 0.1 alpha - http://www.annomille.it Annomille
AnsearchBot Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/) Ansearch
AnswerBus AnswerBus (http://www.answerbus.com/) AnswerBus
Answerchase PROve AnswerBot 4.0 Answerchase
Antibot antibot-V1.3.3.1/debian-linux-sarge Antidot
Any Search Info Mozilla/4.0 (Sleek Spider/1.2) Search-Info
Anzwers Australia AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) Anzwers Australia
Apexoo Spider Apexoo Spider 1.0 Apexoo
ApocalXExplorerBot ApocalXExplorerBot ApocalX
Aport Aport Aport
Appie appie 1.1 (www.walhello.com) Walhello
appie 1.1 (www.walhello.com)
Arabulbot Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html) Arabul
ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby
.com;)
Araby
Arachnoidea Arachnoidea (arachnoidea@euroseek.com) Euroseek
ArchitextSpider ArchitextSpider Excite
Archive.org_bot Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht
tp://www.archive-it.org)
Archive.org
Arexera X-Crawler Arexera
TECOMAC-Crawler/0.x
Arianna www.arianna.it Libero
Arikus_Spider Arikus_Spider Arikus
Asahina Asahina-Antenna/1.x Asahina
Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)
Ask 24x Info ask.24x.info Ask 24x
Ask Jeeves/Teoma Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Ask Jeeves
Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr
awling.html)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w
ebmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we
bmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_cra
wling.html)
Mozilla/2.0 (compatible; Ask Jeeves)
Asked asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) Asked
Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info) Askpeter
Asterias asterias/2.0 Singing Fish
Asterias Crawler Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revis
ion: 3.11
Singingfish
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s
ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revisi
on: 3.10
Astrafind! Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) Seeq
Atlocal AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) @Local
Attentio Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@att
entio.com)
Attentio
Augurnet Swiss augurfind Augurnet Swiss
augurnfind V-1.x
Axada axadine/ (Axadine Crawler; http://www.axada.de/; ) Axada
Axandra Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) Axandra
Axmo AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine
.
Axmo
Ay-Up FastBug http://www.ay-up.com Ay-up
BE Internet Search Engine Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) Rawgrunt
Ba.be Mozilla/4.72 [en] (BACS http://www.ba.be) BA
BaBoom Web Portal BaboomBot/1.x.x (+http://www.baboom.us) Baboum
BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) Babaloo
Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html) Backlink-Check
Baiduspider BaiduImagespider+(+http://www.baidu.jp/search/s308.html) Baidu.com
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.ht
ml)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
BaiDuSpider
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Balihoo Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots
meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
Balihoo
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a
spx; tgautier at balihoo dot com)
BanBot BanBots/1.2 (spider@banbots.com) Banbot
BeamMachine BeamMachine/0.5 (dead link remover of www.beammachine.net) BeamMachine
Beauty (Cosmoty) beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) uCHOOSE
BebopBot BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) Apassion4jazz
BecomeBot Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become
.com/site_owners.html)
BecomeBot
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.
com/site_owners.html)
BecomeJPBot Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom
e.co.jp/site_owners.html)
Become
BeijingCrawler BeijingCrawler Unknown
BigClique BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) BigClique
Biglotron BIGLOTRON (Beta 2;GNU/Linux) Biglotron
Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
Bigsearch
Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c
a/; info@enhancededge.com)
BilBasen FAST Enterprise Crawler 6 used by BilBasen ApS (michael@bilinfo.dk) Bilinfo
BilgiBetaBot BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html
; nutch-agent@lucene.apache.org)
Bilgi
BilgiBot BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) Bilgi
BinGet BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/) Bin-Co
Bingbot Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm) Bing
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Bisnisseek Custom Spider www.bisnisseek.com /1.0 Bisnisseek
Bitacle Robot Bitacle bot/1.1 Bitacle
Bitacle Robot (V:1.0;) (http://www.bitacle.com)
Bitlybot bitlybot Bit.ly
Blaiz Enterprises Blaiz-Bee/1.0 (+http://www.blaiz.net) Blaiz Enterprises
Blaiz-Bee Blaiz-Bee/2.00.5502 (+http://www.blaiz.net) Blaiz
Blaiz-Bee/2.00.5622 ( http://www.blaiz.net)
Blitzsuche Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) RP ONLINE
BlitzBOT@tricus.net
BlitzBOT@tricus.net (Mozilla compatible)
BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers
)
BlogRefs
BlogScope Mozilla/5.0 (compatible; BlogScope/1.0; +http://www.blogscope.net/; U of Toronto
)
U of Toronto
BlogSearch BlogSearch/1.x +http://www.icerocket.com/ Icerocket
BlogSearch/1.0 +http://www.icerocket.com/
BlogWatcher blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) Okumura Group
Blogbot Naamah 1.0.1/Blogbot (http://blogbot.de/) Blogbot
Naamah 1.0a/Blogbot (http://blogbot.de/)
Blogdex BlogBot/1.x Massachusetts Institute of Technology
Blogdimension BlogBot Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com) Blogdimension
Bloglines Bloglines Title Fetch/1.0 (http://www.bloglines.com) Bloglines
Bloglines/3.1 (http://www.bloglines.com)
Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com) Bloglines
BlogzIce BlogzIce/1.0 +http://www.icerocket.com/ IceRocket
BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)
BnF bot Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/) BnF
Boitho boitho.com-dc Boitho
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
boitho.com-robot/1.x (http://www.boitho.com/bot.html)
boitho.com-robot/1.x
Bot bot/1.0 (bot; http://; bot@bot.bot) Unknown
BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) Penn State College of Information Sciences and Technology
Botmobi Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi
nd.mobi/bot.html abuse@mtld.mobi)
Find.mobi
Brandwatch magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) Brandwatch
BravoBrian bSTOP BravoBrian SpiderEngine MarcoPolo BravoBrian
BStop.BravoBrian.it Agent Detector
BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) Brightcloud
Bruinbot BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) University of California
Btbot BTbot/0.x (+http://www.btbot.com/btbot.html) Btbot
BuildCMS crawler BuildCMS crawler (http://www.buildcms.com/crawler) BuildCMS
BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) BuiltWith
BullsEye/Intelliseek BullsEye Intelliseek
BurstFindCrawler BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawl
er@burstfind.com)
Burstfind
Buscaplus Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) Buscaplus
Butterfly Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) G
ecko/2009032608 Firefox/3.0.8
Topsy
CCBot CCBot/1.0 (+http://www.commoncrawl.org/bot.html) CommonCrawl
CEA larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) CEA
CMP libwww-perl/5.41 CMP United Business Media
CUPS PrivacyFinder Cache Bot v1.0 PrivacyBird
Camcrawler Camcrawler (+http://www.camdiscover.com/crawler.html) Sensation Internet Services
CamontSpider CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html STFC
CanadianContent Search RoboCrawl (www.canadiancontent.net) CanadianContent
RoboCrawl (http://www.canadiancontent.net)
Carleson carleson/1.0 Cosmix
Catall Spider Catall Spider Catall
Catall-Spider Catall-Spider/3.3.3(www.Catall.de) Catall
CazoodleBot CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood
lebot; cazoodlebot@cazoodle.com)
Cazoodle
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.co
m)
Ccubee ccubee/x.0 Empyreum
Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect
ion.com/bot.html )
Changedetection
Charlotte Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/) Searchme
Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)
Chen Li Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; chenlibiti@163.com) Chen Li
Christcentral ChristCRAWLER 2.0 Christcentral
Mozilla/4.0 (compatible; ChristCrawler.com, ChristCrawler@ChristCENTRAL.com)
CipinetBot CipinetBot/1.0 (http://www.cipinet.com/bot.html) Cipinet
CipinetBot (http://www.cipinet.com/bot.html)
Cityreview Robot Cityreview Robot (+http://www.cityreview.org/crawler/) Cityreview
CjLogbot Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot) CjLog
Claymont Search Claymont.com Claymont Search
CloakDetect CloakDetect/0.9 (+http://fulltext.seznam.cz/) Seznam
Clushbot Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)
Clush
Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)
Clushbot/2.x (+http://www.clush.com/bot.html)
Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html)
Cnet robot Mozilla/4.6 [en] (http://www.cnet.com/) Search.com
CoBITSProbe CoBITSProbe Academia Sinica
Cobion Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) Cobion
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
oBot ((compatible;Win32))
Combine Combine/3 http://combine.it.lth.se/ Combine
Combine/2.0
Combine/2.0 http://combine.it.lth.se/
Cometrics-bot cometrics-bot, http://www.cometrics.de Cometrics
Cometsystems Crawler (cometsearch@cometsystems.com) Cometsystems
Crawler (cometsearch@cometsystems.com)
Comperio FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) Comperio
Compete.com larbin_2.2.0 (crawl@compete.com) Compete Inc
Computerorgs htdig/3.1.6 (http://computerorgs.com) Computerorgs.com
Comrite Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen
e.apache.org)
Comrite
ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl) Convera
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
Converas RetrievalWare Convera Internet Spider V6.x Convera
ConveraCrawler/0.2
ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)
CrawlConvera0.1 (CrawlConvera@yahoo.com)
CoolBot CoolBot SuchMaschine21
Copyright sheriff copyright sheriff (+http://www.copyrightsheriff.com/) Copyright sheriff
Cortina Vision Research Lab image spider at vision.ece.ucsb.edu Vision Research Lab
CougarSearch CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml) CougarSearch
Cowbot Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Naver
Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
CrawlerBoy CrawlerBoy Pinpoint.com Motricity
Crawling jpeg Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) Yamana Laboratory - Waseda University Japan
Crawllybot Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) Crawlly
Croccrawler CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) Croccrawler.com
CsCrawler Hi! I'm CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo
glespam/crawler.html RPT-HTTPClient/0.3-3
University of Kassel
Csci_b659/0.13 csci_b659/0.13 Indiana University School of Informatics
Cuasar Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ Cuasar
CurryGuide CurryGuide SiteScan 1.1 CurryGuide
CyberAlerts Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) CyberAlerts
Cydral CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) Cydral
CydralSpider CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com) Cydral
CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com)
DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net
/aboutkr.html)
Daum
DAUM Web Robot Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi
cations Corp., Korea; +http://ws.daum.net/aboutkr.html)
Daum
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm
unications Corp., Korea)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu
nications Corp., Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm
unications Corp., Korea)
DNS-Digger Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) Dnsdigger
DailyOrbit Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit
DataFountains DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents
.shtml)
University of California
DataFountains/DMOZ Downloader
DataSpear Spider Bot DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.co
m)
DataSpear
DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html
; dssb@dataspear.com)
DataparkSearch DataparkSearch/4.xx (http://www.dataparksearch.org/) dpSearch
DataparkSearch/4.47 (+http://dataparksearch.org/bot)
DaviesBot DaviesBot/1.7 (www.wholeweb.net) Wholeweb
Daypop daypopbot/0.x Daypop
DbDig dbDig(http://www.prairielandconsulting.com) Connections
De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) De.com
DeepIndexer DeepIndexer.ca Deepindex
Deepak-USC/ISI deepak-USC/ISI University of Southern California
Deepindex Deepindex V2 Deepindex
DeepIndex (www.en.deepindex.com)
DeepIndex
Denmex Websearch Denmex websearch (http://search.denmex.com) Denmex Websearch
DepSpid Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net) DepSpid
Dev-spider2 dev-spider2.searchpsider.com/1.3b Searchspider
DiaGem Japan DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) DiaGem Japan
Die Kraehe -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de Die Kraehe
Diggit Digger/1.0 JDK/1.3.0rc3 Diggit
Direct Hit Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) Teoma
Disco-crawl disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@
discoveryengine.com)
Discoveryengine
disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@disc
overyengine.com)
Ditto DittoSpyder Ditto
DoCoMo DoCoMo/1.0/Nxxxi/c10 NTT DoCoMo
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.go
o.ne.jp/door/crawler.html)
DoCoMo/1.0/Nxxxi/c10/TB
DoCoMo/2.0 P900iV(c100;TB;W24H11)
Dodgebot dodgebot/experimental Agmlab
DotBot DotBot/1.0.1 Dotnetdotcom
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne
tdotcom.org)
Doubanbot Doubanbot/1.0 (bot@douban.com http://www.douban.com) Douban
Download-Tipp Download-Tipp Linkcheck (http://download-tipp.de/) Download-Tipp
EyeCatcher (Download-tipp.de)/1.0
Drecombot Drecombot/1.0 (http://career.drecom.jp/bot.html) Drecom Japan
DtSearchSpider dtSearchSpider dtSearch
DuckDuckBot DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) DuckDuckGo
Dumbot Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) DumbFind.com
Dumbot(version 0.1 beta - dumbfind.com)
Dumbot(version 0.1 beta)
E-SocietyRobot e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) Yamana Laboratory
E-StyleISP eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) e-StyleISP
EApolloBot eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at
global-opto dot com)
EApollo
EMPAS_ROBOT EMPAS_ROBOT Empas
ESISmartSpider ESISmartSpider smart-spider.com
Earthcom Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info) Earthcom
EARTHCOM.info/1.xbeta [www.earthcom.info]
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
EARTHCOM.info/1.x [www.earthcom.info]
EasyDL EasyDL/3.xx http://keywen.com/Encyclopedia/Bot Keywen
EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
EasyDL/3.xx
Echo.com Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 Echo.com
Echo.fr EchO!/2.0 Echo.fr
Egothor Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) Charles University in Prague
Egotobot EgotoBot/4.8 (+http://www.egoto.com/about.htm) Egoto.com
Elfbot elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) uCHOOSE
Elsop LinkScan/11.0beta2 Unix LinkScan
LinkScan/x.x Unix
LinkScan/9.0g Unix
EmeraldShield.com Web Spider EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) Emeraldshield
Enfish Tracker Enfish Tracker Enfish
Enoola enoola (http://www.enoola.com) Enoola
Enterprise Search Search/1.0 (http://www.innerprise.net/es-spider.asp) Innerprise
Enterprise_Search/1.0.xxx
Enterprise_Search/1.0
ES.NET_Crawler/2.0 (http://search.innerprise.net/)
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
Entireweb Speedy_Spider (http://www.entireweb.com) Entireweb
Speedy Spider (Beta/x.x; speedy@entireweb.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
WorldLight
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) Envolk
envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
EroCrawler EroCrawler EroCrawler
Eruvo-bot eruvo-bot 4.8.1 (http://www.eruvo.com) Eruvo
EuripBot EuripBot/0.4 (+http://www.eurip.com) PreCheck Eurip.com
EuripBot/0.4 (+http://www.eurip.com) GetFile
EuripBot/0.5 (+http://www.eurip.com) PreCheck
EuripBot/0.2 (+http://www.eurip.com) GetRobots
Euro-spider Euro-Spider Shopping 1.0 Euro-spider
Evaal Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com) Evaal
EvaalSE EvaalSE - bot@evaal.com Evaal
Eventax eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) Eventax
Everest-Vulcan Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra
wlerhelp)
Vulcan
Evrinid-RT Mozilla/5.0 (compatible; Evrinid-RT; +http://www.evri.com/evrinid) Evri
Exabot Exalead NG/MimeLive Client (convert/http/0.120) Exalead
Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)
NG/2.0
Exabot/2.0
ExaBotTest/3.0
Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko)
Exabot-Test/1.0
Exabot/3.0
ExaBotTest/2.0
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Exabot-Images Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) Exalead
NG/4.0.1229
Exabot-Images/1.0
ExactSEEK eseek-larbin_2.6.2 (crawler@exactseek.com) ExactSEEK
ExactSeek Crawler/0.1
exactseek-crawler-2.63 (crawler@exactseek.com)
exactseek.com
exactseek-pagereaper-2.63 (crawler@exactseek.com)
ExactSeek_Spider ExactSeek_Spider ExactSeek
Excalibur Excalibur Internet Spider V6.5.4 Convera
Execrawl Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) Execrawl
FAST-WebCrawler FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/
faqfastwebcrawler.html)
FAST
FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch
/faqfastwebcrawler.html)
FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastw
ebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support
/crawler.asp)
FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net)
FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net)
FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/fa
qfastwebsearch/faqfastwebcrawler.html)
fastlwspider/1.0
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST Enterprise Crawler/6.4.18 (crawler@fast.no)
FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea
rch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear
ch/faqfastwebcrawler.html)
FDSE Mozilla/4.0 (compatible; FDSE robot) Abadoor
FaXobot Faxobot/1.0 Faxo
Factbot Factbot 1.09 (see http://www.factbites.com/webmasters.php) Factbites
factbot : http://www.factbites.com/robots
FairShare Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairsh
are.cc)
FairShare
Fast Search PycURL FAST
Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de) Fastbot
FauBot Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/) Buzzvolume
Favo.eu crawler favo.eu crawler/0.6 (http://www.favo.eu) Favo
Feed24 Feed24.com Feed24
FeedChecker FeedChecker/0.01 University of Tokyo
Feedfetcher-Google Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) Google
Feedfinder feedfinder/1.371 Python-urllib/1.17 +http://www.aaronsw.com/2002/feedfinder/ Feedfinder
Felix Felix - Mixcat Crawler (+http://mixcat.com) MixCat
Filangy Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi
langy-agent@filangy.com)
Filangy
FindLinks findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) University of Leipzig
http://wortschatz.uni-leipzig.de/findlinks/
findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Findexa Crawler Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) Findexa
FineBot FineBot Finesearch
Firefly Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) Fireball
Firefly/1.0
FirstGov FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov U.S.Government
Firstsbot firstsbot Firstsfind
Flapbot Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor perio
d com)
Flaptor
Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.
flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
Flatland Industries
great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan
dindustries.com/flatlandbot.php; jason@flatlandindustries.com)
flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu
stries.com/flatlandbot.php; jason@flatlandindustries.com)
FlickBot FlickBot 2.0 RPT-HTTPClient/0.3-3 DivX.com
Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se
archhippo.com)
Searchhippo
FnooleBot FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) Fnoole
Folkd.com Spider Folkd.com Spider/0.1 beta 1 (www.folkd.com) Folkd
ForAll.pl-Crawler ForAll.pl-Crawler/1.0 ForAll
Francis Francis/1.0 (francis@neomo.de http://www.neomo.de/) Neomo
FreshNotes crawler FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com FreshNotes
FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com
Freshmeat freshmeat.net URL validator/1.1 Freshmeat
FuchsBot FuchsBot +http://www.fuchsbot.tld FuchsBot
FurlBot Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn
.furlbot@looksmart.net)
Furl
FuseBulb FuseBulb.Com FuseBulb
FyberSpider FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) FyberSearch
FyberSpider (+http://www.fybersearch.com/fyberspider.php)
GAIS Robot GAIS Robot/1.0B2 Seed
GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.co
m)
Unknown
GPU p2p crawler Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin
e.php)
GPU
GSiteCrawler GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) GSiteCrawler
Gaaz gazz/x.x (gazz@nttrd.com) Infobee
Gaisbot Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Gais
Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot Mozilla/4.0 (compatible; www.galaxy.com) Galaxy
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Gamekitbot gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) Uchoose
GammaSpider GammaSpider/1.0 Gammasite
GenieKnows Mozilla/5.0 (wgao@genieknows.com) GenieKnows
larbin_2.6.3 (wgao@genieknows.com)
Mozilla/5.0 wgao@genieknows.com
geniebot wgao@genieknows.com
GeonaBot GeonaBot 1.x; http://www.geona.com/ Geona
Georgia Institute of Technology larbin_2.6.2 (listonATccDOTgatechDOTedu) Georgia Institute of Technology
Geourl Mozilla/5.0 (compatible; geourl/2.0b16 - http://geourl.org/bot) Geourl
GigaBaz Brainbot MicroBaz Gigabaz
gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/)
Gigabot Gigabot/3.0 (http://www.gigablast.com/spider.html) Gigablast
Gigabot/2.0
Gigabot/2.0/gigablast.com/spider.html
Gigabot/2.0; http://www.gigablast.com/spider.html
Gigabot/2.0att
Girafabot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf
a dot com; http://www.girafa.com)
Girafa
GlobalQueue Look.com Multi-mode
GnodSpider GNODSPIDER (www.gnod.net) Gnod
GoForIt GoForIt.com GoForIt
GOFORITBOT ( http://www.goforit.com/about/ )
Goblin Goblin/0.9 (http://www.goguides.org/) GoGuides
Goblin/0.9.x (http://www.goguides.org/goblin-info.html)
Gonzo1 gonzo1[P] +http://www.suchen.de/popups/faq.jsp T-info
Gonzo2 gonzo1[P] mailto:crawleradmin.t-info@telekom.de T-info
gonzo2[P] +http://www.suchen.de/faq.html
gonzo2[P] mailto:crawleradmin.t-info@telekom.de
Goo (Japan) Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Google Keyword Tool Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google.com/select
/KeywordToolExternal)
Google Keyword Tool
Google-Adsense Mediapartners-Google/2.1 Google
Mediapartners-Google
Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
Google-Image Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html) Google
Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
Googlebot-Image/1.0
Google-Sitemaps Google-Sitemaps/1.0 Google
Google-WAP Google WAP Proxy/1.0 Google
Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com
GoogleBot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Google
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible;
Googlebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/)
Googlebot/2.0 (+http://googlebot.com/bot.html)
Googlebot/2.1w (+http://googlebot.com/bot.html)
Googlebot-w/2.1 (+http://googlebot.com/bot.html)
Googlebot/1.0 (googlebot@googlebot.com)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog
lebot/2.1; http://www.google.com/bot.html)
Googlebot/1.0
Googlebot/2.0 beta (googlebot@googlebot.com)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/Test ( http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.google.com/bot.html)
no-user-agent-in-crawltrack-for-that-bot
Googlebot-Mobile Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b
ot.html)
Google
KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google
bot-Mobile/2.1; +http://www.google.com/bot.html)
Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) Greaterera
GrigorBot GrigorBot 0.8 (http://www.grigor.biz/bot.html) Grigor
Gromit Gromit/1.0 Australasian Legal Information Institute
Grub-client Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru
b.org)
Grub
Gsa-crawler gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) IBM
gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw)
Gulliver Gulliver/1.2 Northernlight
Gulliver/1.3
GulperBot Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) University of New-York
Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim
/cgi-bin/Link/GulperBot)
Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) Gungho
GurujiBot GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) Guruji
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
Harvest-NG Harvest-NG/1.0.2 Harvest-NG
Hatena Antenna Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) Hatena Antenna
HatenaScreenshot HatenaScreenshot/1.0 (checker) Hatena
HatenaScreenshot/1.0 (checker)
Hbtronix.spider hbtronix.spider.2 -- http://hbtronix.de/spider.php Hbtronix
HeinrichderMiragoRobot HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) Mirago
Helix Helix/1.x (+http://www.sitesearch.ca/helix/) SiteSearch
HenriLeRobotMirago HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) Mirago
HenryTheMiragoRobot HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) Mirago
Heritrix Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs
.washington.edu/research/networking/websys/)
University of Washington
archive.org_bot
mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org)
Heritrix L3S Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec
ts/crawling/)
L3S Research Center
Heritrix/1.4.0 Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) Chepi
Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) Hermits Search
Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider Hiigle
Hitwise Spider Hitwise Spider v1.0 http://www.hitwise.com Hitwise
Holmes holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl) Szukaj.onet
holmes/x.x
holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/3.11 (http://morfeo.centrum.cz/bot)
holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
HolmesBot HolmesBot (http://holmes.ge) Holmes
HomePageSearch HomePageSearch(hpsearch.uni-trier.de) HomePageSearch
Homerbot Homerbot: www.homerweb.com Homerweb
Honda-Search Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda
-search.com)
Honda-Search
Hoowwwer HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:
crawler-info<at>hiit.fi)
NGIR
HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>
hiit.fi)
Htdig htdig/3.1.x (root@localhost) ht:/dig
Htdig/3.1.6 htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) Acad?mie de Toulouse
I1searchbot i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search
.com)
I1search
ICC-Crawler ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c
ontact(at)ml(dot)nict(dot)go(dot)jp)
NICT
ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con
tact(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a
t)ml(dot)nict(dot)go(dot)jp)
ICCrawler ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) ICCenter
ICRA_Label_spider ICRA_label_spider/x.0 Icra
IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) Id-search
IIITBOT IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr a
t iiit dot ac dot in)
Webkhoj
INGRID Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://webmaster.
ilse.nl/jsp/webmaster.jsp)
Ilse
IP2MapBot IP2MapBot/1.1 http://www.ip2map.com Ip2Map
IPiumBot IPiumBot laurion(dot)com Laurions
IRLbot IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) Texas A&M University
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)
IScraperBot IScraperBot/0.1 Unknown
IWAgent IWAgent/ 1.0 - www.brandprotect.com Brandprotect
Iaskspider2 iaskspider2 (iask@staff.sina.com.cn) Sina
Ichiro ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) Goo
ichiro/1.0 (ichiro@nttr.co.jp)
ichiro/1.0 (ichiro@nttr.co.jp)
IconSurf IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) IconSurf
IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)
Icsbot icsbot-0.1 International Christian school of Seoul
Ideare ideare - SignSite/1.x Ideare
IlTrovatore IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator
e.it)
IlTrovatore
Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo
r more information please visit http://www.ilial.com/crawler; http://www.ilial.c
om/crawler; crawl@ilial.com)
Ilial
ilial/Nutch-0.9-dev
Ilse Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmeld
en.ilse.nl/?aanmeld_mode=webhints)
Ilse
Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.
ilse.nl/?aanmeld_mode=webhints)
ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com) Bdbrandprotect
IncyWincy NetResearchServer/x.x(loopimprovements.com/robot.html) LoopImprovements
IncyWincy(http://www.loopimprovements.com/robot.html)
IncyWincy/2.1(loopimprovements.com/robot.html)
IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovement
s.com/robot.html)
IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovemen
ts.com/robot.html)
IncyWincy (Look) IncyWincy(http://www.look.com) Look
IndexTheWeb IndexTheWeb.com Crawler7 IndexTheWeb
Indonesia Interactive Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia
Interactive
Indonesia Interactive
Indy Library Mozilla/3.0 (compatible; Indy Library) Unknown
InelaBot InelaBot/0.2 (+http://inelegant.org/bot) Inelegant
Inet Library Inet library Inet Library
InfoFly InfoFly/1.0 (http://www.versions-project.org/) Versions-project
InfoLab robot Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) Stanford University
InfoSec Search Bot RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht
ml; lhall@telegenetic.net)
Telegenetic
Infoseek InfoSeek Sidewinder/0.9 Go
Inria larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) Inria
Insitor Search robot Insitor.com search and find world wide! Insitor
Insitornaut Insitornaut Insitor
Internet Ninja Internet Ninja x.0 Dream Train Internet
Internetseer InternetSeer.com Internetseer
Iprospect Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) Iprospect
IpselonBot IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) Ipselon
Iseekbot iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot
com)
Iseek
Ishida Lab larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) Kyoto University
It-bot IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt
rovatore.it/bot.html; bot@iltrovatore.it)
IlTrovatore
Jabot Jabot/7.x.x (http://odin.ingrid.org/) ODIN Directory
Jabot/6.x (http://odin.ingrid.org/)
Jambot Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot
; crawler@jambot.com)
Jambot
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
Jayde Crawler Jayde Crawler. http://www.jayde.com Jayde
Jeanie jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51) Sidedc
Jetbot Jetbot/1.0 JetEye
Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de)
( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesunge
n/search/robot.html )
Neofonie
Jomjaibot Jomjaibot/1.0 Crawl (http://www.jomrank.com/, http://www.thaiwebdb.com) Jomrank
Jongaimpi jongaimpi/2.10 (jonga; http://www.jonga.co.za; info@jonga.co.za) Jonga
Jyxobot Jyxobot/x Jyxo
Jyxobot/1
K2 Spider k2spider Verity
KAIST AITrc Crawler KAIST AITrc Crawler AITrc
KFSW-Bot KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de) KFSW
KIT_Fireball KIT_Fireball/2.0 Dino-online
KS Crawler KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; crawler@kindsight.net)
Kindsight
KSbot KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte
nt/crawler/index.html; crawleradmin@knowledgestorm.com)
Knowledgestorm
KakleBot KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bo
t.html; support@kakle.com)
akle
KaloogaBot kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; info@kalooga.com) Kalooga
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw
ler@kalooga.com)
Kasparek Firefox_1.0.6 (kasparek@naparek.cz) Czech Technical University Prague
Keegeebot Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html) Keegee
Kenjin Spider Kenjin Spider Kenjin
Kevin Kevin http://dznet.com/kevin/ Dznet.com
Kevin http://websitealert.net/kevin/
KicktooBot kicktooBotV1.1 kictooBot@kictoo.com Kicktoo
Kinja-imagebot kinja-imagebot (http://www.kinja.com/) Kinja
Kinjabot kinjabot (http://www.kinja.com) Kinja
KnowItAll KnowItAll(knowitall@cs.washington.edu) University of Washington
Knowledge.com Knowledge.com/0.x knowledge.com
Krugle Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info
.html; webcrawler@krugle.com)
Krugle
Kulokobot kuloko-bot/0.x Kuloko
kulokobot www.kuloko.com kuloko@backweave.com
Kulturarw kulturarw3/0.1 National Library of Sweden
Kumm KummHttp/1.1 (compatible; KummClient; Linux rulez) Sanoma
Kyluka crawl Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k
yluka.com)
Kyluka
LECodeChecker LECodeChecker/3.0 libgetdoc/1.0 Linkexchange
LNSpiderguy LNSpiderguy Lexis-Nexis
LapozzBot LapozzBot/1.4 ( http://robot.lapozz.com) Lapozz
LapozzBot/1.5 (+http://robot.lapozz.hu)
LapozzBot/1.4 (+http://robot.lapozz.hu)
Larbin_2.6.3 larbin_2.6.3 marzia.polito@intel.com Unknown
larbin_2.6.3 larbin2.6.3@unspecified.mail
Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo
.com; webmaster@lawinfo.com)
Lawinfo
Lemur Consulting larbin_2.6.2 (tom@lemurconsulting.com) Lemur Consulting
Lexibot Mata Hari/2.00 BrightPlanet
LexiBot/1.00
Liafa larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) Liafa
LibWeb libWeb/clsHTTP -- hiongun@kt.co.kr Korea Telecom
LibertyW LibertyW (+http://www.lw01.com) LibertyW
LibertyW (+http://www.libertyw.eu)
LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co
m)
Lijit
Linguee Bot Linguee Bot (http://www.linguee.com/bot; bot@linguee.com) Linguee
LinkWalker LinkWalker Seven TwentyFour
Linknzbot linknzbot LinkNZ
Links2Go Mozilla/3.01 (Compatible; Links2Go Similarity Engine) Links2Go
Links4US-Crawler Links4US-Crawler, (+http://links4us.com/) Links4US
LinksManager.com_bot Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck
er.html)
Unknown
Llaut Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) Universitat de les Illes Balears
Lmspider lmspider (lmspider@scansoft.com) Nuance
LocalBot LocalBot/1.0 ( http://www.localbot.co.uk/) LocalBot
Lockstep Spider Lockstep Spider/1.0 Lockstep
Look.com NetResearchServer(http://www.look.com)
LookdirBot LookdirBot Lookdir
Lovel Lovel as 1.0 ( +http://www.everatom.com) Everatom
Ltaa_web_crawler larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) Ecole Polytechnique F?d?rale de Lausanne
Luchs.at URL checker luchs.at URL checker Luchs
Lycos_Spider Lycos_Spider_(modspider) Lycos
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_
Spider_Beta2(T-Rex) )
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide
r_(T-Rex) )
Lycos_Spider_(T-Rex)
MJ12bot Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+) Majestic12
MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)
MQBOT MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.ui
uc.edu)
University of Illinois
MSN Bot msnbot/0.9 (+http://search.msn.com/msnbot.htm) MSN
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
msnbot/0.3 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
MSNBOT_Mobile MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) MSN
MSRBOT MSRBOT (http://research.microsoft.com/research/sv/msrbot) Microsoft
MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/ Microsoft
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
MaSagool MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) Sagool
Mail.Ru Mail.Ru/1.0 Mail.Ru
Mainseek_Bot Mozilla/5.0 (compatible;MAINSEEK_BOT) Mainseek
Mammoth Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1 Eurekster
mammoth/1.0 (+http://www.sli-systems.com/)
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
MantraAgent MantraAgent LookSmart
Mariner Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) Kolibri
Martini Martini LookSmart
MARTINI
Marvin Marvin v0.3 Health On the Net Fondation
Masterseek MasterSeek Masterseek
Maxbot Spider/maxbot.com admin@maxbot.com Maxbot
Maxomobot maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html;
maxomobot@maxomo.com)
Maxomo
MediaCrawler MediaCrawler-1.0 (Experimental) Media Find
MediaSearch MediaSearch/0.1 WWW.FI
Mediater Rechercher libwww/5.3.2 Mediater
MegaSheep MegaSheep v1.0 (www.searchuk.com internet sheep) Search UK
Megaglobe Crawler Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com) Megaglobe
Melbot WebSpider Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.) Melbot
Mercator Mercator-1.x Altavista
Mercator-2.0
Mercator-Scrub-1.1
Merl.com larbin_2.1.1 larbin2.1.1@somewhere.com Mitsubishi Electrical Research Lab
MetaGer_PreChecker MetaGer_PreChecker0.1 MetaGer
Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) Metacarta
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com)
Metadata Labs MLBot (www.metadatalabs.com/mlbot) Metadata Labs
Metaeuro Web Crawler Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta
euro.com; crawler at metaeuro dot com)
Metaeuro
Metager-Linkchecker MetaGer-LinkChecker Metager
MetagerBot MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) Metager
Metaquerier MQbot http://metaquerier.cs.uiuc.edu/crawler University of Illinois
MQbot metaquerier.cs.uiuc.edu/crawler
Metaspinner Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner
.de/)
Metaspinner
Metatagsdir metatagsdir/0.7 (+http://metatagsdir.com/directory/) Metatagsdir
Microsoft Small Business Indexer Microsoft Small Business Indexer Microsoft
Microsoft URL Control Microsoft URL Control - 6.01.9782 Unknown
Microsoft URL Control - 6.00.8862
Miggibot A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/)
miggibot/2006.01.24
Micro-sys
Mirar Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s
ee getmirar.com)
Mirar
Missigua Locator 1.9 Missigua Locator 1.9 Unknown
Misterbot Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot
.fr)
Misterbot
Miva Miva (AlgoFeedback@miva.com) Miva
MnoGoSearch MnogoSearch/3.2.xx MnoGoSearch
Mo College 1.9 Mo College 1.9 Unknown
Moget moget/x.x (moget@goo.ne.jp) Goo
Mogimogi mogimogi/1.0 Goo
Moi-même no-user-agent-in-crawltrack-for-that-bot Moi
MojeekBot Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) Mojeek
MojeekBot/0.x (archi; http://www.mojeek.com/bot.html)
Monogol http://www.monogol.de Monogol
Moreoverbot Moreoverbot/5.00 (+http://www.moreover.com) Moreover
Morris Morris - Mixcat Crawler (+http://mixcat.com) MixCat
Mowserbot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser.
com/bot)
Mowser
MozDex mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) MozDex
Mozdex Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com) MozDex
MultiText MultiText/0.1 Virginia Polytechnic Institute and State University
Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) SWSE
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
MuscatFerret Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) Euro Ferret
Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)
Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)
MusicWalker MusicWalker2.0 (+http://www.somusical.com) SoMusical!
Mxbot Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html) Chainn
My-bytebot my-bytebot/1.0 (+http://spider.my-byte.de/info) My-byte
MyFamilyBot Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx
; SearchBot@MyFamilyInc.com)
Ancestry
Mylinea Mylinea.com Crawler 2.0 Mylinea
NASA Search 1.0 NASA Search 1.0 Unknown
NCSA NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) National Center for Supercomputing Applications
NG-SearchBot NG-Search/0.9.8 (http://www.ng-search.com) NG-Search
NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; )
NII larbin_2.6.2 (hamasaki@grad.nii.ac.jp) National Institut of Informatics (Japan)
NLCrawler Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4.
89 (like Gecko)
Northern Light
NMG Spider NMG Spider/0.3 (szukanko.com) Szukanko
NTT Directory nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) NTT Directory
NWSpider NWSpider 0.9 NWSpider
Nabot nabot_1.0 Naver
NABOT/5.0
Najdi Mozilla/5.0 (compatible; InterseekWeb/3.x) Najdi
Nameprotect NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) Nameprotect
NPBot-1/2.0
NPBot (http://www.nameprotect.com/botinfo.html)
Nationaldirectory NationalDirectory-WebSpider/1.3 Nationaldirectory
NationalDirectoryAddURL/1.0
NatzanBot NatzanBot, http://www.natzan.com Natzan
NaverBot dloader(NaverRobot)/1.0 Naver
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)
NaverBot_dloader/1.5
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.
jsp)
NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com)
NavissoBot NavissoBot Navisso
NavissoBot/1.7 (+http://navisso.com/)
Nebulla Nebullabot/2.2 (http://bot.nebulla.info) Nebulla
Nebulla/V1.0 Spider (http://spider.nebulla.de)
Nebullabot/2.2 (http://bot.nebulla.de)
Nelian Pty Ltd - Spider Nelian Pty Ltd - Spider v2.1 ( http://pcaccessoriesparts.com ) Pcaccessoriesparts
NetNose Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience:
http://www.netnose.com)
NetNose
NetResearchServer NetResearchServer/4.0(loopimprovements.com/robot.html) Loopimprovements
NetSprint -- 2.0 NetSprint -- 2.0 Wirtualna Polska
NetWhatCrawler NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com
; support@netwhat.com)
Netwhat
NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) NetinfoBot
Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) Netluchs
Netprospector Netprospector JavaCrawler Netprospector
Netscape Robozilla/1.0 DMOZ
NextGenSearchBot NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot
)
ZoomInfo
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB
ot)
NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe
nSearchBot.aspx)
NextopiaBot NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x Nextopia
NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1
NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Healthline
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro
blems contact: crawler@healthline.com
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob
lems contact: crawler@healthline.com
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe
ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
NjuiceBot Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot Unknown
Noago Spider Noago Spider Noago
NokodoBot NokodoBot/1.x (+http://nokodo.com/bot.htm) Nokodo
Norbert Norbert the Spider(Burf.com) Burf
Noxtrumbot noxtrumbot/1.0 (crawler@noxtrum.com) Noxtrum
Noyona noyona_0_1 Noyona
Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) Nsyght
NuSearch Spider nuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible
; MSIE 4.01)
NuSearch
Nutch NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
Lucene
Nutch
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.
sourceforge.net)
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis
ts.sourceforge.net)
NutchCVS/0.8-dev NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy
crawl@cs.washington.edu)
University of Washington
NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o
rg/nutch/bot.html; ec2test at lucene.com)
Amazon
OOZBOT OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) Setooz
Obidos-bot obidos-bot (just looking for books.) Weblog bookwatch
Object Sciences Corp. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3
E
Objects Search ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup
port@thesoftwareobjects.com)
SAIC
Ocelli Ocelli/1.3 (http://www.globalspec.com/Ocelli) GlobalSpec
Ocelli/1.x (http://www.globalspec.com/Ocelli)
Octora Octora Beta - www.octora.com Octora
Octora Beta Bot Octora Beta Bot - www.octora.com Octora
OmniExplorer OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer OmniExplorer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler
OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler
OnetSzukaj Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl) Szukaj
Onet.pl SA, http://szukaj.onet.pl
OpenISearch OpenISearch/1.x (www.openisearch.com) OpenISearch
OpenPortal4U DigOut4U Arisem
OpenTaggerBot OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) OpenTagger
OpenTextSiteCrawler OpenTextSiteCrawler/2.9.2 OpenText
OpenWebSpider OpenWebSpider/x OpenWebSpider
Openfind Openfind Robot/1.1A2 Openfind
Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.ht
ml)
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.
openfind.com.tw/robot.html)
OpidooBot OpidooBOT (larbin2.6.3@unspecified.mail) Opidoo
Oracle Oracle Ultra Search Oracle
OrangeBot-Mobile Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19
.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)
Orange
OrangeSpider OrangeSpider Orangeslicer
Orbiter Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) DailyOrbit
Overture-WebCrawler Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor
t/crawler.asp)
FAST
OzMonitor Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free) OzMonitor
Ozelot ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat
.de)
Flying-cat
PJspider PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) Nextopia
PWeBot PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) Programacionweb
Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)
Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) Page-store
PageBitesHyperBot PageBitesHyperBot/600 (http://www.pagebites.com/) PageBites
Page_verifier page_verifier http://www.securecomputing.com/goto/pv Securecomputing
Pagebull Pagebull http://www.pagebull.com/ Pagebull
Pages Jaunes FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) Pages Jaunes
FAST Enterprise Crawler 6 used by Pages Jaunes (fastadmin@pagesjaunes.fr)
FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com)
PalmeraBot Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi
on 0.001
Links24h
Pandora Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/
crawl.html)
National Library of Australia
ParaSite ParaSite/1.0b (http://www.ianett.com/parasite/) Ianett
PatchOneSpider Mozilla/5.0 (compatible; PatchOneSpider/1.6; http://www.patchone.se) PatchOne
Patwebbot Patwebbot (http://www.herz-power.de/technik.html) Patsearch
Peerbot PEERbot www.peerbot.com Peerbot
Petitsage www.petitsage.fr site detector 0.4 Petitsage
PicoSearch PicoSearch/1.0 Pico Search
PictureOfInternet PictureOfInternet/3.0 (http://www.malfunction.org/poi; erik@malfunction.org) Malfunction
Piffany Piffany_Web_Spider_v0.x Piffany
Piffany_Web_Scraper_v0.x
Pilot Hitlist Marketwave Hit List Pilot Hitlist
Pingdom Pingdom GIGRIB v1.1 (http://www.pingdom.com) Pingdom
PipeLiner pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) Pipeline
Pita Pita Stanford University
PlantyNet_WebRobot PlantyNet_WebRobot_V1.9 dhkang@plantynet.com PlantyNet
PlantyNet_WebRobot_V1.9 yangsam@plantynet.com
PluckFeedCrawler PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1
subscribers)
Pluck
Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin
fo.htm)
Pmoz
Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) Podtech
Pogodak Mozilla/5.0 (compatible; pogodak.ba/3.x) Pogodak
Polybot polybot 1.0 (http://cis.poly.edu/polybot/) Polytechnic University Brooklyn
Pompos Pompos/1.3 http://dir.com/pompos.html Iliad (Free)
Pompos/1.x pompos@iliad.fr
Pompos
Pompos/1.2 http://dir.com/pompos.html
Pompos/1.1 http://dir.com/pompos.html
Pompos/1.x http://dir.com/pompos.html
PopJapanSearch Robot/www.pj-search.com PopJapanSearch
Popdexter Popdexter/1.0 Popdex
PrivacyFinder/1.1 PrivacyFinder/1.1 AT&T Privacy Bird Privacy Preferences
ProWebguide ProWebGuide Link Checker (http://www.prowebguide.com) ProWebguide
Probe! PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi
a.com)
Probert Encyclopaedia
Psbot psbot/0.1 (+http://www.picsearch.com/bot.html) Picsearch
Psycheclone psycheclone Unknown
Pubblisito info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca Pubblisito
Pumpkin blogsearchbot-pumpkin-3 Artofcomputing
Purebot Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/) Puritysearch
Python-urllib Python-urllib/1.16 Unknown
QPCreep QPCreep Test Rig ( We are not indexing, just testing ) Quepasa
QihooBot Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.n
et)
Qihoo
Quantcastbot Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) Quantcast
Qube qube/2.0 (+http://qube.qelix.com/v2) Qube
QuepasaCreep QuepasaCreep v0.9.1x Quepasa
QuepasaCreep ( crawler@quepasacorp.com )
QueryN Metasearch QueryN Metasearch QueryN
Qweerybot Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com) Qweery
QweeryBot/3.01 ( http://qweerybot.qweery.nl)
QweeryBot/3.02 ( http://qweerybot.qweery.nl)
R6_CommentReader R6_CommentReader_(www.radian6.com/crawler) Radian6
R6_FeedFetcher R6_FeedFetcher_(www.radian6.com/crawler) Radian6
R6_FeedFetcher(www.radian6.com/crawler)
RAMPyBot RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu
cene.apache.org)
GiveRamp
RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveram
p.com/bot.html; support@giveRAMP.com)
RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com) RSS One
Rambot ramBot xtreme x.x Intersearch
RankMeter Web Snooper SearchUtilities
Rational SiteCheck Rational SiteCheck (Windows NT) GSInnova
Reaper Reaper/2.0x (+http://www.sitesearch.ca/reaper) Marty Anstey
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)
RedBot RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) Rediff
RedCarpet/1.2 RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) Pronto
RedCell RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) Telegenetic
RedKernel RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) RedKernel
RoboPal RoboPal (http://www.findpal.com/) FindPal
Rotondo Rotondo/3.1 libwww/5.3.1 QualiGo
RufusBot RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) Webaroo
RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html) Aktienbetreuer
S2Bot S2Bot/1.0 (http://search2.net) Search2
SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) Samsung
SBIder SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html) Sitesell
SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell
.com/contact-support.html)
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site
sell.com/contact-support.html)
SandCrawler SandCrawler - Compatibility Testing Microsoft
Savvybot savvybot/0.2 WebSavvy
ScanWeb ScanWeb Eserver
Schibstedsokbot schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/
/www.schibstedsok.no/bot/)
Schibsted
ScholarUniverse ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho
laruniverse.com)
ScholarUniverse
Scirus-crawler FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com
/srsapp/contactus/
Scirus
ScollSpider ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) Webwobot
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)
ScorpionBot Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots Fooky
ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ Ant
ScoutJet ScoutJet ScoutJet
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Scrubby Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/) Scrub the web
Scrubby/2.x (http://www.scrubtheweb.com/)
Sdcresearchlabs-testbot sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache
.org/nutch/bot.html; researchbot@shopping.com)
Shopping
Search-Engine-Studio Search-Engine-Studio Xtreeme
Search-Info Spider-Sleek/2.0 (+http://search-info.com/linktous.html) Search-Info
Search.ch search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) Search CH
search.ch V1.4
Search17Bot Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php) Search17
Search4free lwp-trivial/1.34 Search4free
SearchByUsa SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs
a.com)
Search4USA
SearchEngineWorlds ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
SearchEngineWorlds
SearchExpress Spider SearchExpress Spider0.99 SearchExpress
SearchMonkey Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; http://developer.yahoo.com/sea
rchmonkey/useragent)
Yahoo
SearchScout DiamondBot SearchScout
ClariaBot/1.0
Diamond/1.0
SearchSight SearchSight/2.0 (http://SearchSight.com/) SearchSight
SearchSpider Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspi
der.com)
SearchSpider
SearchSpider.com/1.1
SearchdayBot SearchdayBot Searchday
Searchdnabot searchdnabot/Nutch-1.0 (SearchDNA bot; http://searchenginedna.com; crawl at sear
chenginedna dot com)
Searchenginedna
Searchguild SearchGuild_DMOZ_Experiment (chris@searchguild.com) SearchGuild
Searchit-Now Robot Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) Searchit-Now
Searchmee! Spider Searchmee! Spider v0.98a Searchmee!
Seekbot Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2 SeekPort
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
Seeker.lookseek Seeker.lookseek.com Lookseek
Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/) Semager
Semager/1.0 (http://www.semager.de)
Sensis.com.au Web Crawler Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau) Sensis
SeznamBot SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/) Seznam
SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
SeznamBot/1.0 (+http://fulltext.seznam.cz/)
SeznamBot/1.0
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SharewarePlaza Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) SharewarePlaza
SheenBot SheenBot/SheenBot-1.0.4 (Sheen web crawler) Unknown
Sherlock sherlock/1.0 Indiana University School of Informatics
Shim-Crawler Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra
wl@logos.ic.i.u-tokyo.ac.jp)
University of Tokyo
Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl
er/; crawler@logos.ic.i.u-tokyo.ac.jp)
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c
rawl@logos.ic.i.u-tokyo.ac.jp)
ShopWiki ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) ShopWiki
ShopWiki/1.0 ( +http://www.shopwiki.com/)
Shoula Shoula.com Crawler 2.0 Shoula! Search
SietsCrawler SietsCrawler/1.1 (+http://www.siets.biz) Siets
Siigle Orumcex Siigle Orumcex v.001 Turkey (http://www.siigle.com) Siigle
Silk/1.0 silk/1.0 (+http://www.slider.com/silk.htm)/3.7 Slider
silk/1.0
Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html) Sirketce
Site Server 3.0 Robot Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR American College of Radiology
SiteBar SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/) SiteBar
SiteBaseBot Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru) SiteBase
SiteSpider SiteSpider +(http://www.SiteSpider.com/) SiteSpider
SiteTruth SiteTruth.com site rating system SiteTruth
SitiDiBot +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) SitiDi.net
Skampy Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] Skaffe
Skizzle User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE
.com)
Skizzle
SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) SkreemR
Slider Slider_Search_v1-de Slider
Slurp (Yahoo) Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)
Yahoo
Slurp China (Yahoo) Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Yahoo
Slurp Inktomi Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Hotbot-Lycos,NBCi etc.
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp Inktomi (Yahoo) Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071
3 Firefox/2.0.0.5
Yahoo
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch
/slurp)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE
cho/2.0.0.4
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu
rp)
Slurpy Verifier Slurpy Verifier/1.0 Yahoo
SmiffyDCMetaSpider SmiffyDCMetaSpider/1.0 Smiffysplace
SnapPreview Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 Unknown
Snapbot Snapbot/1.0 Hotlinks
Snoopy Snoopy v1.2 Unknown
SnykeBot SnykeBot/0.6 (http://www.snyke.com) Snyke
SoftHypermarket SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) SoftHypermarket
Sogou web spider Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sohu-search sohu-search Sohu
Somewhere Mozilla (Mozilla@somewhere.com) Somewhere
Sosospider Sosospider+(+http://help.soso.com/webspider.htm) Soso
Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32) Unknown
Spbot Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ ) Seoprofiler
Speedfind speedfind ramBot xtreme 8.1 Lotse
Speedy Spider Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Entireweb
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Speedy Spider (Beta/1.0; www.entireweb.com)
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s
peedyspider/)
Sphere Scout Sphere Scout&v4.0 - scout at sphere dot com Sphere
Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg
)
Singapore Press Holdings
SpiderMonkey Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) SpiderMonkey
SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
Spinn3r Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.
1); http://spinn3r.com/robot) Gecko/20021130
Spinn3r
SplatSearch libwww-perl/5.45 Splat
Spock Crawler Spock Crawler (http://www.spock.com/crawler) Spock
Sproose sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr
oose.com)
Sproose
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.c
om)
StackRambler StackRambler/x.x Rambler
StackRambler/2.0 (MSIE incompatible)
StackRambler/2.0
Steeler Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) University of Tokyo
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Strategic Board Bot Strategic Board Bot (+http://www.strategicboard.com) Strategic Board
Suchbaer suchbaer.de Suchbaer
suchbaer.de (CrawlerAgent v0.103)
Suchpad suchpad/1.0 (+http://www.suchpad.de) Suchpad
SummizeBot Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) Summize
SuperSnooper Robot@SuperSnooper.Com SuperSnooper
SurveyBot SurveyBot/2.3 (Whois Source) Domaintools
Susie !Susie (http://www.sync2it.com/susie) Sync2it
Swooglebot Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html) University of Maryland
SycaBoT SycaBoT/1.0 SycaroX
SycaBoT-Audio SycaBoT-Audio SycaroX
SycaBoT-Image SycaBoT-Image SycaroX
SycaBoT-Programme SycaBoT-Programme SycaroX
SycaBoT-Video SycaBoT-Video SycaroX
Sycrawl NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr
awl@cs.washington.edu)
University of Washington
Sygol <http://www.sygol.com/> http://www.sygol.com Sygol
SygolBot SygolBot http://www.sygol.com Sygol
SygolBot http://www.sygol.net
SynooBot SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.
html)
Synoo
Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html)
Synthesio Crawler Synthesio Crawler release MonaLisa (contact at synthesio dot fr) Synthesio
Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Syntryx Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler Syntryx
Szukacz Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl) Szukacz
Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl)
Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl)
TAGword Tagword (http://tagword.com/dmoz_survey.php) TAGword DMOZ survey
TCDBOT TCDBOT/Nutch-0.8 (PhD student research;"http://www.tcd.ie; mcgettrs at t c d dot
IE)"
Trinity College Dublin
Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/) Tags2dir
TailRank Robot Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta
ilrank.com/robot) Gecko/20021130
TailRank
Talkro Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com, Home: http://222.122.15.190/w
ebshot)
DaumSoft
TargetSeek Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/
TargetSeek.html)
Targetgroups
Tasapspider Tasapspider/0.9 (http://www.tasap.com) Tasap
Technoratibot Technoratibot/0.7 Technorati
Tecomi Bot Tecomi Bot (http://www.tecomi.com/bot.htm) Tecomi
Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w
ww.netseer.com/crawler.html; crawler@netseer.com)
Netseer
TelenetDigger TelenetDigger/1.0; htdig/3.1.6 (webdev@staff.telenet.be) Telenet
Teoma teomaagent crawler-admin@teoma.com Teoma
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
teoma_agent1
Teoma MP
teomaagent1 [crawler-admin@teoma.com]
Teradex Mapper Teradex Mapper; mapper@teradex.com; http://www.teradex.com Teradex
TeragramCrawler TeragramCrawler Teragram
TeragramWebcrawler TeragramWebcrawler/1.0 Teragram
TerrawizBot TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) Terrawiz
TheSuBot TheSuBot/0.1 (www.thesubot.de) TheSuBot
Theme Spider Theme Spider (+http://www.themespider.com/spider.html) Themespider
Theophrastus Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th
eophrastus.php)
N.A.Smith
Thumbshots-de-bot thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) Thumbshots
Thunderstones Webinator Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) Thunderstones
TimboBot timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html Breakingblogs
TinEye TinEye/1.1 (http://tineye.com/crawler.html) TinEye
TivraSpider tivraSpider/1.0 (crawler@tivra.com) AT&T
Tkensaku Tkensaku/x.x(http://www.tkensaku.com/q.html) Tkensaku
ToileBot Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1 La Toile du Qu?bec
Topodia Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia
.com/; support@topodia.com)
Topodia
Toutatis Toutatis x.x (hoppa.com) Hoppa
Toutatis x.x-x
Toutatis x-xx.x (hoppa.com)
Traazibot traazibot/testengine (+http://www.traazi.de) Traazi
Trampelpfad Trampelpfad-Spider Trampelpfad
Trampelpfad-Spider-v0.1
TrendTech http://www.trendtech.dk/spider.asp) TrendTech
Truveo Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) Truveo
Turnitin TurnitinBot Turnitin
TurnitinBot TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) Turnitin
Turnpike Emporium Turnpike Emporium LinkChecker/0.1 Turnpike Emporium Directory
TutorGigBot Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) TutorGig
TutorGig/1.5 (+http://www.tutorgig.com/crawler)
TweetmemeBot Mozilla/5.0 (compatible; TweetmemeBot/1.0; +http://www.tweetmeme.com/) TweetMeme
Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0
.6 TweetmemeBot
TwengaBot TwengaBot/1.1 (+http://www.twenga.com/bot.html) Twenga
TwengaBot/2.0 (http://www.twenga.com/bot.html)
Twiceler Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Cuill
Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Twiceler www.cuil.com/twiceler/robot.html
Twiceler-0.9 http://www.cuil.com/twiceler/robot.html
Twiceler www.cuill.com/twiceler/robot.html
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
TygoBot TygoBot Tygo
TygoProwler
Tzilla Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez
illa.de/t_st-odp-entries-agent.html)
Tzilla
UKWizz UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) UKWizz
Mackster( http://www.ukwizz.com )
UN OCHA FAST Enterprise Crawler 6 used by suckling@un.org (UN OCHA) UN OCHA
UOLCrawler UOLCrawler (soscrawler@uol.com.br) UOL
URL-Spider Mozilla/5.0 URL-Spider URL-Spider
URL_Spider_Pro URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) Innerprise
URL_Spider_Pro/x.x
URL Spider Pro/x.xx (innerprise.net)
USyd-NLP-Spider USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) University of Sydney
Ultraseek Ultraseek Go
UnChaosBot <a href="http://www.unchaos.com/"> UnChaos Bot, Hybrid Web Search Engine. </a> (
vadim_gonchar@unchaos.com)
UnCHAOS
<a href="http://www.unchaos.com/"> UnChaos </a>, From Chaos To Order, Hybrid Web
Search Engine. (vadim_gonchar@unchaos.com)
<b> UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at www.unc
haos.com </b> (info@unchaos.com)
Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt University of Koblenz
Unido-bot unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html University of Dortmund
Updated updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) Updated
Updated crawler updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
Updated
Uptimebot UptimeBot(www.uptimebot.com) Uptimebot
Uptimebot
Urlchecker1.0 Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) Euro Directory
Urlfan-bot urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html Urlfan
VBSEO Mozilla/4.0 (vBSEO; http://www.vbseo.com) VBSEO
VIProbot VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; info@vipsolutions.hu) Vipsolutions
VMBot VMBot/0.9 (VMBot; http://www.verticalmatch.com; vmbot@tradedot.com) VerticalMatch
VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com)
VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiu
c.edu
University of Illinois
Vacobot Vacobot; (+http://vaco.ws/bot.html) Vaco
Vagabondo Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http
://webagent.wise-guys.nl/)
WiseGuys
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w
ebagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt
p://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo/3.0 (webagent at wise-guys dot nl)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://
webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Vagabondo-WAP Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/
)/1.0 Profile
WiseGuys
Vakes Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) Vakes
Verizon fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt
ers@fastsearch.com)
Verizon Superpages
FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad
min@superpages.com)
Vermut Mozilla/5.0 (compatible; vermut +http://vermut.aol.com) AOL
Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com)
mozilla/5.0 (compatible; vermut +http://www.aol.com)
Versus versus 0.2 (+http://versus.integis.ch) Hochschule f?r Technik Rapperswil
Versus Crawler versus crawler eda.baykan@epfl.ch Ecole Polytechnique F?d?rale de Lausanne
VeryGoodSearch VeryGoodSearch.com.DaddyLongLegs VeryGoodSearch
Verzamelgids Verzamelgids/2.2 (http://www.verzamelgids.nl) Verzamelgids
verzamelgids.nl - Networking4all Bot/x.x
Vespa Crawler (Yahoo) Vespa Crawler Yahoo! Norway
Visbot VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) Visvo
Visbot/1.0 (+http://www.visvo.com/bot.html;bot@visvo.com)
Vlad Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html) Freshnotes
VoilaBot Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor
t.voilabot@orange-ftgroup.com)
Voila
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/
/www.voila.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi
la.com/)
KE_1.0/2.0 libwww/5.2.8
Vonna.com b o t Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) Vonna
Vortex/2.2 Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) Marty Anstey
Voyager voyager-hc/1.0 Kosmix
voyager/1.0
cfetch/1.0
Vspider vspider/3.x Verity
vspider
W3C-Validator W3C_Validator/1.432.2.10 W3C
W3C-checklink W3C-checklink/4.2 [4.20] libwww-perl/5.803 W3C
W3SiteSearch Crawler W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de W3SiteSearch
W8net Mozilla/5.0 usww.com-Spider-for-w8.net usww
WEP Search WEP Search 00 Unknown
WIRE WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) CWR
WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it)
WISEbot WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) Koreawisenut
WWW.fi crawler www.fi crawler, contact crawler@www.fi WWW.FI
WWWeasel Robot WWWeasel Robot v1.00 (http://wwweasel.de) World Wide Weasel
Wadaino.jp-crawler wadaino.jp-crawler 0.2 (http://wadaino.jp/) Wadaino
WasaBot Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/)
Wasalive
WasaBot/2.4 (+ http://www.wasalive.com ) Java/1.6.0
WasaBot 0.11 (+http://www.wasalive.com/)
Wavefire Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) Wavefire
Waypath Waypath development crawler - info at waypath dot com Waypath
Waypath Scout v2.x - info at waypath dot com
WeRelateBot WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer
elate.org)
WeRelate
WeatherBot WeatherBot v1.4 http://www.ezweather.net EZ Weather
WebAlta Crawler WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta
WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT
5.1; ru-RU)
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows
; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)
WebBOT Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot
/index.html
Carnegie Mellon University
WebCorp WebCorp/1.0 WebCorp
WebCorp/1.1
WebFindBot WebFindBot(http://www.web-find.com) web-find
WebGobbler webGobbler/1.x.x WebGobbler
WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) WebRankSpider
WebSearch WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; S
earch@WebSearch.COM.AU)
WebSearch
WebSearchBench WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsitt Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
University of Dortmund
WebSearchBench WebCrawler v0.1(Experimental)
WSB, http://websearchbench.cs.uni-dortmund.de
WSB WebCrawler V1.0 (Beta), cl@cs.uni-dortmund.de
WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive
rsit?t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de
/
WebStat WebStat/1.0 (Unix; beta; 20040314) University of South Carolina
WebVac WebVac (webmaster@pita.stanford.edu) Stanford University
WebarooBot WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) Webaroo
WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
Webbandit webbandit/4.xx.0 SoftwareSolutions
Webbot webbot(+http://webbot.com/bot.htm) Webbot
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)
Webclipping Webclipping.com Webclipping
Webcrawl webcrawl.net Webcrawl
webcrawl.net
Webduniabot Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) Webdunia
Webglimpse Webglimpse 2.xx.x (http://webglimpse.net) Webglimpse
Weblog Attitude Diffusion Weblog Attitude Diffusion 1.0 Los Alamos National Laboratory
Webmeasurement-bot webmeasurement-bot, http://rvs.informatik.uni-leipzig.de University of Leipzig
Webmeasurement-crawler webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/ University of Dortmund
WebsiteWorth WebsiteWorth v1.0 Sootle
Webspinne Webspinne/1.0 webmaster@webspinne.de Webspinne
Websquash Websquash.com (Add url robot) Websquash
Webster Webster v0.3 ( http://webster.healeys.net/ ) Webster.healeys
Webverzeichnis Webverzeichnis.de - Telefon: 01908 / 26005 Webverzeichnis
Whoiam Mozilla/5.0 whoiam [http://www.axxus.de/] Axxus
Wikiwix-bot wikiwix-bot-3.0 Wikiwix
Willow Willow Internet Crawler by Twotrees V2.1 Twotrees
WinME Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] ?
WinkBot WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB
ot; winkbot@wink.com)
Wink
WiseGuys Vagabondo/2.0 MT WiseGuys
Vagabondo/1.x MT (webagent@wise-guys.nl)
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wi
se-guys.nl/)
Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.
nl/)
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)
Worio Mozilla/5.0 (compatible; woriobot support [at] worio [dot] com +http://worio.com
))
Worio
Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/)
Worio bot Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) Worio
Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com)
Worio heritrix bot worio heritrix bot (+http://worio.com/) Worio
WorldWideWeb-X/3.1 WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/) WorldWideWeb
Wotbox Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 Wotbox
Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)
Wume_crawler wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) Lehigh University
Wwlib/Linux Wwlib/Linux Wolverhampton Web Library
Wwwster wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) Centrum f?r Informations- und Sprachverarbeitung
XP5 Project XP5 [2.03.07-111203] Marty Anstey
Xdefine egothor/3.0a (+http://www.xdefine.org/robot.html) Xdefine
Xirq xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) Xirq
Xyleme SA France cosmos/0.9_(robot@xyleme.com) Xyleme
cosmos/0.8_(robot@xyleme.com)
Y!J Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) Yahoo Japan
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Yacy yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/
en)
Yacy
Yacybot yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net Yacy
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/
bot.html
yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net
yacybot (i386 Linux 2.6.31-14-generic-pae; java 1.6.0_15; Europe/fr) http://yacy
.net/bot.html
yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net
Yahoo Japan robot Y!J-SRD/1.0 Yahoo Japan
DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear
ch/indexing/indexing-27.html)
Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)
Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Yahoo Japan
Yahoo Search Marketing crawler Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) Yahoo
YahooYSMcm/2.0.0
Yahoo! Mindset Yahoo! Mindset Yahoo! Mindset
Yahoo-Blogs Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/
us/ysearch/crawling/crawling-02.html )
Yahoo
Yahoo-MMAudVid Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co
m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo
Yahoo-MMCrawler Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) Yahoo
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Yahoo-Test/4.0 Yahoo-Test/4.0 Yahoo
Yahoo-VerticalCrawler Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com;
http://www.alltheweb.com/
Yahoo
YahooFeedSeeker YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu
blishers.html)
Yahoo
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c
om/rssguide)
YahooSeeker YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web
master@yahoo-inc.com ) (KHTML, like Gecko)
Yahoo
YahooSeeker/M1A1-R2D2 SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in
c dot com)
Yahoo
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile
-search-customer-care AT yahoo-inc dot com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i
nc dot com)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http:
//help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo
-inc dot com)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo
.com/help/us/ysearch/crawling/crawling-01.html)
MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible
;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-
01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo.
com/help/us/ysearch/crawling/crawling-01.html)
SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL
DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/
crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration
/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust
omer-care AT yahoo-inc dot com)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot
com)
SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C
LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch
/crawling/crawling-01.html)
SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0
(compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli
ng/crawling-01.html)
Yandex Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) Yandex
Yandex/1.01.001 (compatible; Win16; I)
YandexBot Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) Yandex
Yarienavoir yarienavoir.net/0.2 Yarienavoir
Yellopet spider.yellopet.com - www.yellopet.com Yellopet
YesupBot Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) Yesup
Yeti Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) 1noon
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti
Yggdrasil yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m
inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres
den dot de)
GoPubMed
YodaoBot Nokia3650/1.0 SymbianOS/6.1 Series60/1.2 Profile/MIDP-1.0 Configuration/CLDC-1.0
/ (compatible; YodaoBot-Mobile/1.0; http://www.youdao.com/help/webmaster/spider/
; )
Yodao
YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide
r/; )
YooW! YooW!/1.8.9 RC1 (+http://www.yoow.eu) YooW!
Yoogli yoogliFetchAgent/0.1 Yoogli
Yoono Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) Yoono
Yoono web-crawler yoono/1.0 web-crawler/1.0 Yoono
YottaCars YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine YottaCars
YottaShopping YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine YottaShopping
Z-Add Link Checker Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) Z-add
Zao Crawler Zao-Crawler Kototoi
Zao-Crawler 0.2b
Zao/0.1 (http://www.kototoi.org/zao/)
ZeBot ZeBot_www.ze.bz (ze.bz@hotmail.com) ZE.BZ
ZeBot_lseek.net (bot@ze.bz)
Zealbot Mozilla/4.0 (compatible; Zealbot 1.0) LookSmart
Zearchit Zearchit Zearchit
Zedzo.digest zedzo.digest/0.1 (http://www.zedzo.com/) Zedzo
Zerxbot zerxbot/Version 0.6 libwww-perl/5.79 Zerx
Zetbot DeepIndex ( http://www.zetbot.com ) Zetbot
Zeus Zeus ThemeSite Viewer Webster Pro V2.9 Win32 Cyber-Robotics
Zeus xxxxx Webster Pro V2.9 Win32
ZBot/1.00 (icaulfield@zeus.com)
ZipppBot ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) Zipp
ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net)
Zippy Zippy v2.0 - Zippyfinder.com Zippyfinder
ZoomSpider ZoomSpider - wrensoft.com WrenSoft
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR
1.1.4322)
Zspider zspider/0.9-dev http://feedback.redkolibri.com/ Redkolibri
ZyBorg (LookSmart) ZyBorg LookSmart
ZyBorg (Wisenut) Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnu
tbot.com)
Wisenut
ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com)
aiHitBot Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) Aihit
eCairn-Grabber eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15 eCairn