CrawlTrack, webmaster dashboard.
Web analytic and SEO
CrawlProtect, your website safety.
Protection against hacking, spam and content theft
Two php/MySQL scripts, free and easy to install
The tools you need to manage and keep control of your site.
Google, Bing, Yahoo or Baidu are constantly enthusiasm to search the web for new pages to add to their index. You can used CrawlTrack to find out when the crawlers went on your site, but if you want more information on IP addresses and user-agents used by the crawlers you will find them here. All visitors on your site (human or crawler) send a query to your site wincluding two different parameterd. It's the IP address from where is send the query and the User-agent (in human case, it's the browser Internet Explorer or Firefox for example who supply that User-agent). These two informations will help you to idenfied these famous crawlers.
Google use different crawler with a specific mission for each of them in theory. In practical, they often help each other.
GoogleBotThe better known, his mission is to collect datas for the Google index.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
IP addresses used:
Bing the Microsoft search engine has changed its name several times, it was formerly known as MSN search. It was for a short time Live search before taking the present name of Bing in 2009. The crawler used has not changed his name for a long time, it was always MSN Bot. But since October 2010, a new bot name Bingbot will progressively replace it.
Baidu, it's the chinese search engine. If you are looking for chinese visitors, it's the place to be.