Your web statistics tool, built for the other 94 percent of hits
CrawlTrack is a php and MySQL dashboard that lives on your own server. It follows search engine crawlers minute by minute, counts your visitors, watches your indexation and blocks a good share of the hacking attempts that reach your pages.
PHP 4.3.2 or newer with GD2, one MySQL database, automatic installation in under five minutes.
Data recorded on the www.crawltrack.net server on 13th May 2009.
Most analytics tools show you six percent of what happens
With nearly all the web analytics solutions on the market you know only the visitor side of your traffic. Those visits matter, they are your readers. But they are not the whole story, and the rest of the traffic shapes your ranking, your server load and your security.
CrawlTrack reads the requests your server already receives. It sees the search engine crawlers, the image crawlers, the advertising robots, the site copiers and the scanners that other scripts simply drop.
- Crawler visits, hour by hour, with the exact pages requested
- Visitor counts, origin, bounce rate and 404 errors
- Daily indexation follow up in the main search engines
- Server load, downloads and hacking attempts in one place
One list, a few hundred robots, updated as they appear
The crawler database is shipped with the script and can be refreshed from the tools page. Detection uses the user agent, and you can add your own crawlers by address range when a robot hides behind a generic name.
- Googlebot
- Google Image
- Google WAP
- Google Adsense
- Bingbot
- Yahoo Slurp
- Baiduspider
- Yandex
- Alexa
- Ask Jeeves and Teoma
- Exabot
- NaverBot
- VoilaBot
- Heritrix
- OpenWebSpider
- ZyBorg
- Scrubby
- AbachoBot
- Boitho
- WiseGuys
Browse the crawler list or read the user agents and addresses used by the main search engines.
Keep control of the statistics you collect
On 14th May 2009 all Google services went down for a few hours. Numerous sites using Google Analytics became very difficult to reach during that time.
- Who has access to your statistics?
- The database sits on your server. You decide who reads it.
- Is it wise not to know?
- You can answer that better than a third party can.
- Do you usually share your strategic secrets?
- With CrawlTrack you do not have to. Your visits stay on your own machine.
When CrawlTrack measures the audience of your site, you have full control and depend on nobody else. Your access to the numbers depends only on you.
Every day, someone tries the door
Webmasters regularly discover that a hacker has taken control of a server, defaced a site or destroyed files and databases. On www.crawltrack.net, CrawlTrack detects between 500 and 10,000 attacks per day.
CrawlTrack blocks numerous code injection and SQL injection attempts. It is one more barrier that an attacker has to pass, and it keeps a record of every attempt with the address, the time and the payload used.
- Injection attempts are stopped before the page is generated
- The attacks log shows the addresses used by the attackers
- Bad robots and site copiers can be refused access
No particular technical skills required
You need a host with php version 4.3.2 or newer, the GD2 graphic library and one MySQL database. The installation is fully automatic: unzip, upload, open the folder in your browser and follow the screens.
- Choose the language, then enter the database details
- Name the first site, add as many others as you like
- Create the administrator account, then place the tag
- Wait for the crawlers, the dashboard fills itself
Notes from the developer
CrawlTrack is written and maintained by Jean Denis Brun, with a growing team of translators and testers. The blog carries release notes, crawler changes and the occasional travel story.
CrawlTrack, your web statistics tool
Download the script, unzip it on your hosting and see what your server has been trying to tell you. If you get stuck, the support forum has been answering questions since 2005.