Skip to the main content

Free and open source since 2005, running on your own server

CrawlTrack Webmaster dashboard
CrawlTrack 3.3.2

Your web statistics tool, built for the other 94 percent of hits

CrawlTrack is a php and MySQL dashboard that lives on your own server. It follows search engine crawlers minute by minute, counts your visitors, watches your indexation and blocks a good share of the hacking attempts that reach your pages.

PHP 4.3.2 or newer with GD2, one MySQL database, automatic installation in under five minutes.

Live crawler view CrawlTrack dashboard showing crawler visits, indexation and visitor statistics CrawlTrack page view statistics table
75,053 Downloads since 2005
85 percent Of hits are crawlers
30 percent Of crawler hits are attacks
1 script On your own hosting
Chart comparing the six percent of hits seen by common analytics scripts with the full hit volume

Data recorded on the www.crawltrack.net server on 13th May 2009.

The blind spot

Most analytics tools show you six percent of what happens

With nearly all the web analytics solutions on the market you know only the visitor side of your traffic. Those visits matter, they are your readers. But they are not the whole story, and the rest of the traffic shapes your ranking, your server load and your security.

CrawlTrack reads the requests your server already receives. It sees the search engine crawlers, the image crawlers, the advertising robots, the site copiers and the scanners that other scripts simply drop.

  • Crawler visits, hour by hour, with the exact pages requested
  • Visitor counts, origin, bounce rate and 404 errors
  • Daily indexation follow up in the main search engines
  • Server load, downloads and hacking attempts in one place

See what the dashboard reports

Crawler coverage

One list, a few hundred robots, updated as they appear

The crawler database is shipped with the script and can be refreshed from the tools page. Detection uses the user agent, and you can add your own crawlers by address range when a robot hides behind a generic name.

  • Googlebot
  • Google Image
  • Google WAP
  • Google Adsense
  • Bingbot
  • Yahoo Slurp
  • Baiduspider
  • Yandex
  • Alexa
  • Ask Jeeves and Teoma
  • Exabot
  • NaverBot
  • VoilaBot
  • Heritrix
  • OpenWebSpider
  • ZyBorg
  • Scrubby
  • AbachoBot
  • Boitho
  • WiseGuys

Browse the crawler list or read the user agents and addresses used by the main search engines.

Server and database illustration representing statistics stored on your own hosting
Your data stays yours

Keep control of the statistics you collect

On 14th May 2009 all Google services went down for a few hours. Numerous sites using Google Analytics became very difficult to reach during that time.

Who has access to your statistics?
The database sits on your server. You decide who reads it.
Is it wise not to know?
You can answer that better than a third party can.
Do you usually share your strategic secrets?
With CrawlTrack you do not have to. Your visits stay on your own machine.

When CrawlTrack measures the audience of your site, you have full control and depend on nobody else. Your access to the numbers depends only on you.

Site safety

Every day, someone tries the door

Webmasters regularly discover that a hacker has taken control of a server, defaced a site or destroyed files and databases. On www.crawltrack.net, CrawlTrack detects between 500 and 10,000 attacks per day.

CrawlTrack blocks numerous code injection and SQL injection attempts. It is one more barrier that an attacker has to pass, and it keeps a record of every attempt with the address, the time and the payload used.

  • Injection attempts are stopped before the page is generated
  • The attacks log shows the addresses used by the attackers
  • Bad robots and site copiers can be refused access
Illustration of a hooded attacker, the threat CrawlProtect is designed to stop
Installation

No particular technical skills required

You need a host with php version 4.3.2 or newer, the GD2 graphic library and one MySQL database. The installation is fully automatic: unzip, upload, open the folder in your browser and follow the screens.

  • Choose the language, then enter the database details
  • Name the first site, add as many others as you like
  • Create the administrator account, then place the tag
  • Wait for the crawlers, the dashboard fills itself
Requirements diagram for CrawlTrack: web server, php, GD2 library and MySQL
Start here

CrawlTrack, your web statistics tool

Download the script, unzip it on your hosting and see what your server has been trying to tell you. If you get stuck, the support forum has been answering questions since 2005.

3.3.2

Released 26th November 2011

Downloaded 75,057 times

Read the changelog