Hunting for the Best web analysis Program

In this blog, I will be telling you about one of our client’s search for the best fitting Web Analysis Program for its website, which incidentally gets its primary income from adverts placed on the website because of its high traffic.

Background Information

Our client is a big newspaper company based in West Africa. They’ve had a website running for sometime (primary revenue is through advertisements), and recently decided that they wanted a complete overhaul of their website and change of their current web hosting and support service provider (due to unsatisfactory support services). After a lot of rigorous examinations, we (the company I work with) were finally chosen to take over from the existing hosting and support service provider, and also produce a fresh new look for their website.

The Launch

We didn’t have any problem with producing a fresh and better look and user interface for their website. We also didn’t have any issue with providing a wonderful hosting solution for their need (we use the services of hostmysite for our clients). When the site was launched, it was all smiles in all of the quarters. These are the marked differences in the new site:

Before

Server - Apache server

Content Management System - zope content management system

Web statistics analyzer - Webalizer

Now

Server - IIS (Dedicated Server)

Content Management System – Custom created content management solution

Default Web statistics analyzer - SmarterStats

The Problems

1. Issues began when the figures generated by SmarterStats were way above what our client was accustomed to seeing with Webalizer, with some figures more than 50 times what is reported by Webalizer on the former site. (The design of the site warranted series of repeat requests from the server, which expanded the log files – generated 67GB of log in 3 months)
2. Another issue was that our client’s closest competition employs the services of our client’s former web host and thus it web analysis tool - Webalizer, in reporting activities on their website, with their report being within the range of our client’s former traffic report as produced by Webalizer.
3. And finally, there are some seemingly unexplainable figures produced by SmarterStats, like why there should be a record of about 147,000 hits, for an IP address that was recorded as having 0 visits (still baffles me).

The Search

Due to the above perceived and actual problems, our client was very reluctant to take the traffic report generated by SmarterStats into the market, thus the main source of income from their website was endangered. Then began the search for the best solution, first option: make the figures more like what they had before and remove problem 3 (above) from the reports generated. Option two: Change SmarterStats

Option one was the logical first choice, we promptly got into a reprogramming of the website, and got into contact with our web host for an explanation of the SmarterStats errors (or seeming errors), but no satisfactory answers were forthcoming. We tried to reach the owners of SmarterStats, but met a brick wall (no response). Thus we were left with option two, Change SmarterStats.

The question now was which is better, or even the best in the market? Urchin (now owned by google) was recommended and setup.

For those of you who have worked with Urchin, you know that you can run Urchin with or without its complementary javascript-based UTM (Urchin Traffic Manager). The advantage of working with the UTM is that you are given a more detailed report on visitors’ activities, than you can get without the UTM, and the 'disadvantage' is that the UTM does not track as many pages as the web server log records, and does not count .js pages as pageviews.

Thus when the Urchin software was first installed on our client’s website (without UTM), the figures generated were very similar to that produced by SmarterStats, and the suspicion of the validity of the figures persisted. However, after later configuring the UTM into the site, many things suddenly changed. Firstly, all the records the Urchin had generated prior to the installation of the UTM were changed, some slightly (hits), and others very profoundly (page views=0, sessions = 0). Secondly, some new figures generated by the report became ridiculously low (from about 200,000 page views to 45,000 page views a week), but the hits remained the same, thus producing such gigantic page view to hit and session to hit ratios. (I think Urchin takes its page views and session figures based on the input from the UTM, while generating its hits based on the input from the web server log files.

And still our client can not publish the figures generated by their web traffic analyzer.

After series of hotly debated issues during long management meetings, it was finally decided that Webalizer should be installed on the web server, but it didn’t work, Webalizer could not just manage the volume of the log files, and eventually, even it too was discarded.

Present Solution

The present solution unanimously agreed upon is to run Urchin (without UTM) and SmarterStats simultaneously. Thus we can objectively compare the reports generated by the two and thus be better informed about the true picture of the traffic on our client’s website, and also get a marketable convincing-on-first-glance report that can be advertised to potential customers.

Comments

It’s always better to run two or more web traffic analyzers simultaneously. And if your income is very much dependent on the traffic figures your website produces (as in the case of this our client), I recommend running at least one web log file analyzer, and two client based (JavaScript) web traffic analyzers (try Google analytics). You can not possibly have perfect software, but you can generate a near perfect result by pooling the strengths of different strong but imperfect soft wares, thus I would say make the best use of available resources!

Please, forward your comments to samizybiz@yahoo.co.uk