The evolving world of corporate web filtering

Eamonn Doyle, Bloxx

Photo of Eamonn Doyle Eamonn Doyle, managing director of Bloxx, charts the evolution of the web filtering marketplace and provides an insight into its future direction.

During the 50 years of BCS' existence, even more interesting than the evolution of technology itself has been the impact these changes have made on our world. A world without computer power as we know it suggests a world akin to a peasant lifestyle in some far off Third World land, or else a Victorian industrial landscape.

The most significant change during those 50 years is the arrival of the internet. The power of the web and the way we work, shop, keep in touch, learn, entertain and educate ourselves has been turned on its head.

As the internet grew so too did the need for control and management. For organisations a lack of control can lead to a degradation of expensive resources that are meant to be serving the needs of the business user and its clients.

It became clear that some sort of management was needed. Bandwidth was expensive so demand from home was low and restricted in performance. In the workplace, employee use was growing but to protect bandwidth, internet policies were required and access limited.

Early web management was centred around the need to cache regularly requested websites but it quickly became clear that managing the types/sizes of file downloads was essential, as was the need for some sort of 'blocking' to exclude seedier websites on the World Wide Web, not to mention the plain distraction of other sorts of sites accessible to staff within the workplace (the volume of which was growing exponentially).

The earliest solutions for managed web access were software-based. Workers once 'knew' where they needed to go on the internet and so the 'black and white' lists of known sites that staff were allowed or not allowed to go to were launched.

Internet site growth, allied to the emergence of powerful web-search engines, quickly meant these lists had to be supplemented by a wider categorization of the internet and broader lists of sites within categories. Typically, these new 'web filtering' suppliers searched for thousands upon thousands of web addresses; their staff then categorized the website and added it into their URL database, under whichever heading was the most appropriate.

The internet usage policies that previously outlined to workers which URL categories/internet access was permissible could now be 'enforced' by directing user PCs to 'proxy' servers that checked a user's URL request against the URL database/service level the particular user was allocated.

Commercially the initial lack of choice in the marketplace for corporate web filtering solutions created a 'land grab' situation as customers flocked to the few providers able to offer a filtering service. Considerable sums were invested in developing and marketing these early solutions thereby increasing awareness of their availability.

Significant cash inflows were secured  when those few suppliers decided to sell their software solutions on a cost per user sales model (whereby clients pay according to the size of network and pay an annual licence fee to continue using the product).

The need for and scope of internet security has grown significantly. Clients no longer worry about a little spam or one nasty website landing on a desktop. In web security, a plethora of issues such as viruses, spyware, phishing, inappropriate workplace use of Instant Messaging etc. have all emerged as significant threats.

New suppliers have joined the IT security marketplace. The marketplace that originally provided 'standalone' solutions to protect networks from multiple web threats has changed forever. The growing volume of threats and widespread use of the 'cost per user' model saw tracts of IT budget being tied up in costly annual licence renewals - as well as a need for internal IT staff to invest significant time managing an abundance of solutions and suppliers.

There has been significant merger and acquisition activity in the IT security sector. This and improving hardware performance has enabled the launch of corporate Unified Threat Management (UTM) devices. 'Old habits die hard,' they say and this is seen through confusing pricing models that try and charge for UTM security appliances, then charge per user for some of the application - but this will not last as users across the corporate and central/local government network community rebel against paying too much for too little.

The next generation of security companies in the market today did not raise investment off the back of the frantic internet boom years and they did not squander cash trying to work out what they would do with the internet and the cash they had managed to raise! Nor for that matter did these new firms spend money establishing the security market.

These are young businesses with high-calibre teams determined to use latest techniques to offer advanced security solutions - with the benefit of not having legacy products being retained due to the revenues associated with historically excessive support charges. The products being brought to market today provide new ways of tackling old issues, but much more effectively - and one area where this change is clear is in web filtering.

Research suggests that the web filtering supplier with the largest URL database in the world (against which to categorize websites and allocate access rights) boasts in the order of 18 million URLs categorized. This sounds large but taken in context there are an estimated 8 billion URLs in the 'surface web' whilst the 'deep web' is estimated to be 400-600 times this size.

With some 7.5 million new URLs created daily, then plainly this must mean clients are purchasing solutions that in fact can only categorize a tiny percentage of the internet - it makes you question what people have been buying!

Some security companies have developed 'active' solutions whereby at the point a website is requested, the web filter searches for inappropriate content (this varies from solution to solution but could be text or image based, for instance) and blocks it from the user desktop if necessary.

This risks performance issues as a result of having to examine every URL request rather than allowing the majority of, for instance, business sites to be quickly accepted.

The latest and current approach provides multi-layered protection on high-performance appliances. By combining a known database of URLs/categories with active scanning for 'suspect' web requests, large corporate users can at last invest in all-in-one web threat protection appliances.

Our industry has always made claims about the size of their URL databases. The fact is that these lists have been compiled over years and with restricted resource. Whatever the claims made for URL list sizes, they are not big enough to solely afford users adequate protection.

The next few years will see the arrival of URL filtering technology which enables much more comprehensive analysis of website content and categorization. The issue will no longer be how many URLs a database solution provides but rather how effectively URL databases perform when they are many times greater in size compared to today.

The review and URL categorization process for databases is limited when predominantly done by humans (accuracy, cost, volume) and will be replaced by highly effective software capable of accurately identifying website content, whatever the category of website it may belong to – and without the use of current limited 'keyword' technology. Language will become irrelevant to this process.

Working in conjunction with this URL list technology will be 'active' categorization which provides the same back office functionality described above but on a local appliance, quickly, effectively - and without the nasty cost per user pricing model so hated by the IT managers of larger networks.

The internet will continue to grow in value to us all and continue to change our lives. The security needed to protect ourselves and those around us is catching up fast and in web filtering at least will provide solutions that are truly effective managers of the entire internet, simply packaged and at an affordable cost.

Appliance solutions suitable for all networks offering effective internet filtering are already here and getting more capable all the time - the days of tolerating excessive charges for inadequate web filtering security are finally drawing to an end.

In 50 years time, at the 100th anniversary of BCS, network managers will compare their latest sophisticated network security solutions with web filtering security that is the equivalent in motoring of a Ford Model T!

Eamonn Doyle has a background in business studies and marketing allied to a career developing sales in IT businesses. He worked for 15 years with Castle Computer Services, 9 years of which he was MD. In addition to his present role as managing director of UK-based IT security company Bloxx, Eamonn is non-executive director of City Information Services. For further information please visit