Don’t Let the Web Bugs Bite: How to Stop Spidering

Spiders and other automated software agents are sent out by their owners to “search and retrieve” useful information from the internet on behalf of their owners. They are invaluable tools for ensuring search engines are kept up-to-date and for market research purposes. But they are also used by companies to dig the dirt on their rivals.

To what extent is spidering legal and what can be done to prevent it? Solicitor Raffi Varoujian of City law firm Field Fisher Waterhouse discusses the options.

Spiders can be assigned a specific set of tasks to ensure that they only bring back information which is useful to the sender. This is done by providing them with either a URL or directions to a server’s file directory. Once there, it collates data from the chosen location and via any hypertext links and returns them to its owner.