Spiders Have to Play by the Rules

Originally Published April 25, 2005

The Internet is the world’s largest repository of information. To cull the Internet trove efficiently, many parties have deployed automated search tools, often referred to as crawlers or spiders, to "mine" data residing on other Web sites. The "harvested" data can then be compiled in various forms. Such practices have allowed content providers to aggregate data useful and convenient for specific purposes. Some, however, have alleged that this automated practice of gathering data is a trespass, or is otherwise illegal or unsavory.

A northern California District Court in Cairo Inc. v. Crossmedia Services Incorporated, No. C 04- 04825 JW, 2005 WL 756610 (N.D. Cal. Apr. 1, 2005), recently added its two cents to this debate. Depending on your point of view, the Cairo court either upheld the right of parties to prevent automated data mining of their Web sites or jeopardized the activity of search engines and other useful Internet applications.