Search Engine Cloaking

A Technical Discussion

Search engine cloaking is a technique used by SEOs to deliver different web pages for the same URL, depending on the identity of who is requesting them. Search engine spiders will get one page, and human visitors will get something else. Typically, the page search engine spiders receive is highly optimized for good placement in the search engine results pages, while the page a human receives is more "user friendly".

The purpose of search engine cloaking is to keep humans from being able to see the optimized HTML that is used to get good search engine rankings. It is used as a defensive measure to keep unscrupulous webmasters from stealing the HTML code, and also to present a more human-oriented (as opposed to algorithm oriented) web page to visitors.

In order to cloak effectively, a decision must be made about the identity of the requestor of the web page. This decision is best handled "server-side", before the page is sent to the requestor. Usually, the cloaked page is actually a CGI script which takes certain cues and figures out whether the requestor is a human or search engine spider. There are at least three possible indicators: the IP address, the User-Agent and/or the existence of a referrer string.

The cloaking script will use one of these indicators to decide whether or not the requestor is a search engine. If it looks at the IP address of the visitor, it compares it against a list of known search engine spider IP addresses. Similarly, if the User-Agent is used, it compares the User-Agent string of the requestor against a list of search engine spider User-Agents. If the referrer string is checked, it looks for the existence of the HTTP REFERER header, and if none is present, the script assumes the requestor is a search engine spider.

Of the above, IP address detection is considered the most reliable. User-Agents can be easily faked, and referrer-based detection is unreliable because so many firewalls strip the HTTP REFERER header, and some search engine spiders do provide this header.

Download KloakItNow!