Picked up from BoingBoing:
The robots.txt file for the White House website keeps search engines from crawling a bunch of directories which include the word “iraq” in their names. Most of the directories don’t exist (the names were created by adding /iraq to a real directory), but “this robots.txt file does exclude external search engine robots from some 75 directories that actually exist on whitehouse.gov.”
For instance, a Google search of whitehouse.gov for the phrase “david kay” doesn’t pull up this page, titled “Statement by David Kay.” (An internal search of whitehouse.gov doesn’t pull it up, either, but at least gets a page that links to it.)
Recent Comments