Last week, a group of newspaper and magazine publishers signed a declaration stating that “Universal access to websites does not necessarily mean access at no cost,” and that they “no longer wish to be forced to give away property without having granted permission.” As Google Points out, the answer is simple:
User-agent: *
Disallow: /
“If a webmaster wants to stop us from crawling a specific page, he or she can do so by adding ‘‘ to the page. In short, if you don’t want to show up in Google search results, it doesn’t require more than one or two lines of code.“
they don’t want to be stranded out on the Internets…
newspapers still want to be found, they just don’t want it to be free when you find it… they don’t want google to cache their pages nor display long summaries nor give away interior pages… that’s all.
I think one problem is that
they might want to keep their content for their own subscribers etc, which is fair enough, but….
If the content is something that is going to be available elsewhere (ie a story that everyone is covering, 9/11, Michael Jackson, national politics) then it won’t make any difference as people will just get it from other sources, BBC, CNN, AP etc.
Whenit comes down to specific exclusives or some great critics, writers etc then fair enough, if that is going to be a saleable product go for it, the market will show you the way. But basic stuff? You can get that for free anywhere.
And if you find the basic content one place there might be no reason to go looking for anything else anywhere else. Maybe think of the basic content as a loss leader to get people to stay and pay for your added value materials?