Google Makes it Easier for News Sites to Opt-Out
In what will be seen as a concession to media baron Rupert Murdoch, Google has made it easier for news sites--such as those Murdoch controls--to opt-out of Google News.
Wed, December 02, 2009
Where they used to have to fill out an online form to opt-out of Google's news aggregation site, publishers will soon have a means to opt-out or set other options automatically, using a small file placed on their sites.
Murdoch has previously threatened to take News Corp. content, including the Times of London, and the Australian, off Google when at some point in the future they become paid sites. His Wall Street Journal and Barron's are already largely subscription-based.
As for the aggregators, "these people are not investing in journalism," Murdoch said. "They're feeding off the hard-earned efforts and investments of others."
"To be impolite, it's theft," he added.
Murdoch has also reportedly been in talks with Microsoft that would result is News Corp. content being removed from Google and enhanced on Microsoft's Bing, which would pay News Corp. a fee in return for exclusivity. Recent reports, however, say the talks have been overplayed in the media.
Google, which also attended the FTC meeting, made its announcement Tuesday in a blog post outlining extensions to the Robots Exclusion Protocol, already used to prevent Google and other search engines from indexing Web sites. The extensions will give publishers control over how their sites are treated by Google News.
"Now, with the news-specific crawler, if a publisher wants to opt out of Google News, they don't even have to contact us - they can put instructions just for user-agent Googlebot-News in the same robots.txt file they have today," wrote Google's Josh Cohen in the post.
Robots.txt is a small file that developers can place in the root directory of their Web sites that contain the Robots Exclusion Protocol commands.
"In addition, once this change is fully in place, it will allow publishers to do more than just allow/disallow access to Google News. They'll also be able to apply the full range of REP directives just to Google News. Want to block images from Google News, but not from Web Search? Go ahead. Want to include snippets in Google News, but not in Web Search? Feel free. All this will soon be possible with the same standard protocol that is REP," Cohen added.