Facebook employee responds on robots.txt controversy

I received a comment from Blake Ross on the legal changes Facebook have recently made, and I wanted to highlight that response. I recommend you look at the document itself, along with both of our commentaries on it, and make up your own mind.


Hey Pete,

I work for Facebook, but this comment should not be
construed as an official company statement.

Your interpretation
of this document isn't correct and frankly doesn't make much sense. If
our goal were to make it difficult for startups to succeed using
Facebook data, we wouldn't have launched an open API that provides
access to all of our data; we wouldn't have launched the fbFund to fund
startups that are built on top of this API; and we wouldn't host an
annual developer conference to help startups use this API. The very
future of our platform is predicated on the notion that we can help
other companies improve their products by leveraging the social graph.

This crawling document exists because we've had problem where shady
companies would try to scrape user information in aggregate and use it
for malicious purposes. For instance, these companies would scrape page
by page from http://www.facebook.com/family/
and then try to resell these bulk lists.

Blake Ross


Hi Blake,
thanks for taking the time to comment. I might be willing
to give Facebook the benefit of the doubt on this, if it wasn't for the
$14,000 legal bill I just paid.

The fact is that you've made
all the information on the profile pages public, complete with
micro-format data to help crawlers. There's some simple technical fixes
you could add to solve the specific problem you mention, starting with
amending robots.txt and changing your ID formats:

You've chosen to leave all that information out in the open so
you can benefit from the search traffic, and instead try to change the
established rules of the web so you can selectively sue anyone you
decide is a threat.

I'm really pretty bummed about this because
I've been a long-time fan of Facebook, you can see me raving about your
XHProf work here:

The sad fact is, your leadership has decided to change the
open rules that have allowed the web to be such an interesting and
innovative place for the past decade.

Facebook has always been a closed system where developers are expected to live in a culture of asking permission before doing anything, and existing at the whim of the company's management. The web I love is an open world where you are free to innovate as long as you stick to the mutually agreed rules. This is a land grab by Facebook, they've moved into the open web for the commercial benefits they'll reap, but want to change the rules so they can retain absolute control.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: