How Google processes 20 petabytes of data each day
By Tim Finin on Wednesday, January 9th, 2008 at 9:20 am.The latest CACM has an article by Google fellows Jeffrey Dean and Sanjay Ghemawat with interesting details on Google’s text processing engines. Niall Kennedy summarized it this way on his blog post, Google processes over 20 petabytes of data per day.
“Google currently processes over 20 petabytes of data per day through an average of 100,000 MapReduce jobs spread across its massive computing clusters. The average MapReduce job ran across approximately 400 machines in September 2007, crunching approximately 11,000 machine years in a single month.”
If big numbers numb your mind, 20 petabytes is 20,000,000,000,000,000 bytes (or 22,517,998,136,852,480 for the obsessive-compulsives among us) — enough data to fill up over five million 4G ipods a day, which, if laid end to end would …
Kevin Burton has a copy of the paper on his blog.
Jeffrey Dean and Sanjay Ghemawat, MapReduce: simplified data processing on large clusters, Communications of the ACM, pp 107-113, 51:1, January 2008.
MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google’s clusters every day, processing a total of more than twenty petabytes of data per day.
Dean and Ghemawat conclude their paper by summarizing the key reasons why MapReduce has worked so well for Google.
Related posts: • Kowari 1.1.0 pre-release 2; • Google earth crowdsourcing map data; • Google Earth brings GIS to the everyday people;“First, the model is easy to use, even for programmers without experience with parallel and distributed systems, since it hides the details of parallelization, fault tolerance, locality optimization, and load balancing. Second, a large variety of problems are easily expressible as MapReduce computations. For example, MapReduce is used for the generation of data for Google’s production Web search service, for sorting, data mining, machine learning, and many other systems. Third, we have developed an implementation of MapReduce that scales to large clusters of machines comprising thousands of machines. The implementation makes efficient use of these machine resources and therefore is suitable for use on many of the large computational problems encountered at Google.”
UMBC eBiquity Blog
January 9th, 2008 at 12:21 pm
>> If big numbers numb your mind, 20 petabytes is 20,000,000,000,000,000 bytes (or 22,517,998,136,852,480 for the obsessive-compulsives among us)
Why would it be 2^51 * 10? Peta means 10^15 or 1 000 000 000 000 000 so 20 000 000 000 000 000 is correct.
Nonetheless, very impressive.
January 9th, 2008 at 6:13 pm
I have been looking into MapReduce lately and I found Hadoop, which is an open source implementation lucene.apache.org/hadoop/
There is also an interesting article on how to run MapReduce using Amazon EC2
http://wiki.apache.org/lucene-hadoop/AmazonEC2
February 18th, 2008 at 6:52 pm
@Simon: A petabyte can be described as 10^15 bytes or 2^50 bytes, depending on your standard… so obviously 10*2^51 = 20 * 2^50 = 20 petabytes in that particular notation.
Some light reading on the topic:
http://en.wikipedia.org/wiki/Binary_prefix
http://en.wikipedia.org/wiki/Petabyte
And this is how I finally realised why my new 4GB iPod only holds 3.73GB according to Windows… Apple describes it in base 10 (so 4 * 10^9 bytes) and Windows describes it in base 2 (about 3.73 * 2^30). I feel cheated
March 2nd, 2008 at 4:55 pm
the wasted electricity to run these amusements must cost millions, when will people come to terms with efficiency. If the internet becomes the new TV, will it be as mindless, how can it not. Consumption of mass media includes infotainment, music and visual popular culture. Use sparingly, consume less and do more. Afterall everything is an abstraction from something, find the source.