Comments on: How Google processes 20 petabytes of data each day http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/ EBB is the ebiquity research group\\\'s blog at the University of Maryland, Baltimore County (UMBC). We focus on technologies that facilitate the design, implementation and control of distributed, intelligent information systems -- mobile and pervasive computing, ad hoc networking, multiagent systems, knowledge representation and reasoning, and the semantic web. As the tides of technology ebb and flow, we hope the good ideas wash up on our beach and the bad ones drift back out to sea. Thu, 19 Nov 2009 00:39:45 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: Smart Grid Data About to Swamp Utilities http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-31152 Smart Grid Data About to Swamp Utilities Mon, 12 Oct 2009 22:14:41 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-31152 [...] updates energy info every 15 minutes can deliver 400 MB per smart meter per year. For perspective, Google processes about 20 PB of data per day, and 1 PB is equivalent to the amount of data contained in 20 million [...] [...] updates energy info every 15 minutes can deliver 400 MB per smart meter per year. For perspective, Google processes about 20 PB of data per day, and 1 PB is equivalent to the amount of data contained in 20 million [...]

]]> By: Kevin P. Rice http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-29223 Kevin P. Rice Fri, 24 Apr 2009 01:15:22 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-29223 correction: bits correction: bits

]]> By: Kevin P. Rice http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-29222 Kevin P. Rice Fri, 24 Apr 2009 01:14:56 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-29222 By the way... a byte is not necessarily eight bites (IEEE 1541-2002). We should be talking about octets! LOL! By the way… a byte is not necessarily eight bites (IEEE 1541-2002). We should be talking about octets! LOL!

]]> By: Kevin P. Rice http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-29221 Kevin P. Rice Fri, 24 Apr 2009 01:12:58 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-29221 ISO/IEC 80000-13:2008 has been adopted by law in some countries. Therefore, 20 PB (petabytes) = 20,000,000,000,000,000 bytes 20 PiB (pebibytes) = 22,517,998,136,852,480 bytes If you feel the difference is not worth discussing, maybe you'd like to loan the excess out of your bandwidth? ISO/IEC 80000-13:2008 has been adopted by law in some countries. Therefore,

20 PB (petabytes) = 20,000,000,000,000,000 bytes
20 PiB (pebibytes) = 22,517,998,136,852,480 bytes

If you feel the difference is not worth discussing, maybe you’d like to loan the excess out of your bandwidth?

]]> By: RpgFan http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-28397 RpgFan Thu, 26 Feb 2009 23:54:17 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-28397 4GB is actually correct. 4GB = 4000MB = 4000000KB. However, Windows (and other operating systems) use base 2 since computers use base 2. Why HDD manufacturers use the SI system while everybody else uses the binary system is beyond me. Perhaps it is due to how HDDs are designed? I'm not big on hardware though, so I'm not sure if pendrives, SD cards, flash cards and other such media as well as RAM use the SI system or the binary system. Still, the part of ISO/IEC 80000-13 dealing with binary prefixes may be standardized, so I wonder if we should start using the binary prefixes now... After all, IPv6 hasn't really caught on, despite the fact that it is such a wonderful solution. Maybe binary prefixes will go the same way. Plenty of people who implement the technology, but few people who actually adopt the usage of it... :( As for the number "20 petabytes", you really must wonder whether it was 20 "petabytes" or 20 "pebibytes". :P 4GB is actually correct. 4GB = 4000MB = 4000000KB.
However, Windows (and other operating systems) use base 2 since computers use base 2. Why HDD manufacturers use the SI system while everybody else uses the binary system is beyond me. Perhaps it is due to how HDDs are designed?

I’m not big on hardware though, so I’m not sure if pendrives, SD cards, flash cards and other such media as well as RAM use the SI system or the binary system.

Still, the part of ISO/IEC 80000-13 dealing with binary prefixes may be standardized, so I wonder if we should start using the binary prefixes now… After all, IPv6 hasn’t really caught on, despite the fact that it is such a wonderful solution. Maybe binary prefixes will go the same way. Plenty of people who implement the technology, but few people who actually adopt the usage of it… :(

As for the number “20 petabytes”, you really must wonder whether it was 20 “petabytes” or 20 “pebibytes”. :P

]]> By: Brandon Hansen http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-28380 Brandon Hansen Thu, 26 Feb 2009 17:51:59 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-28380 I just read the other day that Google is starting to keep their indexing in RAM, which is stored across 1,000 servers. That is insane. They are definitely doing some crazy stuff over there. I just read the other day that Google is starting to keep their indexing in RAM, which is stored across 1,000 servers. That is insane.

They are definitely doing some crazy stuff over there.

]]> By: omar fidel http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-18127 omar fidel Sun, 02 Mar 2008 21:55:52 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-18127 the wasted electricity to run these amusements must cost millions, when will people come to terms with efficiency. If the internet becomes the new TV, will it be as mindless, how can it not. Consumption of mass media includes infotainment, music and visual popular culture. Use sparingly, consume less and do more. Afterall everything is an abstraction from something, find the source. the wasted electricity to run these amusements must cost millions, when will people come to terms with efficiency. If the internet becomes the new TV, will it be as mindless, how can it not. Consumption of mass media includes infotainment, music and visual popular culture. Use sparingly, consume less and do more. Afterall everything is an abstraction from something, find the source.

]]> By: Nick B http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-17824 Nick B Mon, 18 Feb 2008 23:52:22 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-17824 @Simon: A petabyte can be described as 10^15 bytes or 2^50 bytes, depending on your standard... so obviously 10*2^51 = 20 * 2^50 = 20 petabytes in that particular notation. Some light reading on the topic: http://en.wikipedia.org/wiki/Binary_prefix http://en.wikipedia.org/wiki/Petabyte And this is how I finally realised why my new 4GB iPod only holds 3.73GB according to Windows... Apple describes it in base 10 (so 4 * 10^9 bytes) and Windows describes it in base 2 (about 3.73 * 2^30). I feel cheated :( @Simon: A petabyte can be described as 10^15 bytes or 2^50 bytes, depending on your standard… so obviously 10*2^51 = 20 * 2^50 = 20 petabytes in that particular notation.

Some light reading on the topic:
http://en.wikipedia.org/wiki/Binary_prefix
http://en.wikipedia.org/wiki/Petabyte

And this is how I finally realised why my new 4GB iPod only holds 3.73GB according to Windows… Apple describes it in base 10 (so 4 * 10^9 bytes) and Windows describes it in base 2 (about 3.73 * 2^30). I feel cheated :(

]]> By: Akshay Java http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-16500 Akshay Java Wed, 09 Jan 2008 23:13:20 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-16500 I have been looking into MapReduce lately and I found Hadoop, which is an open source implementation lucene.apache.org/hadoop/ There is also an interesting article on how to run MapReduce using Amazon EC2 http://wiki.apache.org/lucene-hadoop/AmazonEC2 I have been looking into MapReduce lately and I found Hadoop, which is an open source implementation lucene.apache.org/hadoop/
There is also an interesting article on how to run MapReduce using Amazon EC2
http://wiki.apache.org/lucene-hadoop/AmazonEC2

]]> By: Simon Reinhardt http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/comment-page-1/#comment-16491 Simon Reinhardt Wed, 09 Jan 2008 17:21:53 +0000 http://ebiquity.umbc.edu/blogger/2008/01/09/how-google-processes-20-petabytes-of-data-each-day/#comment-16491 >> If big numbers numb your mind, 20 petabytes is 20,000,000,000,000,000 bytes (or 22,517,998,136,852,480 for the obsessive-compulsives among us) Why would it be 2^51 * 10? Peta means 10^15 or 1 000 000 000 000 000 so 20 000 000 000 000 000 is correct. Nonetheless, very impressive. :-) >> If big numbers numb your mind, 20 petabytes is 20,000,000,000,000,000 bytes (or 22,517,998,136,852,480 for the obsessive-compulsives among us)

Why would it be 2^51 * 10? Peta means 10^15 or 1 000 000 000 000 000 so 20 000 000 000 000 000 is correct.

Nonetheless, very impressive. :-)

]]>