6 ms·

How much is one terabyte of data?

46 points by Judyrabbit 3y ago

HPsquared 3y ago
1 million is 100^3 (i.e. a cube 1 metre across, contains a million 1 cm cubes) 1 billion is 1000^3 (1 m cube contains a billion 1 mm cubes) 1 trillion is 10,000^3 (I can't visualize 0.1 mm, so increase size: the big cube is now 10 m on side, with 1 mm parts) So the million and billion you could have on your desk made of parts that you can see and handle; the trillion is just a bit too big for that, you'd need a big room with a high ceiling. Edit: and, of course, you can also see a million in 2D. A sheet of grid paper 1 m across, with 1mm grid squares. Or, more practically, try counting the pixels on a 720p monitor: 921600 pixels on that screen. Use checkerboard pattern.
- bumbledraven 3y ago
  There are ink pens with 0.1 mm tips. https://www.jetpens.com/Copic-Multiliner-SP-Pen-0.1-mm-Black/pd/3385 https://www.jetpens.com/Copic-Multiliner-SP-Pen-0.1-mm-Black...
  - readingnews 3y ago
    There be wallet burning dragons here...
    - gilleain 3y ago
      There absolutely are when it comes to materials like pens, but I bought a set of 3 pens (0.1 mm, 0.3 mm, 0.5 mm) for less than ten GBP. Specifically, Mitsubishi pencil co. unipen fine line. Sure you can probably buy much more expensive examples, but it's not essential. (Side note : don't try to use a gel ink pen across varnish, or it stops working until you clean it off with solvent)
- hnuser123456 3y ago
  0.1mm is about the thickness of a sheet of paper.
  - HPsquared 3y ago
    A 0.1mm cube is like a grain of fine sand, can't really see the size of it. I couldn't imagine stacking them by hand with tweezers, but just about doable for 1mm.
  - HPsquared 3y ago
    Thinking again about it, you could have sheets of paper stacked into a cube. 1 m^2 sheets, with 0.1mm checkerboard pattern, stacked a metre high. That's some fine detail...
itslennysfault 3y ago
Interesting read, but it could REALLY use some visualization to drive the point home.
nofinator 3y ago
If you're looking for a visualization, a YouTube team of VFX artists did some neat ones last year in this video: https://youtu.be/J-K2yeQylCk https://youtu.be/J-K2yeQylCk
eulenteufel 3y ago
The first thing that comes to my mind is a quantum state of 36 qubits represented in 64-bit complex floats.
- forinti 3y ago
  The first thing I thought of was a pile of 3 million 360KB floppies.
ykonstant 3y ago
A couple of AAA games ٩(ఠ益ఠ)۶
- causi 3y ago
  Really wish games would give you a "cleanup" option in the settings. If I'm only going to use the Ultra textures it could delete all the others, and same for if someone is only going to use Medium, etc.
  - supertrope 3y ago
    As much as I want to be able to un-select game modes, ultra textures, and foreign language audio, this would increase complexity and QA required.
    - causi 3y ago
      Letting me or third-party software go in and delete the files manually without pitching a fit would be an acceptable compromise. I'd even purchase such a utility.
      - supertrope 3y ago
        Some studios have tinkered with the idea of games that can be played while the download completes. There has to be a way to trace all assets that are needed for the first chapter. https://patents.google.com/patent/US11123634B1/ https://patents.google.com/patent/US11123634B1/
        PlatinumBench 3y ago
        Blizzard has had this for a number of years, at least for World Of Warcraft. You can start playing and it downloads the rest of the game in the background. I think the loading screens are a bit longer since my guess would be that the program checks to make sure the needed files are there for the zone and, if not, it downloads them as a priority.
        3y ago
fibadi-fabadi 3y ago
* Mama joke joined in chat.
kenm47 3y ago
1 tb of searchable data is $25/mo if you do it right.
- textread 3y ago
  I am interested in the back of the envelope calculations you did to come to this conclusion. Would you please elaborate if possible? I know of an early stage YC startup that has a 6TB Postgres DB. Would it be fair to say that the DB hosting (neglecting replica, engineering time) can be done at $150/month?
dadzilla 3y ago
From Groq using Llama 2 70b: To calculate the volume of space required to contain the world's population, we need to first convert the area calculated earlier to square feet. Area (square feet) = 600 square miles × 5,280 square feet/square mile = 3,160,000,000 square feet Next, we need to assume a height at which the population can comfortably stand. Let's assume an average height of 5 feet. Volume = Area (square feet) × Height (feet) Volume = 3,160,000,000 square feet × 5 feet Volume = 15,800,000,000 cubic feet Now, we need to convert the volume from cubic feet to cubic miles. There are 1,476,333,333 cubic feet in a cubic mile, so: Volume (cubic miles) = Volume (cubic feet) ÷ 1,476,333,333 Volume (cubic miles) = 15,800,000,000 ÷ 1,476,333,333 Volume (cubic miles) = 10.7 cubic miles Therefore, to contain the entire world population, we would need a volume of approximately 10.7 cubic miles, assuming an average height of 5 feet and a density similar to that of a solid object. Please note that this calculation is purely theoretical and doesn't take into account factors like personal space, comfort, and actual population density.
- gcr 3y ago
  Please don't just paste AI-generated summaries into a low-effort comment. I don't know why your summary only picked up on the first half of the first paragraph, but estimating global population volume has nothing to do with the article. Also, even taking the assumptions at face value, your model's calculations are wrong by several orders of magnitude. There are not 5,280 square feet in one square mile and 600*5280 is not 3,160,000,000.
- cyclotron3k 3y ago
  If you weren't concerned with preserving life, you could take the current population of Earth, multiply by the average weight of a human (69kg according to Wolfram Alpha's sources), assume that we have the same density as water, and find that we could all fit in a 0.5km³ box.
xhrpost 3y ago
I think it should also be pointed out that a lot of DB's have a lot of redundant data due to indexing. When your DB is small, an index seems negligible, but once you get into millions or even billions of rows, you see first hand just how much space they can take up.
JackSlateur 3y ago
But for a city wide or even some state wide institutions, it (300rps) is really a big number. Well, I laughed. It is common for monitoring infrastructure to poll at the minute-range, and to have dozens of probes per server. 300 is really few for a whole company
favourable 3y ago
https://en.m.wikipedia.org/wiki/Zettabyte_Era https://en.m.wikipedia.org/wiki/Zettabyte_Era > The Zettabyte Era or Zettabyte Zone[1] is a period of human and computer science history that started in the mid-2010s. The precise starting date depends on whether it is defined as when the global IP traffic first exceeded one zettabyte, which happened in 2016, or when the amount of digital data in the world first exceeded a zettabyte, which happened in 2012. A zettabyte is a multiple of the unit byte that measures digital storage, and it is equivalent to 1,000,000,000,000,000,000,000 (1021) bytes. > According to Cisco Systems, an American multinational technology conglomerate, the global IP traffic achieved an estimated 1.2 zettabytes (an average of 96 exabytes (EB) per month) in 2016. Global IP traffic refers to all digital data that passes over an IP network which includes, but is not limited to, the public Internet. The largest contributing factor to the growth of IP traffic comes from video traffic (including online streaming services like Netflix and YouTube).
SushiHippie 3y ago
> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s
- thworp 3y ago
  This tripped me up as well. While modern NAND SSDs do slow down to about a third of the advertised speeds on non-sequential reads/writes, that is still 2-3 GB/s. The "normal" (SATA) SSDs are just limited by SATA's link speed, they could easily get to similar rates.
- thrwwycbr 3y ago
  "Up to xyz MB/s" This very detail is the reason USB flash drives have useless performance numbers on them, which oftentimes lead to 1/100th of the advertised performance. Everything that's bigger than the sector size (4k) is usually far lower than those numbers.
  - wtallis 3y ago
    Getting peak sequential read speed out of SSDs and USB flash drives usually requires issuing read commands that are much larger than 4kB, for two main reasons: the page size of the underlying flash memory is larger than 4kB, and SSDs and some USB flash drives stripe data access across multiple channels so you need to ask for at least a full strip of data. Queuing many commands for contiguous sequential reads can usually get close to the same performance, but has more overhead (and isn't an option for USB drives that don't support UASP).
- danking00 3y ago
  It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3]. On the other points of the article, even if you had a huge disk array plugged into the machine, how many cores can you also plug into that computer? I suppose there will always be a (healthy, productive) race here between the vertical scaling of GPUs + NVMe SSDs and the horizontal scaling of CPUs and blob storage. EDIT: formatting. [1] First Google result is Tom's hardware: https://www.tomshardware.com/features/ssd-benchmarks-hierarchy https://www.tomshardware.com/features/ssd-benchmarks-hierarc... [2] https://cloud.google.com/compute/docs/disks/local-ssd#nvme_local_ssd_performance https://cloud.google.com/compute/docs/disks/local-ssd#nvme_l... [3] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisioned-iops.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisio... [4] The ephemerality has two downsides. First, you have to get the data onto that local SSD from some other, probably slower, storage system (I haven't benchmarked GCS lately, but that's probably your best bet for quickly downloading a bunch of data?). Second, you need to use non-spot instances which are 3-6x the price.

46 points by Judyrabbit 3y ago

1 million is 100^3 (i.e. a cube 1 metre across, contains a million 1 cm cubes) 1 billion is 1000^3 (1 m cube contains a billion 1 mm cubes) 1 trillion is 10,000^3 (I can't visualize 0.1 mm, so increase size: the big cube is now 10 m on side, with 1 mm parts) So the million and billion you could have on your desk made of parts that you can see and handle; the trillion is just a bit too big for that, you'd need a big room with a high ceiling. Edit: and, of course, you can also see a million in 2D. A sheet of grid paper 1 m across, with 1mm grid squares. Or, more practically, try counting the pixels on a 720p monitor: 921600 pixels on that screen. Use checkerboard pattern.

bumbledraven 3y ago
There are ink pens with 0.1 mm tips. https://www.jetpens.com/Copic-Multiliner-SP-Pen-0.1-mm-Black/pd/3385 https://www.jetpens.com/Copic-Multiliner-SP-Pen-0.1-mm-Black...
- readingnews 3y ago
  There be wallet burning dragons here...
  - gilleain 3y ago
    There absolutely are when it comes to materials like pens, but I bought a set of 3 pens (0.1 mm, 0.3 mm, 0.5 mm) for less than ten GBP. Specifically, Mitsubishi pencil co. unipen fine line. Sure you can probably buy much more expensive examples, but it's not essential. (Side note : don't try to use a gel ink pen across varnish, or it stops working until you clean it off with solvent)
hnuser123456 3y ago
0.1mm is about the thickness of a sheet of paper.
- HPsquared 3y ago
  A 0.1mm cube is like a grain of fine sand, can't really see the size of it. I couldn't imagine stacking them by hand with tweezers, but just about doable for 1mm.
- HPsquared 3y ago
  Thinking again about it, you could have sheets of paper stacked into a cube. 1 m^2 sheets, with 0.1mm checkerboard pattern, stacked a metre high. That's some fine detail...

itslennysfault 3y ago

Interesting read, but it could REALLY use some visualization to drive the point home.

nofinator 3y ago

If you're looking for a visualization, a YouTube team of VFX artists did some neat ones last year in this video: https://youtu.be/J-K2yeQylCk https://youtu.be/J-K2yeQylCk

eulenteufel 3y ago

The first thing that comes to my mind is a quantum state of 36 qubits represented in 64-bit complex floats.

forinti 3y ago
The first thing I thought of was a pile of 3 million 360KB floppies.

ykonstant 3y ago

A couple of AAA games ٩(ఠ益ఠ)۶

causi 3y ago
Really wish games would give you a "cleanup" option in the settings. If I'm only going to use the Ultra textures it could delete all the others, and same for if someone is only going to use Medium, etc.
- supertrope 3y ago
  As much as I want to be able to un-select game modes, ultra textures, and foreign language audio, this would increase complexity and QA required.
  - causi 3y ago
    Letting me or third-party software go in and delete the files manually without pitching a fit would be an acceptable compromise. I'd even purchase such a utility.
    - supertrope 3y ago
      Some studios have tinkered with the idea of games that can be played while the download completes. There has to be a way to trace all assets that are needed for the first chapter. https://patents.google.com/patent/US11123634B1/ https://patents.google.com/patent/US11123634B1/
      - PlatinumBench 3y ago
        Blizzard has had this for a number of years, at least for World Of Warcraft. You can start playing and it downloads the rest of the game in the background. I think the loading screens are a bit longer since my guess would be that the program checks to make sure the needed files are there for the zone and, if not, it downloads them as a priority.
      - 3y ago

fibadi-fabadi 3y ago

* Mama joke joined in chat.

kenm47 3y ago

1 tb of searchable data is $25/mo if you do it right.

textread 3y ago
I am interested in the back of the envelope calculations you did to come to this conclusion. Would you please elaborate if possible? I know of an early stage YC startup that has a 6TB Postgres DB. Would it be fair to say that the DB hosting (neglecting replica, engineering time) can be done at $150/month?

dadzilla 3y ago

From Groq using Llama 2 70b: To calculate the volume of space required to contain the world's population, we need to first convert the area calculated earlier to square feet. Area (square feet) = 600 square miles × 5,280 square feet/square mile = 3,160,000,000 square feet Next, we need to assume a height at which the population can comfortably stand. Let's assume an average height of 5 feet. Volume = Area (square feet) × Height (feet) Volume = 3,160,000,000 square feet × 5 feet Volume = 15,800,000,000 cubic feet Now, we need to convert the volume from cubic feet to cubic miles. There are 1,476,333,333 cubic feet in a cubic mile, so: Volume (cubic miles) = Volume (cubic feet) ÷ 1,476,333,333 Volume (cubic miles) = 15,800,000,000 ÷ 1,476,333,333 Volume (cubic miles) = 10.7 cubic miles Therefore, to contain the entire world population, we would need a volume of approximately 10.7 cubic miles, assuming an average height of 5 feet and a density similar to that of a solid object. Please note that this calculation is purely theoretical and doesn't take into account factors like personal space, comfort, and actual population density.

gcr 3y ago
Please don't just paste AI-generated summaries into a low-effort comment. I don't know why your summary only picked up on the first half of the first paragraph, but estimating global population volume has nothing to do with the article. Also, even taking the assumptions at face value, your model's calculations are wrong by several orders of magnitude. There are not 5,280 square feet in one square mile and 600*5280 is not 3,160,000,000.
cyclotron3k 3y ago
If you weren't concerned with preserving life, you could take the current population of Earth, multiply by the average weight of a human (69kg according to Wolfram Alpha's sources), assume that we have the same density as water, and find that we could all fit in a 0.5km³ box.

xhrpost 3y ago

I think it should also be pointed out that a lot of DB's have a lot of redundant data due to indexing. When your DB is small, an index seems negligible, but once you get into millions or even billions of rows, you see first hand just how much space they can take up.

JackSlateur 3y ago

But for a city wide or even some state wide institutions, it (300rps) is really a big number. Well, I laughed. It is common for monitoring infrastructure to poll at the minute-range, and to have dozens of probes per server. 300 is really few for a whole company

favourable 3y ago

https://en.m.wikipedia.org/wiki/Zettabyte_Era https://en.m.wikipedia.org/wiki/Zettabyte_Era > The Zettabyte Era or Zettabyte Zone[1] is a period of human and computer science history that started in the mid-2010s. The precise starting date depends on whether it is defined as when the global IP traffic first exceeded one zettabyte, which happened in 2016, or when the amount of digital data in the world first exceeded a zettabyte, which happened in 2012. A zettabyte is a multiple of the unit byte that measures digital storage, and it is equivalent to 1,000,000,000,000,000,000,000 (1021) bytes. > According to Cisco Systems, an American multinational technology conglomerate, the global IP traffic achieved an estimated 1.2 zettabytes (an average of 96 exabytes (EB) per month) in 2016. Global IP traffic refers to all digital data that passes over an IP network which includes, but is not limited to, the public Internet. The largest contributing factor to the growth of IP traffic comes from video traffic (including online streaming services like Netflix and YouTube).

SushiHippie 3y ago

> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s

thworp 3y ago
This tripped me up as well. While modern NAND SSDs do slow down to about a third of the advertised speeds on non-sequential reads/writes, that is still 2-3 GB/s. The "normal" (SATA) SSDs are just limited by SATA's link speed, they could easily get to similar rates.
thrwwycbr 3y ago
"Up to xyz MB/s" This very detail is the reason USB flash drives have useless performance numbers on them, which oftentimes lead to 1/100th of the advertised performance. Everything that's bigger than the sector size (4k) is usually far lower than those numbers.
- wtallis 3y ago
  Getting peak sequential read speed out of SSDs and USB flash drives usually requires issuing read commands that are much larger than 4kB, for two main reasons: the page size of the underlying flash memory is larger than 4kB, and SSDs and some USB flash drives stripe data access across multiple channels so you need to ask for at least a full strip of data. Queuing many commands for contiguous sequential reads can usually get close to the same performance, but has more overhead (and isn't an option for USB drives that don't support UASP).
danking00 3y ago
It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3]. On the other points of the article, even if you had a huge disk array plugged into the machine, how many cores can you also plug into that computer? I suppose there will always be a (healthy, productive) race here between the vertical scaling of GPUs + NVMe SSDs and the horizontal scaling of CPUs and blob storage. EDIT: formatting. [1] First Google result is Tom's hardware: https://www.tomshardware.com/features/ssd-benchmarks-hierarchy https://www.tomshardware.com/features/ssd-benchmarks-hierarc... [2] https://cloud.google.com/compute/docs/disks/local-ssd#nvme_local_ssd_performance https://cloud.google.com/compute/docs/disks/local-ssd#nvme_l... [3] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisioned-iops.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisio... [4] The ephemerality has two downsides. First, you have to get the data onto that local SSD from some other, probably slower, storage system (I haven't benchmarked GCS lately, but that's probably your best bet for quickly downloading a bunch of data?). Second, you need to use non-spot instances which are 3-6x the price.