10 ms·

TFA, to me, touches about something I've wondered about a very long time ago: what are the implications of CPU and storage growing at much faster rates than hum

TFA, to me, touches about something I've wondered about a very long time ago: what are the implications of CPU and storage growing at much faster rates than human population? Back in the 486 days you wouldn't be keeping, in RAM, data about every single human on earth (let's take "every single human on earth" as the maximum number of humans we'll offer our services to with on our hypothetical server). Nowadays keeping in RAM, say, the GPS coordinates of every single human on earth (if we had a mean to fetch the data) is doable. On my desktop. In RAM. I still don't know what the implications are. But I can keep the coordinates of every single humans on earth in my desktop's RAM. Let that sink in. P.S: no need to nitpick if it's actually doable on my desktop today. That's not the point. If it's not doable today, it'll be doable tomorrow.

moeny 4y ago
's/(human|desktop)/smartphone/g'
- untech 4y ago
  Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.
  - cellis 4y ago
    Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.
    - toast0 4y ago
      Lat long and assuming at ground gets you most of the way there.
      - Dylan16807 4y ago
        "Add it all up, and the US has around 340 billion square feet of building stock[3]. This is about 12,200 square miles, or 0.00032% of US land area" Judging by that, you need a negligible increase in the number of locations you can represent to handle everywhere stable and off the ground someone could be. Much less than one bit per person. If you want to deal with people currently in airplanes then you could give them an extra couple bytes. It's less than a million people so it won't affect your total storage at all.
    - withinboredom 4y ago
      GPS altitude is notoriously bad via consumer devices. It's better to store altitude as less than 2 bytes with the range from being a few km below sea level to the maximum altitude consumer devices will report (3,000 km -- though maybe less because there is also a speed limit at which they will stop reporting too so you can't just buy a GPS device and build a guided missile).
    - ilyt 4y ago
      Most people don't live in caves or in the sky city, 2 is enough. And in places you care (multi-floor buildings) you aren't getting GPS signal inside anyway...
  - Dylan16807 4y ago
    The planet has 2^47 square meters of surface, so more like 6 bytes. Plus you can group together people in the same area and/or sort positions as integers and store only the deltas between them, so you can probably get down to 2-3 bytes per person. And you can get dozens of models of smartphone with 16GB of RAM right now. So there might be a gap there but it's a very small gap. The phone of tomorrow will have the RAM. Edit: Thinking about it more, with 2^33 people and 2^47 locations the average delta would be 2^14, and it's pretty easy to guarantee that fits into 2 bytes per person. And with a more accurate world population count you'd free up at least a gigabyte for your phone to actually operate with.
  - renonce 4y ago
    You can build that phone. Phones made an engineering decision to put more battery rather than more RAM, and it's just a matter of putting more LPDDR5 chips onto the circuit board.
Waterluvian 4y ago
In undergrad I had a bonus GIS lab assignment to complete the task using the prof’s instructions from the 80s.. maybe 90s? (a lot of FORTRAN) and then complete it again using modern GIS software. Such an eye opener. The thing that stuck out the most to me was how many hoops we jumped through to batch the job and commit the incremental results to disk because, bahahahha, fitting even 1% of it in RAM was out of the question.
narag 4y ago
TFA, to me, touches about something I've wondered about a very long time ago... It was a little more than ten years ago for me. I realized that a hard disk could store a database of every human alive, including basic information, links (family relations) and maybe a photo. I still don't know what the implications are. Maybe we don't want to know, but it's not really that difficult to think about.
- doublepg23 4y ago
  Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (ex: I have a co-worker who's name is incorrectly spelled three different ways in prod services) the hard part?
  - withinboredom 4y ago
    Data hygiene is a problem everywhere. Most companies I worked at just throw out 'bad data' and file a bug. Occasionally, if the data is a secondary source, it will be recreated from a primary source (assuming the primary source is still available). In the particular case, your coworker would be stored by some identifier (like an SSN or similar) and their name would be stored as "aliases" and allow multiple names. I have two nicknames that I answer to, depending on when in my life you met me, and my family calls me by my proper name. Online, I go by several handles depending on whether I want the reader to be able to figure out my real name. I even used to work somewhere where I was called by this handle (withinboredom) more than my real name.
  - narag 4y ago
    Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (...) the hard part? Not for someone that works in consulting, or at least it wasn't. I remember that I had Access access to the production database that stored all the customers, present and pass. They wouldn't give me a password to that, but apparently they thought it was safe to enter the password without me looking so I could try some queries and test my last fixes. Not sure if it still works the same, but I did some dumb query, left the computer on and, next day morning, a temporal file was in my %TEMP% with a lot of data of millions of persons worldwide. Had I be so inclined, with an external hard disk I could have started my homegrown NSA project. Now think of this: how many times have you heard that the data of millions of customers were on sale after a data breach? Do you doubt that, let's say, China has every single person in the West on file? Our own governments have us legally or semi-legally (exchange) anyway.
hacker_9 4y ago
Wait until you hear about DNA
pornel 4y ago
Thanks to Snowden's leaks we know one of those is surveillance. From scanning every message of every person, it's going to expand to recognizing every face from every camera, and transcribing and analyzing every spoken word recorded.
kaba0 4y ago
Hold it in memory? Sure. But actually working with it besides copying it around and doing some operation on a “row-by-row” basis? Not really.
saalweachter 4y ago
A bacteria has order of 50 billion atoms. (Eukaryotic human cells, 100 trillion.) That's getting to the point you could store 20 bytes per atom in a terabyte. (The big bottleneck is that you need picosecond resolution simulation steps and to cover minutes to see a protein fold.)
adam_arthur 4y ago
The implication is that scalability problems in software will get easier and easier over time, and far fewer developers will be needed to maintain these systems. Which is already largely true today with the advent of serverless. Most maintenance work can center around application logic rather than scaling physical machines/maintaining versioning. It's clear that many modern applications would take an order of magnitude more people to run even just 20 years ago. That trend will only continue

moeny 4y ago

's/(human|desktop)/smartphone/g'

untech 4y ago
Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.
- cellis 4y ago
  Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.
  - toast0 4y ago
    Lat long and assuming at ground gets you most of the way there.
    - Dylan16807 4y ago
      "Add it all up, and the US has around 340 billion square feet of building stock[3]. This is about 12,200 square miles, or 0.00032% of US land area" Judging by that, you need a negligible increase in the number of locations you can represent to handle everywhere stable and off the ground someone could be. Much less than one bit per person. If you want to deal with people currently in airplanes then you could give them an extra couple bytes. It's less than a million people so it won't affect your total storage at all.
  - withinboredom 4y ago
    GPS altitude is notoriously bad via consumer devices. It's better to store altitude as less than 2 bytes with the range from being a few km below sea level to the maximum altitude consumer devices will report (3,000 km -- though maybe less because there is also a speed limit at which they will stop reporting too so you can't just buy a GPS device and build a guided missile).
  - ilyt 4y ago
    Most people don't live in caves or in the sky city, 2 is enough. And in places you care (multi-floor buildings) you aren't getting GPS signal inside anyway...
- Dylan16807 4y ago
  The planet has 2^47 square meters of surface, so more like 6 bytes. Plus you can group together people in the same area and/or sort positions as integers and store only the deltas between them, so you can probably get down to 2-3 bytes per person. And you can get dozens of models of smartphone with 16GB of RAM right now. So there might be a gap there but it's a very small gap. The phone of tomorrow will have the RAM. Edit: Thinking about it more, with 2^33 people and 2^47 locations the average delta would be 2^14, and it's pretty easy to guarantee that fits into 2 bytes per person. And with a more accurate world population count you'd free up at least a gigabyte for your phone to actually operate with.
- renonce 4y ago
  You can build that phone. Phones made an engineering decision to put more battery rather than more RAM, and it's just a matter of putting more LPDDR5 chips onto the circuit board.

Waterluvian 4y ago

In undergrad I had a bonus GIS lab assignment to complete the task using the prof’s instructions from the 80s.. maybe 90s? (a lot of FORTRAN) and then complete it again using modern GIS software. Such an eye opener. The thing that stuck out the most to me was how many hoops we jumped through to batch the job and commit the incremental results to disk because, bahahahha, fitting even 1% of it in RAM was out of the question.

narag 4y ago

TFA, to me, touches about something I've wondered about a very long time ago... It was a little more than ten years ago for me. I realized that a hard disk could store a database of every human alive, including basic information, links (family relations) and maybe a photo. I still don't know what the implications are. Maybe we don't want to know, but it's not really that difficult to think about.

doublepg23 4y ago
Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (ex: I have a co-worker who's name is incorrectly spelled three different ways in prod services) the hard part?
- withinboredom 4y ago
  Data hygiene is a problem everywhere. Most companies I worked at just throw out 'bad data' and file a bug. Occasionally, if the data is a secondary source, it will be recreated from a primary source (assuming the primary source is still available). In the particular case, your coworker would be stored by some identifier (like an SSN or similar) and their name would be stored as "aliases" and allow multiple names. I have two nicknames that I answer to, depending on when in my life you met me, and my family calls me by my proper name. Online, I go by several handles depending on whether I want the reader to be able to figure out my real name. I even used to work somewhere where I was called by this handle (withinboredom) more than my real name.
- narag 4y ago
  Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (...) the hard part? Not for someone that works in consulting, or at least it wasn't. I remember that I had Access access to the production database that stored all the customers, present and pass. They wouldn't give me a password to that, but apparently they thought it was safe to enter the password without me looking so I could try some queries and test my last fixes. Not sure if it still works the same, but I did some dumb query, left the computer on and, next day morning, a temporal file was in my %TEMP% with a lot of data of millions of persons worldwide. Had I be so inclined, with an external hard disk I could have started my homegrown NSA project. Now think of this: how many times have you heard that the data of millions of customers were on sale after a data breach? Do you doubt that, let's say, China has every single person in the West on file? Our own governments have us legally or semi-legally (exchange) anyway.

hacker_9 4y ago

Wait until you hear about DNA

pornel 4y ago

Thanks to Snowden's leaks we know one of those is surveillance. From scanning every message of every person, it's going to expand to recognizing every face from every camera, and transcribing and analyzing every spoken word recorded.

kaba0 4y ago

Hold it in memory? Sure. But actually working with it besides copying it around and doing some operation on a “row-by-row” basis? Not really.

saalweachter 4y ago

A bacteria has order of 50 billion atoms. (Eukaryotic human cells, 100 trillion.) That's getting to the point you could store 20 bytes per atom in a terabyte. (The big bottleneck is that you need picosecond resolution simulation steps and to cover minutes to see a protein fold.)

adam_arthur 4y ago

The implication is that scalability problems in software will get easier and easier over time, and far fewer developers will be needed to maintain these systems. Which is already largely true today with the advent of serverless. Most maintenance work can center around application logic rather than scaling physical machines/maintaining versioning. It's clear that many modern applications would take an order of magnitude more people to run even just 20 years ago. That trend will only continue