3 ms·

Given exponentially decreasing costs for storage, there's an argument for archiving everything of potential value and low liability risk. Ditto for store-width

by asah 4y ago

Given exponentially decreasing costs for storage, there's an argument for archiving everything of potential value and low liability risk. Ditto for store-width and testing recoverability.

crabbone 4y ago
Exponentially decreasing cost of storage is a myth. Storage is super expensive. It may be tempting to consider only some metrics (eg. the price of acquisition of the hardware), but the operational costs aren't getting much lower if at all.
- cmeacham98 4y ago
  At sufficient scale the per-GB (or per-TB/per-PB/etc) price of the storage dwarfs all other costs. A small business might have trouble archiving everything, but most large businesses should see and be able to take advantage of this difference.
  - crabbone 4y ago
    Does your company make Docker images? -- They start at somewhere around 1GB a piece. If, say, you use them for testing, you will have to create roughly an image per commit in your repository. 1K commits per projects -- that's about an output of a small (<5) team working for under a year. We are already at 1TB. Now, these images generate logs when run... Some of them may generate entire test datasets for testing... And we haven't even touched the data we want to store yet! Now, consider this: You need to store your Docker images somewhere, right? The registry. This registry needs backups every now and then, otherwise it will... well, one day you'll lose data, perhaps the entire registry full of images. So, you might think incremental snapshots is the answer, but then there's problem with incremental snapshots: what if one of them is damaged? So, you'd want independent snapshots... And then you want to store them in something that's resilient to failure and can have its parts replaced on demand, so, something like RAID5, so that also adds something like 30% to your storage. Another thing to consider: what if whatever you want to store isn't composed of independent images that can be reproduced on demand... what if, it's like a Web server with a session? Well, then you might need consistency groups that store your container's state together with the state of the database and whatever else that is using across, potentially, multiple computers... Oy-wey! Now we are looking into buying enterprise-level storage system from something like NetApp... We are probably spending six digit figures only on this single Web server with a database sharded into three servers on yearly basis... And we still don't have geographical redundancy for contingencies like datacenter being hit by a tsunami etc.
    - cmeacham98 4y ago
      If you only have a 5 person team of SDEs, then your company is within the "small businesses" that might not be able to store everything like I mentioned. Also, if you aren't deduping Docker images by layers, you're doing it very very wrong.
      - crabbone 4y ago
        Where did I say that the company has only 5 people?.. I gave it as a unit of measurement. Typically, hierarchies of s/w companies build on small-ish teams, in my experience, 5 would be the average team size. While there's usually some way to aggregate across teams (in terms of resources), largely, the resources a team needs are independent of other teams. In other words, if you have a team use X resources, then, roughly, two teams will use 2X resources. But resources per programmer don't make sense in the same way how man-hours don't (because you get fractional people). And, the truth is that even though large businesses have a potential to save during aggregation, developing this ability is yet another cost that they need to pay all the while the business is running. And the more efficient they want this aggregation to be, the more they need to invest into it. Which means, that large businesses are bound to be more wasteful than small businesses. So, to give a more concrete example of why there's tendency in resource waste to be of the form: for N teams, where resource use would be X per team, the cumulative resource use is (N + e)X, where e is small, but is a function of N, perhaps, something like log(N). Suppose your company's artifacts are PyPI-style Python packages (hopefully, Wheels). For a very small company, you might get away with just having a Git repo with your code, and don't even produce Wheel: just do code checkouts, add current source to PYTHONPATH, and you are good to go. Suppose your company now grows another team that needs to use the code written by the first team. Well, now they need to agree on their artifacts, the format, the location, so that two teams may independently create and access artifacts. The packages are relatively small and don't require much auditing of external dependencies though, so they simply pay for a small cloud instance to store their packages (eg. use GitHub releases or foundries or w/e it's called). Suppose your company grows to ten or so teams. Now you start getting conflicting requirements between individual artifacts of different teams, you can no longer use simple foundries provided by services like Github to store your artifacts because you discovered you also need to use patched versions of third-party dependencies, and you have increased audit requirements. You now buy services from something like Artifactory, where you store and mirror a whole lot of Python packages you don't develop. And, as you grow bigger, you'll realize that solutions like Artifactory also have their limits. At some huge scale, you'll get into business of running your own datacenters, and, perhaps, even your own power plants to power those datacenters. At each level on this growth path, you'll have to adjust your infrastructure to generate less waste compared to what would've happened if you tried to use the methodology you used at the previous level + the infrastructure necessary to bridge between multiple groups, however, each such step will never completely eliminate the price of aggregation, it will simply attempt to make it manageable. ---- In other words: a price of servicing of 1GB of storage for Google is lower than it would've been for an small startup, but they also need a lot more storage to manage per team than a small startup. This is why someone like Google would be in a good position to sell their services of managing storage for small shops, for example.
komali2 4y ago
Is storage that cheap? I can get a slightly used 18tb drive for, iirc, 250$ usd, but that's not actually useful on its own. I need two to actually have 18tb of space, so 500$. One 18tb drive isn't an 18tb drive, it's an 18tb ticking time bomb of catastrophic data loss. I think that cost is kinda ignored when people talk about storage costs. (Two isn't enough for some... Raidz2 with two drive failure survivability is what I most often hear recommended, because resilvering after one drive failure has high chance of killing another drive, at which point you're just fucked)
- pbalau 4y ago
  This doesn't include the costs for everything else needed, electricity, the other hardware etc etc. Plus, if you want 18tb, you actually need 3 drives: 2 for live use, one as a spare. If any of the live drives dies, you plop in the spare, rebuild the raid, move the data out and change the drives (since it might be impossible to acquire another 18tb drive).
  - asah 4y ago
    Guys - you're painting the bike shed here. Whether it's a disk or the tape cartridge, if storage costs halve then that means half of everything else (devices, machines, cages, machine rooms, data centers, etc) to store the same bytes, which is ultimately reflected in TCO.
    - horsawlarway 4y ago
      The problem and expense then shifts to recall. It's not enough to have the data on a drive somewhere - someone has to know which drive and how to access it. That problem gets harder and harder the more data you have. Especially as we move away from "everyone in the building can see the archives" style libraries to "Zero trust architectures" where you often can't see even a reasonable portion of the entire data store, only someone with the right credentials and access can. And this is a challenge right now... I've worked closely with clients that have data lakes that they are struggling to pull value out of because they can't find the data they need, or the data has so many security details (banking) involved that making the decision about giving access to someone requesting data takes WEEKS to resolve) Essentially - just keeping the data is insufficient. You need the ability to take advantage of it, and that usually costs much more than simple storage.
at_a_remove 4y ago
I mean, there is an argument for it, but the counterpoint is that you have to have an archivist and an index. Storing data for the Long Haul, such that it is findable, is not a simple matter in many cases. What I found deeply fascinating, as a technie working with/for librarians, is that these are people who have a solid and agreed-upon understanding of where to place physical objects which contain information, but when it came to storing data? They might as well be anyone off the street. Everyone had their idiosyncratic methodology, which wasn't even consistent over time.

by asah 4y ago

Given exponentially decreasing costs for storage, there's an argument for archiving everything of potential value and low liability risk. Ditto for store-width and testing recoverability.

crabbone 4y ago

Exponentially decreasing cost of storage is a myth. Storage is super expensive. It may be tempting to consider only some metrics (eg. the price of acquisition of the hardware), but the operational costs aren't getting much lower if at all.

cmeacham98 4y ago
At sufficient scale the per-GB (or per-TB/per-PB/etc) price of the storage dwarfs all other costs. A small business might have trouble archiving everything, but most large businesses should see and be able to take advantage of this difference.
- crabbone 4y ago
  Does your company make Docker images? -- They start at somewhere around 1GB a piece. If, say, you use them for testing, you will have to create roughly an image per commit in your repository. 1K commits per projects -- that's about an output of a small (<5) team working for under a year. We are already at 1TB. Now, these images generate logs when run... Some of them may generate entire test datasets for testing... And we haven't even touched the data we want to store yet! Now, consider this: You need to store your Docker images somewhere, right? The registry. This registry needs backups every now and then, otherwise it will... well, one day you'll lose data, perhaps the entire registry full of images. So, you might think incremental snapshots is the answer, but then there's problem with incremental snapshots: what if one of them is damaged? So, you'd want independent snapshots... And then you want to store them in something that's resilient to failure and can have its parts replaced on demand, so, something like RAID5, so that also adds something like 30% to your storage. Another thing to consider: what if whatever you want to store isn't composed of independent images that can be reproduced on demand... what if, it's like a Web server with a session? Well, then you might need consistency groups that store your container's state together with the state of the database and whatever else that is using across, potentially, multiple computers... Oy-wey! Now we are looking into buying enterprise-level storage system from something like NetApp... We are probably spending six digit figures only on this single Web server with a database sharded into three servers on yearly basis... And we still don't have geographical redundancy for contingencies like datacenter being hit by a tsunami etc.
  - cmeacham98 4y ago
    If you only have a 5 person team of SDEs, then your company is within the "small businesses" that might not be able to store everything like I mentioned. Also, if you aren't deduping Docker images by layers, you're doing it very very wrong.
    - crabbone 4y ago
      Where did I say that the company has only 5 people?.. I gave it as a unit of measurement. Typically, hierarchies of s/w companies build on small-ish teams, in my experience, 5 would be the average team size. While there's usually some way to aggregate across teams (in terms of resources), largely, the resources a team needs are independent of other teams. In other words, if you have a team use X resources, then, roughly, two teams will use 2X resources. But resources per programmer don't make sense in the same way how man-hours don't (because you get fractional people). And, the truth is that even though large businesses have a potential to save during aggregation, developing this ability is yet another cost that they need to pay all the while the business is running. And the more efficient they want this aggregation to be, the more they need to invest into it. Which means, that large businesses are bound to be more wasteful than small businesses. So, to give a more concrete example of why there's tendency in resource waste to be of the form: for N teams, where resource use would be X per team, the cumulative resource use is (N + e)X, where e is small, but is a function of N, perhaps, something like log(N). Suppose your company's artifacts are PyPI-style Python packages (hopefully, Wheels). For a very small company, you might get away with just having a Git repo with your code, and don't even produce Wheel: just do code checkouts, add current source to PYTHONPATH, and you are good to go. Suppose your company now grows another team that needs to use the code written by the first team. Well, now they need to agree on their artifacts, the format, the location, so that two teams may independently create and access artifacts. The packages are relatively small and don't require much auditing of external dependencies though, so they simply pay for a small cloud instance to store their packages (eg. use GitHub releases or foundries or w/e it's called). Suppose your company grows to ten or so teams. Now you start getting conflicting requirements between individual artifacts of different teams, you can no longer use simple foundries provided by services like Github to store your artifacts because you discovered you also need to use patched versions of third-party dependencies, and you have increased audit requirements. You now buy services from something like Artifactory, where you store and mirror a whole lot of Python packages you don't develop. And, as you grow bigger, you'll realize that solutions like Artifactory also have their limits. At some huge scale, you'll get into business of running your own datacenters, and, perhaps, even your own power plants to power those datacenters. At each level on this growth path, you'll have to adjust your infrastructure to generate less waste compared to what would've happened if you tried to use the methodology you used at the previous level + the infrastructure necessary to bridge between multiple groups, however, each such step will never completely eliminate the price of aggregation, it will simply attempt to make it manageable. ---- In other words: a price of servicing of 1GB of storage for Google is lower than it would've been for an small startup, but they also need a lot more storage to manage per team than a small startup. This is why someone like Google would be in a good position to sell their services of managing storage for small shops, for example.

komali2 4y ago

Is storage that cheap? I can get a slightly used 18tb drive for, iirc, 250$ usd, but that's not actually useful on its own. I need two to actually have 18tb of space, so 500$. One 18tb drive isn't an 18tb drive, it's an 18tb ticking time bomb of catastrophic data loss. I think that cost is kinda ignored when people talk about storage costs. (Two isn't enough for some... Raidz2 with two drive failure survivability is what I most often hear recommended, because resilvering after one drive failure has high chance of killing another drive, at which point you're just fucked)

pbalau 4y ago
This doesn't include the costs for everything else needed, electricity, the other hardware etc etc. Plus, if you want 18tb, you actually need 3 drives: 2 for live use, one as a spare. If any of the live drives dies, you plop in the spare, rebuild the raid, move the data out and change the drives (since it might be impossible to acquire another 18tb drive).
- asah 4y ago
  Guys - you're painting the bike shed here. Whether it's a disk or the tape cartridge, if storage costs halve then that means half of everything else (devices, machines, cages, machine rooms, data centers, etc) to store the same bytes, which is ultimately reflected in TCO.
  - horsawlarway 4y ago
    The problem and expense then shifts to recall. It's not enough to have the data on a drive somewhere - someone has to know which drive and how to access it. That problem gets harder and harder the more data you have. Especially as we move away from "everyone in the building can see the archives" style libraries to "Zero trust architectures" where you often can't see even a reasonable portion of the entire data store, only someone with the right credentials and access can. And this is a challenge right now... I've worked closely with clients that have data lakes that they are struggling to pull value out of because they can't find the data they need, or the data has so many security details (banking) involved that making the decision about giving access to someone requesting data takes WEEKS to resolve) Essentially - just keeping the data is insufficient. You need the ability to take advantage of it, and that usually costs much more than simple storage.

at_a_remove 4y ago

I mean, there is an argument for it, but the counterpoint is that you have to have an archivist and an index. Storing data for the Long Haul, such that it is findable, is not a simple matter in many cases. What I found deeply fascinating, as a technie working with/for librarians, is that these are people who have a solid and agreed-upon understanding of where to place physical objects which contain information, but when it came to storing data? They might as well be anyone off the street. Everyone had their idiosyncratic methodology, which wasn't even consistent over time.