7 ms·

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org.

by atoav 3y ago

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the disk 40 Gb. He could have easily just allocated 100 Gb and call it a day, this way we had to go to him 3 times to scale it up in 10 Gb steps each time, including a the stress of figuring out why things are failing (something that the admin should have seen on his monitoring system). Admins are my heros, but please if you allocate disk space, just take the biggest expected backup and multiply it by π. And if you need to be stingy with storage for some reason, be stingy, but decide when and where to be stingy — and at least keep an eye on the monitoring and upsize the storage before it is too late.

fillipvt 3y ago
Just wondering, why pi?
- panarky 3y ago
  I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet.
  - placesalt 3y ago
    > I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet. Good to know. FWIW i should also be avoided. It's tempting to use, since most programs use it as a counter, so it /should/ standardize the logfilesizes. But in practice it's very tricky to get a definitive disk space requirement with it.
londons_explore 3y ago
This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.
- alanbernstein 3y ago
  On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.
  - CamJN 3y ago
    Since apfs, “large” empty files can also take only the 4K or so for the fs entry on disk, gotta make sure it’s not a sparse file, and doing so is a bit tricky, better to fill it with junk that doesn’t compress well.
  - deleted 3y ago
    [deleted]
  - JackFr 3y ago
    And my mother sets all her clocks ahead 10 minutes so she's never late.
    - 0cf8612b2e1e 3y ago
      It is not an entirely unreasonable idea. If a system runs out of disk space, an unexpectedly large number of operations will fail. Which can make recovery more problematic than you would assume. If you can immediately recover some disk space and have breathing room, it could make the difference in restoring service.
      - SAI_Peregrinus 3y ago
        Keeping a bit of disk reserved for recovery is extremely common with copy-on-write filesystems like ZFS & BTRFS. Even deletion takes some extra space, so without a reservation it's effectively impossible to delete any files from a full disk.
tetha 3y ago
Be stingy in a smart way. I'd like a description why you need the storage, an estimation how much and a projection of growth over the next 6 - 12 month, though the latter one can wait for a month or three for something new. Beyond a certain scale we'd need a PO or a project to write up the cost too. And yes, we'll start bugging you again once it starts filling up to 80% or more, because we don't want your systems to fail :) But this way, I directly get an idea of the increase in storage we will need over the next year in order to plan the next hardware expansion.
pgt 3y ago
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it." – Upton Sinclair
stuff4ben 3y ago
Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave me a direct view in how hard it is to manage storage for developers. And as a developer using Artifactory, I don't want to worry about storage, I just want my builds and CI pipelines to complete.
- geerlingguy 3y ago
  This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't really use their dev or stage environments, or have a process for documenting and reproducing bugs).
  - b112 3y ago
    With silliness like that, you can bet it was the cost of their logging provider, as the feature they didn't like.
  - Veserv 3y ago
    90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.
    - floorballchamp 3y ago
      Plus, logs have enormous compression potential since their entropy is so low. That's the property exploited by every logging-as-a-service out there.
doubled112 3y ago
I agree with most of your points, but with system resources, sometimes it is simply that you can give them, but you can never take them back.
- JackFr 3y ago
  If cost accounting is done correctly, business units will give them back willingly.
xaerise 3y ago
I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.
- JackFr 3y ago
  Why wouldn't you assume that if I'm asking for it I have a good reason? Are you going to rearchitect my system for me?
  - syndicatedjelly 3y ago
    Accountability? Is that attitude any different from just asking for money and refusing to explain what it’s for?
  - bandyaboot 3y ago
    Wouldn’t you expected to have to provide some level of justification if you were, say, requesting a new development machine?
    - Gud 3y ago
      There is a difference between spending $2000+ on a new computer and $10, which is about what a terabyte costs. Probably just having the discussion itself would waste more resources than just giving the storage space.
    - mrguyorama 3y ago
      The meeting to bring in the relevant stakeholders and discuss that reasoning literally costs more than just fucking buying some cloud space.
      - FireBeyond 3y ago
        Had a boss that would swoop into “suspicious” meetings. “There’s 10 people here whose time I bill out at $250/hr each, spending an hour discussing whether to buy a $1,000 software license? Why?”
ooterness 3y ago
Nitpick: "b" is for bits, "B" is for bytes. Please don't casually mislabel units.
PeterStuer 3y ago
He was just training the users to go the shadowIT route with self bought non controlled storage, most probably in the form of personal usb drives, and/or departamental consumer NAS devices.
ilyt 3y ago
We tried. Devs when give more space just didn't bother to clean old crap and exact same thing happened but with few months of delay. But then we generally just ask how much do they need and bill project for it so that's generally also on them. But, unlike Toyota, we do have disk space alerts. Sometimes the problem is also entirely political, the management needs to tell client and charge them for more storage and won't accept the change till that happens. Meanwhile clock is ticking...
- atoav 3y ago
  In my case the data was a dummy database of more dummy users than are at our org (maybe 50% more). So once this goes to production it would likely get smaller. The problem in this case was twofold: - admin had the job to implement database backups. He didn't factor in backup size when allocating disk space. So this was wntirely his own fault. - the database does store certain transactions for a certain period, so this grew initially until it it setteled at a certain level. Because the margin of storage was slim, this caused the problem
  - ilyt 3y ago
    ...did you communicate any of that ? Because most of our cases where that happens was either lack of planning or lack of communicating that plan. By far most common one was "neither dev nor client knows the data volume in longer period". Which is fine as long as that's also communicated, but that's also often a problem. But I'm not denying of course that there are just shitty incompetent ops departments, just for the other customer we had dealing with ops department that had: * backup storage (which was some remote FTPS server IIRC) provisioned so slow the backup wouldn't copy within 24 hours. And the backup size was below TB. * weeks long delays with any resize.

by atoav 3y ago

fillipvt 3y ago

Just wondering, why pi?

panarky 3y ago
I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet.
- placesalt 3y ago
  > I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet. Good to know. FWIW i should also be avoided. It's tempting to use, since most programs use it as a counter, so it /should/ standardize the logfilesizes. But in practice it's very tricky to get a definitive disk space requirement with it.

londons_explore 3y ago

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

alanbernstein 3y ago
On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.
- CamJN 3y ago
  Since apfs, “large” empty files can also take only the 4K or so for the fs entry on disk, gotta make sure it’s not a sparse file, and doing so is a bit tricky, better to fill it with junk that doesn’t compress well.
- deleted 3y ago
  [deleted]
- JackFr 3y ago
  And my mother sets all her clocks ahead 10 minutes so she's never late.
  - 0cf8612b2e1e 3y ago
    It is not an entirely unreasonable idea. If a system runs out of disk space, an unexpectedly large number of operations will fail. Which can make recovery more problematic than you would assume. If you can immediately recover some disk space and have breathing room, it could make the difference in restoring service.
    - SAI_Peregrinus 3y ago
      Keeping a bit of disk reserved for recovery is extremely common with copy-on-write filesystems like ZFS & BTRFS. Even deletion takes some extra space, so without a reservation it's effectively impossible to delete any files from a full disk.

tetha 3y ago

Be stingy in a smart way. I'd like a description why you need the storage, an estimation how much and a projection of growth over the next 6 - 12 month, though the latter one can wait for a month or three for something new. Beyond a certain scale we'd need a PO or a project to write up the cost too. And yes, we'll start bugging you again once it starts filling up to 80% or more, because we don't want your systems to fail :) But this way, I directly get an idea of the increase in storage we will need over the next year in order to plan the next hardware expansion.

pgt 3y ago

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it." – Upton Sinclair

stuff4ben 3y ago

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave me a direct view in how hard it is to manage storage for developers. And as a developer using Artifactory, I don't want to worry about storage, I just want my builds and CI pipelines to complete.

geerlingguy 3y ago
This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't really use their dev or stage environments, or have a process for documenting and reproducing bugs).
- b112 3y ago
  With silliness like that, you can bet it was the cost of their logging provider, as the feature they didn't like.
- Veserv 3y ago
  90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.
  - floorballchamp 3y ago
    Plus, logs have enormous compression potential since their entropy is so low. That's the property exploited by every logging-as-a-service out there.

doubled112 3y ago

I agree with most of your points, but with system resources, sometimes it is simply that you can give them, but you can never take them back.

JackFr 3y ago
If cost accounting is done correctly, business units will give them back willingly.

xaerise 3y ago

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

JackFr 3y ago
Why wouldn't you assume that if I'm asking for it I have a good reason? Are you going to rearchitect my system for me?
- syndicatedjelly 3y ago
  Accountability? Is that attitude any different from just asking for money and refusing to explain what it’s for?
- bandyaboot 3y ago
  Wouldn’t you expected to have to provide some level of justification if you were, say, requesting a new development machine?
  - Gud 3y ago
    There is a difference between spending $2000+ on a new computer and $10, which is about what a terabyte costs. Probably just having the discussion itself would waste more resources than just giving the storage space.
  - mrguyorama 3y ago
    The meeting to bring in the relevant stakeholders and discuss that reasoning literally costs more than just fucking buying some cloud space.
    - FireBeyond 3y ago
      Had a boss that would swoop into “suspicious” meetings. “There’s 10 people here whose time I bill out at $250/hr each, spending an hour discussing whether to buy a $1,000 software license? Why?”

ooterness 3y ago

Nitpick: "b" is for bits, "B" is for bytes. Please don't casually mislabel units.

PeterStuer 3y ago

He was just training the users to go the shadowIT route with self bought non controlled storage, most probably in the form of personal usb drives, and/or departamental consumer NAS devices.

ilyt 3y ago

We tried. Devs when give more space just didn't bother to clean old crap and exact same thing happened but with few months of delay. But then we generally just ask how much do they need and bill project for it so that's generally also on them. But, unlike Toyota, we do have disk space alerts. Sometimes the problem is also entirely political, the management needs to tell client and charge them for more storage and won't accept the change till that happens. Meanwhile clock is ticking...

atoav 3y ago
In my case the data was a dummy database of more dummy users than are at our org (maybe 50% more). So once this goes to production it would likely get smaller. The problem in this case was twofold: - admin had the job to implement database backups. He didn't factor in backup size when allocating disk space. So this was wntirely his own fault. - the database does store certain transactions for a certain period, so this grew initially until it it setteled at a certain level. Because the margin of storage was slim, this caused the problem
- ilyt 3y ago
  ...did you communicate any of that ? Because most of our cases where that happens was either lack of planning or lack of communicating that plan. By far most common one was "neither dev nor client knows the data volume in longer period". Which is fine as long as that's also communicated, but that's also often a problem. But I'm not denying of course that there are just shitty incompetent ops departments, just for the other customer we had dealing with ops department that had: * backup storage (which was some remote FTPS server IIRC) provisioned so slow the backup wouldn't copy within 24 hours. And the backup size was below TB. * weeks long delays with any resize.