12 ms·

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

zimpenfish 3y ago
Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.
- lostlogin 3y ago
  It’s year ago now, but my father bought a second hand hard drive and found it full of banking data with customer names and all.
  - deleted 3y ago
    [deleted]
  - b112 3y ago
    Um, does he still have that data? Asking for a friend!
- hnbear 3y ago
  As an ex-bank employee one time I needed space and was told I couldn’t get it because all the filers were full. When would we get new filers? Dunno, because the datacenter was also full.
willcipriano 3y ago
There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.
- ReactiveJelly 3y ago
  Steelman possibility: There's 20 people in 10 departments giving him conflicting non-written requests for disk space, his own request for new disks is held up somewhere, and his boss keeps telling him "Just do what you're assigned" which doesn't include preventing a full disk condition.
- sjdfbnionio 3y ago
  Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering practices, not by limiting resources.
  - wil421 3y ago
    Are you talking consumer prices, enterprise redundant SSDs in a Data Center, or enterprise cloud storage? In my experience the latter two could add another digit to your price.
    - b112 3y ago
      And all of that needs to be backed up, secured, stored, tested for validity regularly, and so on. More disk space isn't one drive, but redundancy too And backups. And even network bandwidth. This thread sort of highlights the problems. DEVs don't get it. Nor does management.
  - greedo 3y ago
    This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.
    - willcipriano 3y ago
      I'd bet the salary of the guy who is making this a pain could buy enough storage that they don't have to worry about it anymore.
atoav 3y ago
I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the disk 40 Gb. He could have easily just allocated 100 Gb and call it a day, this way we had to go to him 3 times to scale it up in 10 Gb steps each time, including a the stress of figuring out why things are failing (something that the admin should have seen on his monitoring system). Admins are my heros, but please if you allocate disk space, just take the biggest expected backup and multiply it by π. And if you need to be stingy with storage for some reason, be stingy, but decide when and where to be stingy — and at least keep an eye on the monitoring and upsize the storage before it is too late.
- fillipvt 3y ago
  Just wondering, why pi?
  - panarky 3y ago
    I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet.
    - placesalt 3y ago
      > I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet. Good to know. FWIW i should also be avoided. It's tempting to use, since most programs use it as a counter, so it /should/ standardize the logfilesizes. But in practice it's very tricky to get a definitive disk space requirement with it.
- londons_explore 3y ago
  This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.
kenoh 3y ago
I'm just here to admire the fact that a Google search for "Toyototian" results this 1 hour old comment.
- xeromal 3y ago
  I was considering what to call a Toyota employee and thought Toyototian sounded right. Accidentally added an additional to* but oh well. It's pretty incredible how fast Google adds stuff to their index this quickly!
  - ddoolin 3y ago
    I work for Toyota but I've never pondered what our demonym is; I think Toyototian (Toyotian?) has something going for it though. :)
    - xeromal 3y ago
      Yeah, I think toyotian works. Haha
sct202 3y ago
We had an on-prem server at a new factory I was working at and less than a year in the whole thing shut down because we ran out of disk space for logs so none of the automation could successfully report back that they completed tasks to move to the next task. It turns out it was apparently terabytes of unallocated disk space on the server, so it was a quick fix but just felt so avoidable.
- pnutjam 3y ago
  In a past job I took over for an enterprise linux setup where all the log volumes where way too small. It was dozens of alerts every week to just grow the log volume or delete some stuff. It took me a couple years to get everything resized properly and logs rotating correctly. This was around 700 servers with all the change requirements of a heavily audited environment.
jl6 3y ago
Often this is intentional administrative backpressure. Finance tells IT to minimize costs. IT knows that the majority of users are poor at housekeeping, so tight control over storage allocation provides an incentive to delete-first-expand-later. Unfortunately, this strategy frustrates users who have a genuine need to expand storage and who are unable to efficiently obtain an exception to the process.
- sam_bristow 3y ago
  Being slow to provide resources when I need them also means that I'm less likely to relenquish them when I'm done.
  - acdha 3y ago
    Also more likely to request far more than you need so you don’t have to deal with them as frequently. Everywhere I’ve seen that strategy tried had massive overspending and outages due to it. What’s worked best is cloud style usage billing because that aligns the incentives with the people making the decisions, but it’s really non-trivial to get the accurate full cost.
Bluecobra 3y ago
I can say the same thing but as a sysadmin working at a place where it was pulling teeth to get money to buy additional storage. Upper management was so daft/cheap and they can't see past six months so they decide to buy the cheapest enterprise solution with the least amount of upgrade capacity.
- mobutu 3y ago
  My experience as an admin was quite similar. Developers just couldn't seem to grasp the need for someone to foot the bill for their constant requests for 2 TB LUNs. I'm just glad I'm done dealing with that shit.
lovehashbrowns 3y ago
I work for a retailer where the service I’m responsible for is used by every cash register around the world for certain operations. When I came in, the RDS DB for this service had 60GB allocated to it, had literally just run out of space and caused an outage. The last team just gave it an additional 20GB. A month later, I was put in charge of it and it was already 5GB away from running out of space again. I put an end to that and gave it 250GB. The cost is minimal compared to a store not being able to open due to an outage. The instances for the service itself had 20GB of EBS allocated to them. Luckily they don’t need much local storage. But that’s typical here. There’s a Jenkins instance that is even more of a pain. I’m not responsible for it but every week or two one of the worker nodes runs out of space because they’re given 8GB of storage space. I’m just watching the disaster unfold over the course of a year and a half as I’m constantly telling that team to just up the storage space on the worker nodes instead of constantly having to fiddle with cron jobs. It’s not even an expense thing. They just… don’t want to increase the storage space. It drives me insane.
- floorballchamp 3y ago
  I'd guess the worry is that once you increase the storage, you never decrease it again. Ever. It's a one-way street. So, once everything is 5x over-provisioned, then the services tend to fill that space anyway (cause why not be wasteful if it doesn't cost anything) and a year later you are in the same seat again. I'm not saying this is real, but the worry certainly is.
  - lovehashbrowns 3y ago
    That's certainly real and something to consider when provisioning systems. I'm fully on board with that. The problem is when the cost of the cost-savings solution vastly outweighs the cost of over-provisioning infrastructure. Like this Jenkins issue bubbling up ~2-4 times a month vs just giving the worker nodes more storage space. There's been times where it happened during the night and people got paged. Or comparing the cost of one store not being able to open on time because the RDS database's space ran out. VPs and directors start yelling and there's suddenly like 20+ people involved in figuring out why this one store didn't open on time. What's the cost of that compared to just giving the DB 250GB of space so this never comes up again? But you are also 100% correct and I've seen that happen here, too. There's some instances I'm responsible for that were using EFS for their local storage. Costing thousands of dollars every month for absolutely no reason. I switched those to reasonably-sized EBS volumes and that alone was half of my annual savings goal. I was completely flabbergasted seeing these instances using EFS while others were stuck on 8GB EBS volumes. Backups on the EFS drives had ballooned to the many TBs. And the backups were worthless! Instances themselves are ephemeral. They use S3 for long-term storage & metadata is on a database. Those are the things that should be backed up & their cost compared to EFS is minuscule.

zimpenfish 3y ago

Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.

lostlogin 3y ago
It’s year ago now, but my father bought a second hand hard drive and found it full of banking data with customer names and all.
- deleted 3y ago
  [deleted]
- b112 3y ago
  Um, does he still have that data? Asking for a friend!
hnbear 3y ago
As an ex-bank employee one time I needed space and was told I couldn’t get it because all the filers were full. When would we get new filers? Dunno, because the datacenter was also full.

willcipriano 3y ago

There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.

ReactiveJelly 3y ago
Steelman possibility: There's 20 people in 10 departments giving him conflicting non-written requests for disk space, his own request for new disks is held up somewhere, and his boss keeps telling him "Just do what you're assigned" which doesn't include preventing a full disk condition.
sjdfbnionio 3y ago
Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering practices, not by limiting resources.
- wil421 3y ago
  Are you talking consumer prices, enterprise redundant SSDs in a Data Center, or enterprise cloud storage? In my experience the latter two could add another digit to your price.
  - b112 3y ago
    And all of that needs to be backed up, secured, stored, tested for validity regularly, and so on. More disk space isn't one drive, but redundancy too And backups. And even network bandwidth. This thread sort of highlights the problems. DEVs don't get it. Nor does management.
- greedo 3y ago
  This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.
  - willcipriano 3y ago
    I'd bet the salary of the guy who is making this a pain could buy enough storage that they don't have to worry about it anymore.

atoav 3y ago

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the disk 40 Gb. He could have easily just allocated 100 Gb and call it a day, this way we had to go to him 3 times to scale it up in 10 Gb steps each time, including a the stress of figuring out why things are failing (something that the admin should have seen on his monitoring system). Admins are my heros, but please if you allocate disk space, just take the biggest expected backup and multiply it by π. And if you need to be stingy with storage for some reason, be stingy, but decide when and where to be stingy — and at least keep an eye on the monitoring and upsize the storage before it is too late.

fillipvt 3y ago
Just wondering, why pi?
- panarky 3y ago
  I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet.
  - placesalt 3y ago
    > I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet. Good to know. FWIW i should also be avoided. It's tempting to use, since most programs use it as a counter, so it /should/ standardize the logfilesizes. But in practice it's very tricky to get a definitive disk space requirement with it.
londons_explore 3y ago
This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

kenoh 3y ago

I'm just here to admire the fact that a Google search for "Toyototian" results this 1 hour old comment.

xeromal 3y ago
I was considering what to call a Toyota employee and thought Toyototian sounded right. Accidentally added an additional to* but oh well. It's pretty incredible how fast Google adds stuff to their index this quickly!
- ddoolin 3y ago
  I work for Toyota but I've never pondered what our demonym is; I think Toyototian (Toyotian?) has something going for it though. :)
  - xeromal 3y ago
    Yeah, I think toyotian works. Haha

sct202 3y ago

We had an on-prem server at a new factory I was working at and less than a year in the whole thing shut down because we ran out of disk space for logs so none of the automation could successfully report back that they completed tasks to move to the next task. It turns out it was apparently terabytes of unallocated disk space on the server, so it was a quick fix but just felt so avoidable.

pnutjam 3y ago
In a past job I took over for an enterprise linux setup where all the log volumes where way too small. It was dozens of alerts every week to just grow the log volume or delete some stuff. It took me a couple years to get everything resized properly and logs rotating correctly. This was around 700 servers with all the change requirements of a heavily audited environment.

jl6 3y ago

Often this is intentional administrative backpressure. Finance tells IT to minimize costs. IT knows that the majority of users are poor at housekeeping, so tight control over storage allocation provides an incentive to delete-first-expand-later. Unfortunately, this strategy frustrates users who have a genuine need to expand storage and who are unable to efficiently obtain an exception to the process.

sam_bristow 3y ago
Being slow to provide resources when I need them also means that I'm less likely to relenquish them when I'm done.
- acdha 3y ago
  Also more likely to request far more than you need so you don’t have to deal with them as frequently. Everywhere I’ve seen that strategy tried had massive overspending and outages due to it. What’s worked best is cloud style usage billing because that aligns the incentives with the people making the decisions, but it’s really non-trivial to get the accurate full cost.

Bluecobra 3y ago

I can say the same thing but as a sysadmin working at a place where it was pulling teeth to get money to buy additional storage. Upper management was so daft/cheap and they can't see past six months so they decide to buy the cheapest enterprise solution with the least amount of upgrade capacity.

mobutu 3y ago
My experience as an admin was quite similar. Developers just couldn't seem to grasp the need for someone to foot the bill for their constant requests for 2 TB LUNs. I'm just glad I'm done dealing with that shit.

lovehashbrowns 3y ago

I work for a retailer where the service I’m responsible for is used by every cash register around the world for certain operations. When I came in, the RDS DB for this service had 60GB allocated to it, had literally just run out of space and caused an outage. The last team just gave it an additional 20GB. A month later, I was put in charge of it and it was already 5GB away from running out of space again. I put an end to that and gave it 250GB. The cost is minimal compared to a store not being able to open due to an outage. The instances for the service itself had 20GB of EBS allocated to them. Luckily they don’t need much local storage. But that’s typical here. There’s a Jenkins instance that is even more of a pain. I’m not responsible for it but every week or two one of the worker nodes runs out of space because they’re given 8GB of storage space. I’m just watching the disaster unfold over the course of a year and a half as I’m constantly telling that team to just up the storage space on the worker nodes instead of constantly having to fiddle with cron jobs. It’s not even an expense thing. They just… don’t want to increase the storage space. It drives me insane.

floorballchamp 3y ago
I'd guess the worry is that once you increase the storage, you never decrease it again. Ever. It's a one-way street. So, once everything is 5x over-provisioned, then the services tend to fill that space anyway (cause why not be wasteful if it doesn't cost anything) and a year later you are in the same seat again. I'm not saying this is real, but the worry certainly is.
- lovehashbrowns 3y ago
  That's certainly real and something to consider when provisioning systems. I'm fully on board with that. The problem is when the cost of the cost-savings solution vastly outweighs the cost of over-provisioning infrastructure. Like this Jenkins issue bubbling up ~2-4 times a month vs just giving the worker nodes more storage space. There's been times where it happened during the night and people got paged. Or comparing the cost of one store not being able to open on time because the RDS database's space ran out. VPs and directors start yelling and there's suddenly like 20+ people involved in figuring out why this one store didn't open on time. What's the cost of that compared to just giving the DB 250GB of space so this never comes up again? But you are also 100% correct and I've seen that happen here, too. There's some instances I'm responsible for that were using EFS for their local storage. Costing thousands of dollars every month for absolutely no reason. I switched those to reasonably-sized EBS volumes and that alone was half of my annual savings goal. I was completely flabbergasted seeing these instances using EFS while others were stuck on 8GB EBS volumes. Backups on the EFS drives had ballooned to the many TBs. And the backups were worthless! Instances themselves are ephemeral. They use S3 for long-term storage & metadata is on a database. Those are the things that should be backed up & their cost compared to EFS is minuscule.