4 ms·

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when

by diego 6y ago

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

mtone 6y ago
Assuming we're talking about VMs (2021 etc.), for a SME is there any downside to giving 2TB of space to your discs and let dynamic allocation do the work? Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage. Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and these days is probably just voluntary management overhead. RAM is another story.
- qw3rty01 6y ago
  One VM using excessively more disk space than it's supposed to can potentially cause data corruption in all the other VMs on that system. For just spinning VMs up and down for testing, you probably won't run into that issue, but on a production system, it could potentially cause some massive downtime
  - cure 6y ago
    Virtual machine disk space (e.g. Xen, Linode, AWS EC2, or similar) does not work this way. Each VM gets a dedicated amount of disk space allocated to it, they don't all share a pool of free space.
    - jodrellblank 6y ago
      Yes they do with the "dyanmic allocation" the parent comment mentions; VMware datastore has 1TB total, you put VMs in with dynamically expanding disks they are sharing the same 1TB of free space and will fill it if they all want their max space at the same time and you've overprovisioned their max space. And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed size disks. Even then, snapshots will grow forever and fill the space, and then you hope you have a "spacer.img" file you can delete from the datastore, because you can't remove snapshots when the disk is full and you're stuck. It's the same problem, at a lower level.
      - cure 6y ago
        I see, a VMware feature, thanks for clarifying. I suppose it's a nice idea in theory, but you'd have to be crazy to use that in production, or for any workload that you care about. It would just be a ticking time bomb.
        tremon 6y ago
        Hyper-V can do that too, and so can you under Linux. It's called thinly-allocated disks, sparse files, or the dm-thin device mapper target. Professional SANs also allow you to overallocate the total size of the iSCSI volumes offered. Yes, I've seen that time bomb go off on multiple occasions. Never on my watch though.
- jodrellblank 6y ago
  If you are using less space than the underlying datastore, there's no benefit to dynamic allocation, you may as well give the servers larger fixed disks. If you are thinking that one server might need more than the fixed size for a sudden growth, then you need to be monitoring to deal with that because that will run out of your space. If you are overprovisioning the datastore, you have the same problem at a level lower, and need to be monitoring that and alerting for that instead (as well). > "just buy ample cheap storage"; "That was mostly a rust performance thing and these days is probably just" In the UK a 6TB enterprise rust disk is £150 and a 2TB enterprise SSD is £300, it's 6x the price to SSD everything, and take 3x more drive bays so add more for that. And you can never "just" buy more storage than you ever need - apart from the obvious "when you bought it, you thought you were buying enough, because if you thought you needed more you would have bought more", so that amounts to saying "just know the future better", but it can't happen because Parkinson's Law ("work expands so as to fill the time available for its completion") applies to storage, the more there is available, the more things appear to fill it up. Room for a test restore of the backups in that space. Room for a clone of the database to do some testing. Room for a trial of a new product. Room for a copy of all the installers and packagers for convenience. Room for a massive central logging server there. What do you mean it's full?
rozap 6y ago
Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignore it.
- goatinaboat 6y ago
  It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station In scuba diving there used to be "J-valves". When you had 50 bar left in the tank they would cut out. Then you would pull to reenable your air and return to the surface. Unsurprisingly they are no longer popular.
- copperfoil 6y ago
  > The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making finding files to delete easier? Then what happens after that? If you delete the file and realize there's nothing else to delete, you'd have to solve the problem the same way if you didn't use this method.
  - vineyardmike 6y ago
    >you'd have to solve the problem the same way if you didn't use this method. What if the solution required some amount of free space? (eg. installing a package or swap)
    - tremon 6y ago
      ramdisk? It wouldn't be the first time I'd extract a .deb to tmpfs to resolve a temporary issue. Don't think I've ever encountered a critical issue where "add more swap" would be a serious disaster recovery solution. I've certainly seen situations where swap was nearing 100% full, and although I would have minutes off wall-clock time to formulate a strategy, those minutes have never allowed me to input more than a handful of characters or so.
copperfoil 6y ago
Linux servers aren't like mass consumer products. It's assumed users know what they're doing and can build and configure what they need on top of it. > This is 2021 and the technology is from the 90s I don't see how this is a valid point. Is integrated circuit technology outdated because it was developed in the 60s?
lostcolony 6y ago
You car also doesn't drop from "alarm" to "empty" in thirty seconds. A HD on a VM with a bad behaving process can.
- tremon 6y ago
  That's a quality problem. Your car can absolutely drop from "alarm" to "empty" in 30 seconds if there's a leak in the tank. We just don't build fuel tanks that don't spontaneously develop leaks, partly because the manufacturer can be held liable.
dmingod666 6y ago
Exactly, it's 1990s 'cool' - the time it took him to write the blog, he could have written a script that would send him updates on all his devices...
- ineedasername 6y ago
  There's no reason not to have multiple fail-safes. Receiving the alert on a device at 3am would still mean he could free up 8gb immediately and have breathing room to solve the problem. And remember this is for a single admin. Asking such a person to be on call 24-7 all year, vacations, holidays, weekends... Having a quick way to get breathing room can significantly reduce the stress & cognitive load of worrying about such things in your off-time.
  - dmingod666 6y ago
    He didn't mention he has alerts. Sure if alerts is your first line of defense, this is a nice thing to do.
Yizahi 6y ago
Alerting is also a hack really. In 2021 the operating SYSTEM should work as a system - complexly managing it's resources and make intelligent decisions. Ideally OS should dynamically reserve as much resources as needed on it's own.

by diego 6y ago

mtone 6y ago

Assuming we're talking about VMs (2021 etc.), for a SME is there any downside to giving 2TB of space to your discs and let dynamic allocation do the work? Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage. Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and these days is probably just voluntary management overhead. RAM is another story.

qw3rty01 6y ago
One VM using excessively more disk space than it's supposed to can potentially cause data corruption in all the other VMs on that system. For just spinning VMs up and down for testing, you probably won't run into that issue, but on a production system, it could potentially cause some massive downtime
- cure 6y ago
  Virtual machine disk space (e.g. Xen, Linode, AWS EC2, or similar) does not work this way. Each VM gets a dedicated amount of disk space allocated to it, they don't all share a pool of free space.
  - jodrellblank 6y ago
    Yes they do with the "dyanmic allocation" the parent comment mentions; VMware datastore has 1TB total, you put VMs in with dynamically expanding disks they are sharing the same 1TB of free space and will fill it if they all want their max space at the same time and you've overprovisioned their max space. And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed size disks. Even then, snapshots will grow forever and fill the space, and then you hope you have a "spacer.img" file you can delete from the datastore, because you can't remove snapshots when the disk is full and you're stuck. It's the same problem, at a lower level.
    - cure 6y ago
      I see, a VMware feature, thanks for clarifying. I suppose it's a nice idea in theory, but you'd have to be crazy to use that in production, or for any workload that you care about. It would just be a ticking time bomb.
      - tremon 6y ago
        Hyper-V can do that too, and so can you under Linux. It's called thinly-allocated disks, sparse files, or the dm-thin device mapper target. Professional SANs also allow you to overallocate the total size of the iSCSI volumes offered. Yes, I've seen that time bomb go off on multiple occasions. Never on my watch though.
jodrellblank 6y ago
If you are using less space than the underlying datastore, there's no benefit to dynamic allocation, you may as well give the servers larger fixed disks. If you are thinking that one server might need more than the fixed size for a sudden growth, then you need to be monitoring to deal with that because that will run out of your space. If you are overprovisioning the datastore, you have the same problem at a level lower, and need to be monitoring that and alerting for that instead (as well). > "just buy ample cheap storage"; "That was mostly a rust performance thing and these days is probably just" In the UK a 6TB enterprise rust disk is £150 and a 2TB enterprise SSD is £300, it's 6x the price to SSD everything, and take 3x more drive bays so add more for that. And you can never "just" buy more storage than you ever need - apart from the obvious "when you bought it, you thought you were buying enough, because if you thought you needed more you would have bought more", so that amounts to saying "just know the future better", but it can't happen because Parkinson's Law ("work expands so as to fill the time available for its completion") applies to storage, the more there is available, the more things appear to fill it up. Room for a test restore of the backups in that space. Room for a clone of the database to do some testing. Room for a trial of a new product. Room for a copy of all the installers and packagers for convenience. Room for a massive central logging server there. What do you mean it's full?

rozap 6y ago

Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignore it.

goatinaboat 6y ago
It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station In scuba diving there used to be "J-valves". When you had 50 bar left in the tank they would cut out. Then you would pull to reenable your air and return to the surface. Unsurprisingly they are no longer popular.
copperfoil 6y ago
> The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making finding files to delete easier? Then what happens after that? If you delete the file and realize there's nothing else to delete, you'd have to solve the problem the same way if you didn't use this method.
- vineyardmike 6y ago
  >you'd have to solve the problem the same way if you didn't use this method. What if the solution required some amount of free space? (eg. installing a package or swap)
  - tremon 6y ago
    ramdisk? It wouldn't be the first time I'd extract a .deb to tmpfs to resolve a temporary issue. Don't think I've ever encountered a critical issue where "add more swap" would be a serious disaster recovery solution. I've certainly seen situations where swap was nearing 100% full, and although I would have minutes off wall-clock time to formulate a strategy, those minutes have never allowed me to input more than a handful of characters or so.

copperfoil 6y ago

Linux servers aren't like mass consumer products. It's assumed users know what they're doing and can build and configure what they need on top of it. > This is 2021 and the technology is from the 90s I don't see how this is a valid point. Is integrated circuit technology outdated because it was developed in the 60s?

lostcolony 6y ago

You car also doesn't drop from "alarm" to "empty" in thirty seconds. A HD on a VM with a bad behaving process can.

tremon 6y ago
That's a quality problem. Your car can absolutely drop from "alarm" to "empty" in 30 seconds if there's a leak in the tank. We just don't build fuel tanks that don't spontaneously develop leaks, partly because the manufacturer can be held liable.

dmingod666 6y ago

Exactly, it's 1990s 'cool' - the time it took him to write the blog, he could have written a script that would send him updates on all his devices...

ineedasername 6y ago
There's no reason not to have multiple fail-safes. Receiving the alert on a device at 3am would still mean he could free up 8gb immediately and have breathing room to solve the problem. And remember this is for a single admin. Asking such a person to be on call 24-7 all year, vacations, holidays, weekends... Having a quick way to get breathing room can significantly reduce the stress & cognitive load of worrying about such things in your off-time.
- dmingod666 6y ago
  He didn't mention he has alerts. Sure if alerts is your first line of defense, this is a nice thing to do.

Yizahi 6y ago

Alerting is also a hack really. In 2021 the operating SYSTEM should work as a system - complexly managing it's resources and make intelligent decisions. Ideally OS should dynamically reserve as much resources as needed on it's own.