4 ms·

My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues whe

My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.

yabones 6y ago
100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are other places that should get their own mounts, like /home and /usr, but before you know it you've got an OpenBSD install on your hands with 15 partitions :)
- nijave 6y ago
  I place I used to work achieved something similar with lvm thin provisioning and split out something like /, /home, /var, /var/log and maybe a couple others. I think they also had something clever with lvm snapshots to rollback bad updates (snapshot system, upgrade, verify) so even if an update went rogue and deleted some important, unrelated files it could be undid
- ClumsyPilot 6y ago
  Sounds like a poor-mans quota system
- sedachv 6y ago
  OpenBSD default partition allocation is really well thought out: https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION At least put /tmp in its own partition as well. Multiple partitions can also save a lot of recovery time when there is a multiple bad sector event that corrupts an entire partition beyond recovery.
- kortex 6y ago
  /var gets its own disk on my machine! Yay academic machine learning docker containers. What's a few 30GB docker images between friends? So yeah /var gets to live with training data on SSD which gets cold swapped onto spinny platters as needed. /home is another, on the "main sdd". /usr isn't a bad choice, either.
  - sillysaurusx 6y ago
    Yay academic machine learning docker containers. What's a few 30GB docker images between friends? :) Hopefully docker will lose the fight for mindshare in the ML space.
    - kortex 6y ago
      What do you mean, "lose mindshare"? Docker is one of the absolute best things to happen to ML. My work is largely algo integration - I take academics' ML code and bundle it for API consumption. Before this, I was a bench chemist, so I kind of have a thing for reproducing experiments. The academic ML scene has a reproducibility crisis that makes other science reproducibility crises look like Phys 100 labs. These things depend on someone's conda env with versions pinned to nightly builds, inscrutable code written by a postdoc that has since left, datasets which have been mutated since the paper was published, on and on. Docker gives me a fighting chance to actually get reproducible results without going stark raving mad. Should these researchers have better software process to avoid this situation in the first place? Heck yes. But these are people doing things like `os.system("rm " + filepath)` and git committing entire models, they are just really green and most don't know any better yet and academia isn't really known for its mandatory Practical Software Engineering classes for CS majors.
      - sillysaurusx 6y ago
        Ah, you're a docker zealot. I misread the joke. You had a flash of self-awareness. Docker containers take massive amounts of disk space, and you need arcane knowledge just to use them. I'd rather focus on doing ML, not learning docker. In other words, "I'm getting to old for this." (33 is up there.) It bugged me in the game industry when some old programmer said "shaders are a young person's game," implying that they wouldn't even look into how they worked. Now that I am that older programmer, or getting there, I see what they mean. Docker arcana is a young person’s game. If Docker solved the reproducibility crisis, you might have a point. But it doesn't. Most of the crisis is the fact that (a) datasets are trapped behind institutions that won't make them available, (b) the models themselves aren't made available (OpenAI), and (c) the code itself isn't available (also OpenAI). Those three things are the main problem. Forcing everyone to use a 30GB docker container just to do basic ML isn't going to do anything but waste time and turn newbies away from ML. You're a fine debator though. It was an enjoyable read; have an upvote.
Cacti 6y ago
Do people not create partitions any more? I thought this was sysadmin 101 for, like, forever. Databases, web servers, etc. should never ever fill up the entire disk. Separate partitions for boot, swap, /, /home, /var, and /tmp are the minimum common sense partitions.

yabones 6y ago

100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are other places that should get their own mounts, like /home and /usr, but before you know it you've got an OpenBSD install on your hands with 15 partitions :)

nijave 6y ago
I place I used to work achieved something similar with lvm thin provisioning and split out something like /, /home, /var, /var/log and maybe a couple others. I think they also had something clever with lvm snapshots to rollback bad updates (snapshot system, upgrade, verify) so even if an update went rogue and deleted some important, unrelated files it could be undid
ClumsyPilot 6y ago
Sounds like a poor-mans quota system
sedachv 6y ago
OpenBSD default partition allocation is really well thought out: https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION At least put /tmp in its own partition as well. Multiple partitions can also save a lot of recovery time when there is a multiple bad sector event that corrupts an entire partition beyond recovery.
kortex 6y ago
/var gets its own disk on my machine! Yay academic machine learning docker containers. What's a few 30GB docker images between friends? So yeah /var gets to live with training data on SSD which gets cold swapped onto spinny platters as needed. /home is another, on the "main sdd". /usr isn't a bad choice, either.
- sillysaurusx 6y ago
  Yay academic machine learning docker containers. What's a few 30GB docker images between friends? :) Hopefully docker will lose the fight for mindshare in the ML space.
  - kortex 6y ago
    What do you mean, "lose mindshare"? Docker is one of the absolute best things to happen to ML. My work is largely algo integration - I take academics' ML code and bundle it for API consumption. Before this, I was a bench chemist, so I kind of have a thing for reproducing experiments. The academic ML scene has a reproducibility crisis that makes other science reproducibility crises look like Phys 100 labs. These things depend on someone's conda env with versions pinned to nightly builds, inscrutable code written by a postdoc that has since left, datasets which have been mutated since the paper was published, on and on. Docker gives me a fighting chance to actually get reproducible results without going stark raving mad. Should these researchers have better software process to avoid this situation in the first place? Heck yes. But these are people doing things like `os.system("rm " + filepath)` and git committing entire models, they are just really green and most don't know any better yet and academia isn't really known for its mandatory Practical Software Engineering classes for CS majors.
    - sillysaurusx 6y ago
      Ah, you're a docker zealot. I misread the joke. You had a flash of self-awareness. Docker containers take massive amounts of disk space, and you need arcane knowledge just to use them. I'd rather focus on doing ML, not learning docker. In other words, "I'm getting to old for this." (33 is up there.) It bugged me in the game industry when some old programmer said "shaders are a young person's game," implying that they wouldn't even look into how they worked. Now that I am that older programmer, or getting there, I see what they mean. Docker arcana is a young person’s game. If Docker solved the reproducibility crisis, you might have a point. But it doesn't. Most of the crisis is the fact that (a) datasets are trapped behind institutions that won't make them available, (b) the models themselves aren't made available (OpenAI), and (c) the code itself isn't available (also OpenAI). Those three things are the main problem. Forcing everyone to use a 30GB docker container just to do basic ML isn't going to do anything but waste time and turn newbies away from ML. You're a fine debator though. It was an enjoyable read; have an upvote.

Cacti 6y ago

Do people not create partitions any more? I thought this was sysadmin 101 for, like, forever. Databases, web servers, etc. should never ever fill up the entire disk. Separate partitions for boot, swap, /, /home, /var, and /tmp are the minimum common sense partitions.