5 ms·

I wonder if anyone controls the physical layout of bytes on the disk, at least for things like installing large software packages on an HDD, or a major release

by listic 11y ago

I wonder if anyone controls the physical layout of bytes on the disk, at least for things like installing large software packages on an HDD, or a major release of an operating system on a DVD. It doesn't look likely: every time I install Ubuntu, I feel the process could be made much faster.

gambiting 11y ago
We do for PS4 disc images. Basically you want to lay out the data on the disc in such a way that the game can still be played while the system copies everything to the HDD. There are pages upon pages of manuals in PS4 SDK how to do it efficiently.
cnvogel 11y ago
At least for Windows there's now a way to install the OS so that it only occupies one continuous large filesystem image with the compressed install files, and only stores the changes to this frozen filesystem images in the "traditional" way. It's called "wimboot". http://labalec.fr/erwan/?p=1078 http://labalec.fr/erwan/?p=1078 That way installing the OS mainy consists of copying this huge image file, hence the "physical layout of bytes on the disk" will mostly be fixed. I'd guess that this could also be replicated in Linux, but personally I don't know if this is being done. I usually "debootstrap" or "pacstrap" my installs from a bootable USB stick :-).
- caf 11y ago
  Yes, you could replicate that in Linux using overlayfs, with a readonly mount of the static image as the the lower layer and a writeable upper layer using an ordinary filesystem. It would be interesting to try it.
- shubb 11y ago
  Sure. On one project we built a custom Linux distribution by mounting a file as a loop back file system and copying the right files in. We installed a grub boot sector at the start. Then we copied it as a file into a simple live usb. The live usb also containing a script that used dd to copy the data from that file over the start of the first disk of a system you plugged it into. Nice quick install usb, minimal work. We were installing into standard hardware so it was pretty good. But if your hardware varies this approach isn't great. We could also have created a fresh filesystem on the device and untarred into it but we wanted to keep se Linux attributes.
  - cnvogel 11y ago
    dd'ing to the target device, and then growing the fs to the full disk extent seems to be a splendid idea. Especially as Linux tends to replace quite a lot of code over the usual curse of updates... For a cluster system I had built a netbootable system for reinstall that would restore a dump of the reference system's fs on a node and adjust hostnames, ssh host keys etc... Also was quite fast at that time..
- derefr 11y ago
  That's effectively how CoreOS works.
jeremiep 11y ago
Ubuntu targets a LOT of different hardware, it would be much harder to optimize the layout on their ISO image than it is to optimize for a fixed hardware platform like consoles.
Cthulhu_ 11y ago
Do you still use a CD / DVD to install your OS? That's probably the problem right there; CD's/DVD's are relatively slow. Go for an install off a flash disk (where physical layout shouldn't matter), and / or a web install where only the relevant software compiled for your hardware is downloaded.
- yellowapple 11y ago
  CDs/DVDs don't have to be slow (or at least not as slow as they typically are). Most speed problems are the same speed problems one encounters with spinny / non-SSD hard drives: namely, that data isn't sequential, requiring a lot of skipping around the disk in order to piece together files. Web installs are also much slower unless you have a reliable internet connection (which is not a given). Installing from a USB stick (preferably of the 3.0 variety in a 3.0 port) is indeed the best option for all but the oldest of machines (in which case you'll need a boot floppy to start the machine up with a bootloader capable of initializing the boot medium (USB, CD/DVD/BD, network) and booting from it).
goalieca 11y ago
One really nice benefit of SSDs is that there is no needle or laser to seek. :)

by listic 11y ago

gambiting 11y ago

We do for PS4 disc images. Basically you want to lay out the data on the disc in such a way that the game can still be played while the system copies everything to the HDD. There are pages upon pages of manuals in PS4 SDK how to do it efficiently.

cnvogel 11y ago

At least for Windows there's now a way to install the OS so that it only occupies one continuous large filesystem image with the compressed install files, and only stores the changes to this frozen filesystem images in the "traditional" way. It's called "wimboot". http://labalec.fr/erwan/?p=1078 http://labalec.fr/erwan/?p=1078 That way installing the OS mainy consists of copying this huge image file, hence the "physical layout of bytes on the disk" will mostly be fixed. I'd guess that this could also be replicated in Linux, but personally I don't know if this is being done. I usually "debootstrap" or "pacstrap" my installs from a bootable USB stick :-).

caf 11y ago
Yes, you could replicate that in Linux using overlayfs, with a readonly mount of the static image as the the lower layer and a writeable upper layer using an ordinary filesystem. It would be interesting to try it.
shubb 11y ago
Sure. On one project we built a custom Linux distribution by mounting a file as a loop back file system and copying the right files in. We installed a grub boot sector at the start. Then we copied it as a file into a simple live usb. The live usb also containing a script that used dd to copy the data from that file over the start of the first disk of a system you plugged it into. Nice quick install usb, minimal work. We were installing into standard hardware so it was pretty good. But if your hardware varies this approach isn't great. We could also have created a fresh filesystem on the device and untarred into it but we wanted to keep se Linux attributes.
- cnvogel 11y ago
  dd'ing to the target device, and then growing the fs to the full disk extent seems to be a splendid idea. Especially as Linux tends to replace quite a lot of code over the usual curse of updates... For a cluster system I had built a netbootable system for reinstall that would restore a dump of the reference system's fs on a node and adjust hostnames, ssh host keys etc... Also was quite fast at that time..
derefr 11y ago
That's effectively how CoreOS works.

jeremiep 11y ago

Ubuntu targets a LOT of different hardware, it would be much harder to optimize the layout on their ISO image than it is to optimize for a fixed hardware platform like consoles.

Cthulhu_ 11y ago

Do you still use a CD / DVD to install your OS? That's probably the problem right there; CD's/DVD's are relatively slow. Go for an install off a flash disk (where physical layout shouldn't matter), and / or a web install where only the relevant software compiled for your hardware is downloaded.

yellowapple 11y ago
CDs/DVDs don't have to be slow (or at least not as slow as they typically are). Most speed problems are the same speed problems one encounters with spinny / non-SSD hard drives: namely, that data isn't sequential, requiring a lot of skipping around the disk in order to piece together files. Web installs are also much slower unless you have a reliable internet connection (which is not a given). Installing from a USB stick (preferably of the 3.0 variety in a 3.0 port) is indeed the best option for all but the oldest of machines (in which case you'll need a boot floppy to start the machine up with a bootloader capable of initializing the boot medium (USB, CD/DVD/BD, network) and booting from it).

goalieca 11y ago

One really nice benefit of SSDs is that there is no needle or laser to seek. :)