21 ms·

Build yourself a Linux

497 points by AlexeyBrin 9y ago

marenkay 9y ago
Huge fang on Linux From Scratch myself, after reading this I wonder if someone has tried the same with FreeBSD! Or the Darwin sources released for OSX (.. not talking about dormant PureDarwin project)
apeacox 9y ago
Well, I'd use http://linuxfromscratch.org/ http://linuxfromscratch.org/ for that matter...
- agumonkey 9y ago
  LFS is infinitely more involved and am not sure for good reasons. I don't like to say it because it's someone else project but I feel important parts are missing, format is wrong, and without BLFS the goal is only half realized.
lanna 9y ago
If you are interested in building your own Linux, the LFS project has a lot of detailed information: http://linuxfromscratch.org http://linuxfromscratch.org
- btgeekboy 9y ago
  I see this project as the next logical step beyond LFS. LFS will help you get one up and running from start to finish, but once you do that, you might want to try it a bit more free-form. That's where this project seems to fit in.
  - cmrx64 9y ago
    Certainly you mean the other way around? This project gets you busybox and a kernel, which barely qualifies as "up and running from start to finish", whereas following this doesn't even get you a compiler in the target system.
thu 9y ago
This sounds like Aboriginal Linux: http://landley.net/aboriginal/about.html http://landley.net/aboriginal/about.html
digi_owl 9y ago
Both this and LFS reminds me that Linux makes sense until you get the DEs involved. At that point shit just sprawls all over the place as there are no longer any notion of layers.
- thomastjeffery 9y ago
  The problem is described perfectly with the name: desktop environment. The very goal of a desktop environment is to create an entire environment, rather than using the one that is already there. Desktop environments like GNOME and MADE have so many worthless parts that are tightly coupled dependencies, they end up becoming the same bloated mess that Windows and OS X are. If a desktop environment focused on creating separate packages that played well with the underlying system, they would not have this problem.
- djsumdog 9y ago
  Back in 2002 I ran LFS for a semester and a half. Then I got tired of saving all my ./configure flags and switch to Redhat and gained a respect for package management. :-P Eventually I started using Gentoo. Still, I'm really glad I built and ran an LFS for nearly a year. It helped me build a really in-dept knowledge of Linux that's helped me as I later moved into embedded development. Today I still use Gentoo (which feels like LFS with package management) and Void and would recommend either for Linux devs.
  - digi_owl 9y ago
    I am of two minds about package managers after having used Gobolinux for some years. Because once i got familiar with Gobolinux and how it handles things, i can't shake the feel that most package managers introduce as many problems as they solve. The biggest is that while _nix have sonames to handle multiple lib versions installed at the same time, most package managers balk at there being more than one package of a given name installed at the same time (Debian apparently works around this by including soname-like info in the package names).
    - tanderson92 9y ago
faragon 9y ago
Beautiful.
erikb 9y ago
I really wondered what's so special about this project that aims to achieve what probably more projects aimed to achieved than there are lines of code in the kernel. It's not the goal, it's not the OS. But the documentation is a sight to behold! Very clear, detailed, interesting writing style, and it puts together quite a few frustrating topics in a simple, structured matter. Wow and kudos! Keep on writing docs, please!
- michielderhaeg 9y ago
  Your kind words are much appreciated. Many people seem to indeed be missing the point. I wasn't trying to create a new distro, I'm just trying to teach people in a way so that it would just take a few hours of your time.
colemickens 9y ago
Seem neat to learn, but for something maintainable, LinuxKit seems interesting.
thom_nic 9y ago
A similar process is building a custom kernel and rootfs for an ARM device such as the beaglebone. Olimex actually has a good tutorial for their device: https://www.olimex.com/wiki/AM335x https://www.olimex.com/wiki/AM335x This is only slightly more complicated due to the need to cross-build but I found it fairly easy with qemu-static-arm and prebuilt cross toolchain packages for Ubuntu/ Debian. The benefit is that you can develop for a target device that is not your PC, so no worry about messing up the bootloader and leaving your PC in a state where you need a recovery CD to fix it and boot. Just get a USB-serial cable :) You can also try buildroot or yocto, although I had no interest in building every package manually versus relying on Debian's repos.
thom_nic 9y ago
A similar process is building a custom kernel and rootfs for an ARM device such as the beaglebone. Olimex actually has a good tutorial for their device: https://www.olimex.com/wiki/AM335x https://www.olimex.com/wiki/AM335x This is only slightly more complicated due to the need to cross-build but I found it fairly easy with qemu-static-arm and prebuilt cross toolchain packages for Ubuntu/ Debian. The benefit is that you can develop for a target device that is not your PC, so no worry about messing up the bootloader and leaving your PC in a state where you need a recovery CD to fix it and boot. Just get a USB-serial cable :) You can also try buildroot or yocto, although I had no interest in building every package manually versus relying on Debian's repos.
throw2016 9y ago
Following this guide will get you something very close to a base Alpine Linuxwith busybox. Alpine is fairly minimal out of the box and even eschews grub for syslinux. The upside with Alpine is if you need features and packages they are an install away. But if the purpose is to learn about compiling the kernel and how the system initalizes this is a decent start.
fizixer 9y ago
Have you even looked at the LFS project[1]? And what does your guide provide that LFS doesn't? [1] http://www.linuxfromscratch.org/ http://www.linuxfromscratch.org/
- partycoder 9y ago
  Exactly my thoughts, though more projects focused on this wouldn't do any harm.
- accurrent 9y ago
  Actually LFS teaches you about the whole GNU Userland, getting a kernel up and running with busybox is much easier. This project does just that - teaching you the bare minimum.
  - jbit 9y ago
    The embedded version of CLFS (Cross-Compiled Linux From Scratch) covers building a musl + busybox system: http://clfs.org/view/clfs-embedded/x86/ http://clfs.org/view/clfs-embedded/x86/ But the more resources available on this kind of thing the better IMHO, there's not really a one size fits all :)
- exDM69 9y ago
  This uses BusyBox instead of GNU userland tools. You can complete this in an afternoon, but you can't get LFS done in that time.
  - agumonkey 9y ago
    except by skipping through ALFS :p
- jokr004 9y ago
  The user who submitted this to HN is not the same person who owns this github repo.
- michielderhaeg 9y ago
airswimmer 9y ago
I don't think this is much useful. You should check out the linuxfromscratch.org
blanket_the_cat 9y ago
This is awesome. I've been building almost the exact same project, along almost the same timeline (based on the commit history). Mostly an excuse to learn more advanced Bash, and Linux Internals/Features I've never had a good excuse to explore. Gonna release next week. Hope I get as warm a reception. Kudos on an awesome project!
akavel 9y ago
Would be cool to port this to Nix package manager machinery (i.e. make this kinda an alternative to NixOS).
Sir_Cmpwn 9y ago
Building a Linux distro from scratch has been one of the most tasking projects I've attempted. There are two phases: the frustration phase, and the tedious phase. Bootstrapping it is an incredibly frustrating process - I restarted from scratch 5 times before any attempt ever got to the tedious phase. The tedious phase never ends. You have to make hundreds of packages to get to a usable desktop system. I've made 407 packages and I still don't have a desktop to show for it (I'm hoping to get sway working tomorrow, I think I have about 30 packages left). Still, I've finally gotten to a somewhat practical self-hosting system. My laptop runs it on the metal and all of the infrastructure (website, bugzilla, git hosting, mirrors) is on a server running my distro too. It's taken almost a year of frustration, but it's very rewarding.
- 0xcde4c3db 9y ago
  Good for you for sticking with it. I've started in that direction a few times, but it really is amazing just how necessary all those hacky distro-maintained build scripts and patches are. It's tempting to think that they're just tweaking things to fit their conventions and handling edge cases, but they're often just getting it to not explode by default.
  - Sir_Cmpwn 9y ago
    I've made several compromises so far. I've been making a note of them with the expectation (or hope) that I'll get to them, or a bright eyed new contributor might (I have some of those :D).
    - sq_ 9y ago
      Do you have a Github repo or something similar for the distro? I'd love to see it if you do.
      - smcl 9y ago
        Or even a blog post or something talking about it. I understand that sharing your experiences with the rest of the world might not be top priority while you're hammering away at sorting out these packages so I don't expect you to stop what you're doing purely to satisfy us on HN! But if you've already got something I'd love to see it :-)
peterwwillis 9y ago
I remember when HOWTOs were actually maintained over time so their instructions were up to date. Blogs killed HOWTOs.
ausjke 9y ago
This is an awesome write-up. did not know losetup can do what kpartx does now with the option -P, I did similar things in the past but this is a good update for me.
asciimo 9y ago
This is almost as complicated as building a Javascript web application.
- philbo 9y ago
  I would upvote this question except no, it isn't. Sorry, JavaScript, but you are the complicatedset!
- duncan_bayne 9y ago
  Except that the recommended tools don't change every few months ;)
  - qb45 9y ago
    Yeah, pmount, hal, ivman, udisks, consolekit, consolekit2, policykit, upstart, systemd, oss, alsa, pulseaudio, xfree, xorg, wayland, mir, kms, gtk, gtk2, gtk3, qt9, motif, unity, loadlin, lilo, grub, grub2, ghtml, webkit, qtwebkit, webkitgtk, webkitgtk2... I guess you're right, they change only every few years :)
    - duncan_bayne 9y ago
      He did say almost as complicated ;) (Also: QuickBASIC 4.5 forever! Assuming that's the meaning of your nick).
      - qb45 9y ago
        Yep, that's what got me started on programming as a kid. And it still has better strings and arrays than C.
    - agumonkey 9y ago
      That's why my setup (arch) was as bare as possible. kernel, grub, x, wmii, emacs, chrome, ffmpeg and mpv. With time it grew alas.
Jaruzel 9y ago
I've crashed and burned a couple of times trying to complete LinuxFromScratch, so I may give this a go - It seems a bit clearer on the core steps.
felixsanz 9y ago
Awesome! Good job. This also helps understand what the distro installer does.
jonathanstrange 9y ago
I have a question related to this article, though not directly. If building completely from scratch turns out too cumbersome and time-consuming, what would be the easiest way of building a minimal, fast starting distro with graphical user interface and networking whose only purpose is to run one application on x86 hardware in kiosk mode? Is there a distro builder for dummies?
- ChickeNES 9y ago
  Not exactly a distro builder, but Alpine (https://alpinelinux.org/ https://alpinelinux.org/) is a lightweight distro that follows an install-only-what-you-need philosophy.
- kasabali 9y ago
  Buildroot
- rijoja 9y ago
  Sure there is LFS. I suppose arch could give you a fairly minimal installation. Really when you think of it harddisc usage is hardly the problem here so even though it might seem silly having a 10GB installation, it will cost you nothing and save you time. But if you want it quick install debian or arch or whatever and simply start only X11 and whatever application you want to run. If there is no window manager or an extremely limited mode, you got kiosk mode.
- slezyr 9y ago
  https://susestudio.com/ https://susestudio.com/ (NOT REALY WHAT YOU WANT) You can pick packages and build a distro from browser. NixOS - You can make one config file to specify how your system should look like. It's easy to make GUI for it.
- Nullabillity 9y ago
  NixOS is awesome for stuff like this. Very easy to define your own "base image" and layer whatever you want on top of it. Binary packages by default, but you can override the compilation flags for any any package, Gentoo-style. Configuration files are also packages, which frees you from the cruft that systems like Chef or Puppet tend to accumulate. Oh, and NixOps is awesome for managing multiple machines. And it can automagically set up VMs corresponding to your production network. SUSE Studio has a nice GUI and is easy to get started with, but AFAIK it doesn't have a nice story for configuration management, or managing the systems after they've been deployed.
agumonkey 9y ago
Let's branch to add : sysvinit/BSD, init, OpenRC, upstart, systemd, SMF, launchd, Epoch, finit .. https://wiki.gentoo.org/wiki/Comparison_of_init_systems https://wiki.gentoo.org/wiki/Comparison_of_init_systems
Siecje 9y ago
Has anyone used Tinycore Linux? http://tinycorelinux.net/ http://tinycorelinux.net/
Ericson2314 9y ago
I feel like just reading Nixpkgs is probably just as edifying, tbh.
rijoja 9y ago
I am following this guide to build a kernel. But it seems like that instead of getting the headers from the kernel source they are using a github repository which only contains the headers to save downloading time. All fine and dandy if the latest commit to this repo wasn't from 3 years ago!!

497 points by AlexeyBrin 9y ago

Huge fang on Linux From Scratch myself, after reading this I wonder if someone has tried the same with FreeBSD! Or the Darwin sources released for OSX (.. not talking about dormant PureDarwin project)

apeacox 9y ago

Well, I'd use http://linuxfromscratch.org/ http://linuxfromscratch.org/ for that matter...

agumonkey 9y ago
LFS is infinitely more involved and am not sure for good reasons. I don't like to say it because it's someone else project but I feel important parts are missing, format is wrong, and without BLFS the goal is only half realized.

lanna 9y ago

If you are interested in building your own Linux, the LFS project has a lot of detailed information: http://linuxfromscratch.org http://linuxfromscratch.org

btgeekboy 9y ago
I see this project as the next logical step beyond LFS. LFS will help you get one up and running from start to finish, but once you do that, you might want to try it a bit more free-form. That's where this project seems to fit in.
- cmrx64 9y ago
  Certainly you mean the other way around? This project gets you busybox and a kernel, which barely qualifies as "up and running from start to finish", whereas following this doesn't even get you a compiler in the target system.

thu 9y ago

This sounds like Aboriginal Linux: http://landley.net/aboriginal/about.html http://landley.net/aboriginal/about.html

digi_owl 9y ago

Both this and LFS reminds me that Linux makes sense until you get the DEs involved. At that point shit just sprawls all over the place as there are no longer any notion of layers.

thomastjeffery 9y ago
The problem is described perfectly with the name: desktop environment. The very goal of a desktop environment is to create an entire environment, rather than using the one that is already there. Desktop environments like GNOME and MADE have so many worthless parts that are tightly coupled dependencies, they end up becoming the same bloated mess that Windows and OS X are. If a desktop environment focused on creating separate packages that played well with the underlying system, they would not have this problem.
djsumdog 9y ago
Back in 2002 I ran LFS for a semester and a half. Then I got tired of saving all my ./configure flags and switch to Redhat and gained a respect for package management. :-P Eventually I started using Gentoo. Still, I'm really glad I built and ran an LFS for nearly a year. It helped me build a really in-dept knowledge of Linux that's helped me as I later moved into embedded development. Today I still use Gentoo (which feels like LFS with package management) and Void and would recommend either for Linux devs.
- digi_owl 9y ago
  I am of two minds about package managers after having used Gobolinux for some years. Because once i got familiar with Gobolinux and how it handles things, i can't shake the feel that most package managers introduce as many problems as they solve. The biggest is that while _nix have sonames to handle multiple lib versions installed at the same time, most package managers balk at there being more than one package of a given name installed at the same time (Debian apparently works around this by including soname-like info in the package names).
  - tanderson92 9y ago

faragon 9y ago

Beautiful.

erikb 9y ago

I really wondered what's so special about this project that aims to achieve what probably more projects aimed to achieved than there are lines of code in the kernel. It's not the goal, it's not the OS. But the documentation is a sight to behold! Very clear, detailed, interesting writing style, and it puts together quite a few frustrating topics in a simple, structured matter. Wow and kudos! Keep on writing docs, please!

michielderhaeg 9y ago
Your kind words are much appreciated. Many people seem to indeed be missing the point. I wasn't trying to create a new distro, I'm just trying to teach people in a way so that it would just take a few hours of your time.

colemickens 9y ago

Seem neat to learn, but for something maintainable, LinuxKit seems interesting.

thom_nic 9y ago

A similar process is building a custom kernel and rootfs for an ARM device such as the beaglebone. Olimex actually has a good tutorial for their device: https://www.olimex.com/wiki/AM335x https://www.olimex.com/wiki/AM335x This is only slightly more complicated due to the need to cross-build but I found it fairly easy with qemu-static-arm and prebuilt cross toolchain packages for Ubuntu/ Debian. The benefit is that you can develop for a target device that is not your PC, so no worry about messing up the bootloader and leaving your PC in a state where you need a recovery CD to fix it and boot. Just get a USB-serial cable :) You can also try buildroot or yocto, although I had no interest in building every package manually versus relying on Debian's repos.

thom_nic 9y ago

throw2016 9y ago

Following this guide will get you something very close to a base Alpine Linuxwith busybox. Alpine is fairly minimal out of the box and even eschews grub for syslinux. The upside with Alpine is if you need features and packages they are an install away. But if the purpose is to learn about compiling the kernel and how the system initalizes this is a decent start.

fizixer 9y ago

Have you even looked at the LFS project[1]? And what does your guide provide that LFS doesn't? [1] http://www.linuxfromscratch.org/ http://www.linuxfromscratch.org/

partycoder 9y ago
Exactly my thoughts, though more projects focused on this wouldn't do any harm.
accurrent 9y ago
Actually LFS teaches you about the whole GNU Userland, getting a kernel up and running with busybox is much easier. This project does just that - teaching you the bare minimum.
- jbit 9y ago
  The embedded version of CLFS (Cross-Compiled Linux From Scratch) covers building a musl + busybox system: http://clfs.org/view/clfs-embedded/x86/ http://clfs.org/view/clfs-embedded/x86/ But the more resources available on this kind of thing the better IMHO, there's not really a one size fits all :)
exDM69 9y ago
This uses BusyBox instead of GNU userland tools. You can complete this in an afternoon, but you can't get LFS done in that time.
- agumonkey 9y ago
  except by skipping through ALFS :p
jokr004 9y ago
The user who submitted this to HN is not the same person who owns this github repo.
michielderhaeg 9y ago

airswimmer 9y ago

I don't think this is much useful. You should check out the linuxfromscratch.org

blanket_the_cat 9y ago

This is awesome. I've been building almost the exact same project, along almost the same timeline (based on the commit history). Mostly an excuse to learn more advanced Bash, and Linux Internals/Features I've never had a good excuse to explore. Gonna release next week. Hope I get as warm a reception. Kudos on an awesome project!

akavel 9y ago

Would be cool to port this to Nix package manager machinery (i.e. make this kinda an alternative to NixOS).

Sir_Cmpwn 9y ago

Building a Linux distro from scratch has been one of the most tasking projects I've attempted. There are two phases: the frustration phase, and the tedious phase. Bootstrapping it is an incredibly frustrating process - I restarted from scratch 5 times before any attempt ever got to the tedious phase. The tedious phase never ends. You have to make hundreds of packages to get to a usable desktop system. I've made 407 packages and I still don't have a desktop to show for it (I'm hoping to get sway working tomorrow, I think I have about 30 packages left). Still, I've finally gotten to a somewhat practical self-hosting system. My laptop runs it on the metal and all of the infrastructure (website, bugzilla, git hosting, mirrors) is on a server running my distro too. It's taken almost a year of frustration, but it's very rewarding.

0xcde4c3db 9y ago
Good for you for sticking with it. I've started in that direction a few times, but it really is amazing just how necessary all those hacky distro-maintained build scripts and patches are. It's tempting to think that they're just tweaking things to fit their conventions and handling edge cases, but they're often just getting it to not explode by default.
- Sir_Cmpwn 9y ago
  I've made several compromises so far. I've been making a note of them with the expectation (or hope) that I'll get to them, or a bright eyed new contributor might (I have some of those :D).
  - sq_ 9y ago
    Do you have a Github repo or something similar for the distro? I'd love to see it if you do.
    - smcl 9y ago
      Or even a blog post or something talking about it. I understand that sharing your experiences with the rest of the world might not be top priority while you're hammering away at sorting out these packages so I don't expect you to stop what you're doing purely to satisfy us on HN! But if you've already got something I'd love to see it :-)

peterwwillis 9y ago

I remember when HOWTOs were actually maintained over time so their instructions were up to date. Blogs killed HOWTOs.

ausjke 9y ago

This is an awesome write-up. did not know losetup can do what kpartx does now with the option -P, I did similar things in the past but this is a good update for me.

asciimo 9y ago

This is almost as complicated as building a Javascript web application.

philbo 9y ago
I would upvote this question except no, it isn't. Sorry, JavaScript, but you are the complicatedset!
duncan_bayne 9y ago
Except that the recommended tools don't change every few months ;)
- qb45 9y ago
  Yeah, pmount, hal, ivman, udisks, consolekit, consolekit2, policykit, upstart, systemd, oss, alsa, pulseaudio, xfree, xorg, wayland, mir, kms, gtk, gtk2, gtk3, qt9, motif, unity, loadlin, lilo, grub, grub2, ghtml, webkit, qtwebkit, webkitgtk, webkitgtk2... I guess you're right, they change only every few years :)
  - duncan_bayne 9y ago
    He did say almost as complicated ;) (Also: QuickBASIC 4.5 forever! Assuming that's the meaning of your nick).
    - qb45 9y ago
      Yep, that's what got me started on programming as a kid. And it still has better strings and arrays than C.
  - agumonkey 9y ago
    That's why my setup (arch) was as bare as possible. kernel, grub, x, wmii, emacs, chrome, ffmpeg and mpv. With time it grew alas.

Jaruzel 9y ago

I've crashed and burned a couple of times trying to complete LinuxFromScratch, so I may give this a go - It seems a bit clearer on the core steps.

felixsanz 9y ago

Awesome! Good job. This also helps understand what the distro installer does.

jonathanstrange 9y ago

I have a question related to this article, though not directly. If building completely from scratch turns out too cumbersome and time-consuming, what would be the easiest way of building a minimal, fast starting distro with graphical user interface and networking whose only purpose is to run one application on x86 hardware in kiosk mode? Is there a distro builder for dummies?

ChickeNES 9y ago
Not exactly a distro builder, but Alpine (https://alpinelinux.org/ https://alpinelinux.org/) is a lightweight distro that follows an install-only-what-you-need philosophy.
kasabali 9y ago
Buildroot
rijoja 9y ago
Sure there is LFS. I suppose arch could give you a fairly minimal installation. Really when you think of it harddisc usage is hardly the problem here so even though it might seem silly having a 10GB installation, it will cost you nothing and save you time. But if you want it quick install debian or arch or whatever and simply start only X11 and whatever application you want to run. If there is no window manager or an extremely limited mode, you got kiosk mode.
slezyr 9y ago
https://susestudio.com/ https://susestudio.com/ (NOT REALY WHAT YOU WANT) You can pick packages and build a distro from browser. NixOS - You can make one config file to specify how your system should look like. It's easy to make GUI for it.
Nullabillity 9y ago
NixOS is awesome for stuff like this. Very easy to define your own "base image" and layer whatever you want on top of it. Binary packages by default, but you can override the compilation flags for any any package, Gentoo-style. Configuration files are also packages, which frees you from the cruft that systems like Chef or Puppet tend to accumulate. Oh, and NixOps is awesome for managing multiple machines. And it can automagically set up VMs corresponding to your production network. SUSE Studio has a nice GUI and is easy to get started with, but AFAIK it doesn't have a nice story for configuration management, or managing the systems after they've been deployed.

agumonkey 9y ago

Let's branch to add : sysvinit/BSD, init, OpenRC, upstart, systemd, SMF, launchd, Epoch, finit .. https://wiki.gentoo.org/wiki/Comparison_of_init_systems https://wiki.gentoo.org/wiki/Comparison_of_init_systems

Siecje 9y ago

Has anyone used Tinycore Linux? http://tinycorelinux.net/ http://tinycorelinux.net/

Ericson2314 9y ago

I feel like just reading Nixpkgs is probably just as edifying, tbh.

rijoja 9y ago

I am following this guide to build a kernel. But it seems like that instead of getting the headers from the kernel source they are using a github repository which only contains the headers to save downloading time. All fine and dandy if the latest commit to this repo wasn't from 3 years ago!!