10 ms·

The Linux Upskill Challenge

275 points by underanalyzer 4y ago

biglost 4y ago
ip a ss -tnaopu grep -C 5 grep -E ‘dick|ass’ nano -c foo grep -A 5 ls -lrth
mattbillenstein 4y ago
I think the next level above this might be https://sadservers.com/ https://sadservers.com/
- asicsp 4y ago
  Discussion: https://news.ycombinator.com/item?id=33344142 https://news.ycombinator.com/item?id=33344142 (597 points | 3 months ago | 128 comments)
- totetsu 4y ago
  hehe.. lsof | grep bad.log ps -ef | grep (pid) sudo apt remove python3 kill -9 (pid)
  - deleted 4y ago
    [deleted]
  - tripdout 4y ago
    I got lucky: ps ax showed a python3 process running /home/ubuntu/badlog.py, so I knew to kill that. If it wasn't named so obviously it would've been much harder to find it.
    - totetsu 4y ago
      lsof is list open files. Good for finding things like what process is writing to a log file. You could also go dig around in /proc
- fduran 4y ago
  Oh hi, author here, AMA :-) I'm happy that a lot of people use and find SadServers beneficial while not costing me a lot of money. Still a lot of features to implement on the website and a shrinking backlog of scenario ideas to materialize (if somebody has ideas for Linux/Docker/Kubernetes etc troubleshooting scenarios, please let me know).
  - jeroenhd 4y ago
    I've just tried several of your challenges and they're all painfully accurate for real-world scenarios. I will definitely point people at these next time I'll get asked how I learned to fix <random Linux configuration problem>! As for suggestions, here are some random things I needed to do recently: - resize the boot partition of an OS (don't know how doable this is with your vserver setup, maybe use one of those WASM Linux emulators?) - set up a systemd service/timer/socket that starts at the right time and responds correctly to reloads/restarts - set up IPv6 correctly - troubleshoot why a device wasn't connecting to the WiFi (DHCP service problem!) - set up a VPN (wireguard/openvpn/etc). Expert mode: make the remote endpoint have an A/AAAA record that the server isn't listening on - troubleshoot why some of my devices couldn't ssh into a server despite the pubkeys being in the authorized_keys folder (old sshd version didn't understand the most recent key algorithm!). Bonus problem: ~/.ssh had the wrong permissions so the authorized keys weren't loading. - renew an ACME/letsencrypt certificate in nginx in proxy mode (location / was proxied but location /.well-known/... shouldn't have been!) - check your preferred smtp daemon to see if it's set as an open relay - upgrade postgres from an old version to a new version without data loss (hard mode: the partition postgres uses by default doesn't have the free space to make a copy and migrate the data) - figure out why the firewall isn't blocking port 1234 despite UFW being enabled and a block-all rule being present (it was because of Docker iptables rules overriding UFW rules) - update a package that has some kind of dependency issue (i.e. an external repository that is no longer needed) - make Ubuntu shut up about Ubuntu Pro and stop it from fetching ads on ssh login - alter a systemd service file so that it no longer runs as root (hard mode: set up dynamic users and other hardening features)
    - lazyant 4y ago
      thanks! (some of these ideas are not possible to implement in the current setup but I can explore some of the other ones)
cbsks 4y ago
I’m on team vim, but it is still interesting to me that the challenge has vim but not emacs. I would think that an introduction to Linux would include both.
- asicsp 4y ago
  Probably because emacs is not installed by default?
  - sigjuice 4y ago
    TRAMP could also work, depending on the situation.
- xylophile 4y ago
  Regardless of which team you're on or why, the base installs of almost every distro includes vi or vim, but many leave out emacs and nano. Since you have to pick one editor to be the first one taught, it makes sense to teach vim to avoid losing the learner.
  - analognoise 4y ago
    This is as it should be. Emacs is about enlightenment; enlightenment is a path to be walked not a destination; it is a journey you must choose. Along the way, you might accidentally edit text files. That's not the point. Anyway, makes sense to me why it's not default included.
    - ttrefa 4y ago
      > Emacs is about enlightenment; And that's why I'm on team Vim. I rather not use a tool that needs/uses cult-like slogans to convince people to keep them hooked. I want the usefulness to be the reason, not some kind of belief system.
      - imwillofficial 4y ago
greazy 4y ago
I'm very happy to see this continuing. The original creator passed it onto the current maintainer before he died. He was super nice.
29athrowaway 4y ago
You can achieve much more learning in 20 days. Important fundamentals are omitted but some oddly specific subjects are cherry-picked and given an unusual amount of attention like running nmap. Running nmap (a portscanner) can get you fired from your job and there's precedent of this. The text formatting is inconsistent. Reader mode makes it better, but a simple markdown file would have clearer. Explaining what a service is before explaining what a task is, is weird. APT is mentioned, but APT is specific to Debian-based distros. Debian is not mentioned a single time in the article. Other distros use other package management systems. You should use nftables instead of iptables. iptables is deprecated. This article just reinforces that "learning Linux" is simply learning some few random commands.
- CalRobert 4y ago
  What alternatives do you suggest?
  - 29athrowaway 4y ago
    Learn about the file system, permissions, processes, signals, threads, the scheduler, standard input and output, networking, etc.
    - CalRobert 4y ago
      Yeah, I just haven't seen a single book/resource that does this - maybe I'm not looking hard enough. "the linux command line" from nostarch press was good but, of course, focused on commands
      - 29athrowaway 4y ago
        Take a look at "The Linux Programming Interface". It is a thick book that you can use as a reference when needed.
- imwillofficial 4y ago
  This type of bizarrely hostile criticism would be so much more helpful as a more thorough list of areas to improve AND detailed suggestions on improvement. Anyone can take shots, but a PR takes guts. Like you come in hot, then proceeded to not tell me how to learn more in 20 days, you tease.
antisceptic 4y ago
sudo -u fred -i # to become fred
- arminiusreturns 4y ago
  sudo -u $user -c '$commandtoexechere" to just run something as the user and drop you back to your own shell.
lrvick 4y ago
Honestly most of the people that approach me to learn Linux, I have them follow the Gentoo Handbook to compile and configure a Linux workstation from scratch.
- wrayjustin 4y ago
  This has been my go-to training approach for decades. Many of those individuals are now in high-seniority roles that encompasses Linux Administration. It is a tried and true method. If you want to be _good_ with Linux (including administration), build it: Follow the Gentoo Handbook, and restart every time something is wrong or fails. Alternatively, if you want to be a semi-competent user you can go the path of installing a user-friendly distribution (like Ubuntu, etc.) and get various Windows-based workstreams working (Gaming, Windows-Only Software through Wine, etc.) But this is less likely to push you to the same tier.
  - cinntaile 4y ago
    I assume this is very similar to the Linux from scratch book? https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/ Any advantages or disadvantages?
    - licebmi__at__ 4y ago
      Gentoo handbook method is a lot friendlier than following LFS, yet you stil have to deal somewhat with linux internals. IMO the main advantage over LFS is that is close enough to day to day linux administration that it serves more as a real experience than LFS which seems more like an academic experiment. OTOH, watching stuff compile is the worst part. I would complement the experience with using something like arch for a few weeks as main driver. Quicker than watch compilation logs, still you have to deal enough with linux internals.
- BirAdam 4y ago
  Well, “Linux” is a broad thing that involves a ton of software. I always advocate that people research a problem domain and what software solutions exist for that domain in a standard Linux installation.
amelius 4y ago
What's the best way to run software on an old Ubuntu system that I don't want to upgrade or run apt-get on, and I don't want to run stuff inside a Virtual Machine, and snap is broken, and docker as well? Compiling stuff seems too much trouble because of all the dependencies. I was hoping for Nix to save me here, but it only compiles for a specific libc version.
- klysm 4y ago
  It sounds like your asking how to install software without using precompiled artifacts and also without compiling stuff.
  - amelius 4y ago
    Well, I want to run new versions of software on a system in which I basically don't want to touch system directories like /usr and /lib etc.
    - igetspam 4y ago
      You can download precompiled binaries of a lot of things from git these days. You can modify your PATH to include a location of your choosing and put the binaries there. It probably won't work well though because you'll be missing dependencies. You could also chase down debs, download those and install them manually and specify your install path. (I don't have the deb switches memorized, so many pages are your friend.) You'll have dependency problems here too but you'll get friendly warnings that can help you hunt down other missing packages and you can work your way up from there. It won't be fun. I'm glad I started in Slackware and used it for about 15 years as my daily driver. I'm comfortable with Linux The Hard Way and when people tell me how hard it is to run Linux because all the weird kinks, I just chuckle.
      - amelius 4y ago
        Hey, I started with Slackware too, and like you said, some of those installation methods won't be fun. I'm looking for something that Just Works^TM.
BirAdam 4y ago
So, in years past, I lamented that people didn’t know basic sysadmin skills and therefore invented complex solutions to what are rather simple problems of configuration within common Linux software. They’d write entirely new pieces of software to deal with something I saw as trivial, and they did this because they didn’t know a solution existed. These days, I don’t lament this at all. As I see it, people wanted to make new on their own instead of learning what was there. This is just how younger people are. As they get older they naturally want to learn how things used to be, and the cycle will repeat. They will be the ones lamenting, and the younger folks will make it all new. I don’t remember who said it but: those who don’t understand UNIX are doomed to reinvent it poorly.
- wrldos 4y ago
  I've seen that quote a thousand times and I disagree with it profusely. After 30 years of it now on my part, I know we can and should do better. And that does mean reinventing it, not being resigned to the status quo.
  - abnry 4y ago
    Care to give some examples? I am still too young in Unix years to be wise enough to judge what is good and what is bad.
    - wrldos 4y ago
      Top three bits of pain from me: 1. Constant weak reparsing of formats in shell pipelines. 2. Completely dangerous and unsafe API and principal programming language. 3. Trying to remember several different domain specific languages.
      - hdjjhhvvhga 4y ago
        > 2. Completely dangerous and unsafe API and principal programming language. Apart from the fact that these days you can hardly call C principal (yes, it's still used for the Linux and BSD kernels, but quite rarely for any new userland project at all), what are your main gripes with the API? It is well understood, stable, most people know its limitations, it's being used by most servers and clients on the Internet, so I'm curious why you call it "completely dangerous and unsafe".
akeck 4y ago
The post mentions using crontab for scheduling tasks. Recently, I needed to use systemd user account job scheduling. I found it pretty nice once I got the hang of it.

275 points by underanalyzer 4y ago

biglost 4y ago

ip a ss -tnaopu grep -C 5 grep -E ‘dick|ass’ nano -c foo grep -A 5 ls -lrth

mattbillenstein 4y ago

I think the next level above this might be https://sadservers.com/ https://sadservers.com/

asicsp 4y ago
Discussion: https://news.ycombinator.com/item?id=33344142 https://news.ycombinator.com/item?id=33344142 (597 points | 3 months ago | 128 comments)
totetsu 4y ago
hehe.. lsof | grep bad.log ps -ef | grep (pid) sudo apt remove python3 kill -9 (pid)
- deleted 4y ago
  [deleted]
- tripdout 4y ago
  I got lucky: ps ax showed a python3 process running /home/ubuntu/badlog.py, so I knew to kill that. If it wasn't named so obviously it would've been much harder to find it.
  - totetsu 4y ago
    lsof is list open files. Good for finding things like what process is writing to a log file. You could also go dig around in /proc
fduran 4y ago
Oh hi, author here, AMA :-) I'm happy that a lot of people use and find SadServers beneficial while not costing me a lot of money. Still a lot of features to implement on the website and a shrinking backlog of scenario ideas to materialize (if somebody has ideas for Linux/Docker/Kubernetes etc troubleshooting scenarios, please let me know).
- jeroenhd 4y ago
  I've just tried several of your challenges and they're all painfully accurate for real-world scenarios. I will definitely point people at these next time I'll get asked how I learned to fix <random Linux configuration problem>! As for suggestions, here are some random things I needed to do recently: - resize the boot partition of an OS (don't know how doable this is with your vserver setup, maybe use one of those WASM Linux emulators?) - set up a systemd service/timer/socket that starts at the right time and responds correctly to reloads/restarts - set up IPv6 correctly - troubleshoot why a device wasn't connecting to the WiFi (DHCP service problem!) - set up a VPN (wireguard/openvpn/etc). Expert mode: make the remote endpoint have an A/AAAA record that the server isn't listening on - troubleshoot why some of my devices couldn't ssh into a server despite the pubkeys being in the authorized_keys folder (old sshd version didn't understand the most recent key algorithm!). Bonus problem: ~/.ssh had the wrong permissions so the authorized keys weren't loading. - renew an ACME/letsencrypt certificate in nginx in proxy mode (location / was proxied but location /.well-known/... shouldn't have been!) - check your preferred smtp daemon to see if it's set as an open relay - upgrade postgres from an old version to a new version without data loss (hard mode: the partition postgres uses by default doesn't have the free space to make a copy and migrate the data) - figure out why the firewall isn't blocking port 1234 despite UFW being enabled and a block-all rule being present (it was because of Docker iptables rules overriding UFW rules) - update a package that has some kind of dependency issue (i.e. an external repository that is no longer needed) - make Ubuntu shut up about Ubuntu Pro and stop it from fetching ads on ssh login - alter a systemd service file so that it no longer runs as root (hard mode: set up dynamic users and other hardening features)
  - lazyant 4y ago
    thanks! (some of these ideas are not possible to implement in the current setup but I can explore some of the other ones)

cbsks 4y ago

I’m on team vim, but it is still interesting to me that the challenge has vim but not emacs. I would think that an introduction to Linux would include both.

asicsp 4y ago
Probably because emacs is not installed by default?
- sigjuice 4y ago
  TRAMP could also work, depending on the situation.
xylophile 4y ago
Regardless of which team you're on or why, the base installs of almost every distro includes vi or vim, but many leave out emacs and nano. Since you have to pick one editor to be the first one taught, it makes sense to teach vim to avoid losing the learner.
- analognoise 4y ago
  This is as it should be. Emacs is about enlightenment; enlightenment is a path to be walked not a destination; it is a journey you must choose. Along the way, you might accidentally edit text files. That's not the point. Anyway, makes sense to me why it's not default included.
  - ttrefa 4y ago
    > Emacs is about enlightenment; And that's why I'm on team Vim. I rather not use a tool that needs/uses cult-like slogans to convince people to keep them hooked. I want the usefulness to be the reason, not some kind of belief system.
    - imwillofficial 4y ago

greazy 4y ago

I'm very happy to see this continuing. The original creator passed it onto the current maintainer before he died. He was super nice.

29athrowaway 4y ago

You can achieve much more learning in 20 days. Important fundamentals are omitted but some oddly specific subjects are cherry-picked and given an unusual amount of attention like running nmap. Running nmap (a portscanner) can get you fired from your job and there's precedent of this. The text formatting is inconsistent. Reader mode makes it better, but a simple markdown file would have clearer. Explaining what a service is before explaining what a task is, is weird. APT is mentioned, but APT is specific to Debian-based distros. Debian is not mentioned a single time in the article. Other distros use other package management systems. You should use nftables instead of iptables. iptables is deprecated. This article just reinforces that "learning Linux" is simply learning some few random commands.

CalRobert 4y ago
What alternatives do you suggest?
- 29athrowaway 4y ago
  Learn about the file system, permissions, processes, signals, threads, the scheduler, standard input and output, networking, etc.
  - CalRobert 4y ago
    Yeah, I just haven't seen a single book/resource that does this - maybe I'm not looking hard enough. "the linux command line" from nostarch press was good but, of course, focused on commands
    - 29athrowaway 4y ago
      Take a look at "The Linux Programming Interface". It is a thick book that you can use as a reference when needed.
imwillofficial 4y ago
This type of bizarrely hostile criticism would be so much more helpful as a more thorough list of areas to improve AND detailed suggestions on improvement. Anyone can take shots, but a PR takes guts. Like you come in hot, then proceeded to not tell me how to learn more in 20 days, you tease.

antisceptic 4y ago

sudo -u fred -i # to become fred

arminiusreturns 4y ago
sudo -u $user -c '$commandtoexechere" to just run something as the user and drop you back to your own shell.

lrvick 4y ago

Honestly most of the people that approach me to learn Linux, I have them follow the Gentoo Handbook to compile and configure a Linux workstation from scratch.

wrayjustin 4y ago
This has been my go-to training approach for decades. Many of those individuals are now in high-seniority roles that encompasses Linux Administration. It is a tried and true method. If you want to be _good_ with Linux (including administration), build it: Follow the Gentoo Handbook, and restart every time something is wrong or fails. Alternatively, if you want to be a semi-competent user you can go the path of installing a user-friendly distribution (like Ubuntu, etc.) and get various Windows-based workstreams working (Gaming, Windows-Only Software through Wine, etc.) But this is less likely to push you to the same tier.
- cinntaile 4y ago
  I assume this is very similar to the Linux from scratch book? https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/ Any advantages or disadvantages?
  - licebmi__at__ 4y ago
    Gentoo handbook method is a lot friendlier than following LFS, yet you stil have to deal somewhat with linux internals. IMO the main advantage over LFS is that is close enough to day to day linux administration that it serves more as a real experience than LFS which seems more like an academic experiment. OTOH, watching stuff compile is the worst part. I would complement the experience with using something like arch for a few weeks as main driver. Quicker than watch compilation logs, still you have to deal enough with linux internals.
BirAdam 4y ago
Well, “Linux” is a broad thing that involves a ton of software. I always advocate that people research a problem domain and what software solutions exist for that domain in a standard Linux installation.

amelius 4y ago

What's the best way to run software on an old Ubuntu system that I don't want to upgrade or run apt-get on, and I don't want to run stuff inside a Virtual Machine, and snap is broken, and docker as well? Compiling stuff seems too much trouble because of all the dependencies. I was hoping for Nix to save me here, but it only compiles for a specific libc version.

klysm 4y ago
It sounds like your asking how to install software without using precompiled artifacts and also without compiling stuff.
- amelius 4y ago
  Well, I want to run new versions of software on a system in which I basically don't want to touch system directories like /usr and /lib etc.
  - igetspam 4y ago
    You can download precompiled binaries of a lot of things from git these days. You can modify your PATH to include a location of your choosing and put the binaries there. It probably won't work well though because you'll be missing dependencies. You could also chase down debs, download those and install them manually and specify your install path. (I don't have the deb switches memorized, so many pages are your friend.) You'll have dependency problems here too but you'll get friendly warnings that can help you hunt down other missing packages and you can work your way up from there. It won't be fun. I'm glad I started in Slackware and used it for about 15 years as my daily driver. I'm comfortable with Linux The Hard Way and when people tell me how hard it is to run Linux because all the weird kinks, I just chuckle.
    - amelius 4y ago
      Hey, I started with Slackware too, and like you said, some of those installation methods won't be fun. I'm looking for something that Just Works^TM.

BirAdam 4y ago

So, in years past, I lamented that people didn’t know basic sysadmin skills and therefore invented complex solutions to what are rather simple problems of configuration within common Linux software. They’d write entirely new pieces of software to deal with something I saw as trivial, and they did this because they didn’t know a solution existed. These days, I don’t lament this at all. As I see it, people wanted to make new on their own instead of learning what was there. This is just how younger people are. As they get older they naturally want to learn how things used to be, and the cycle will repeat. They will be the ones lamenting, and the younger folks will make it all new. I don’t remember who said it but: those who don’t understand UNIX are doomed to reinvent it poorly.

wrldos 4y ago
I've seen that quote a thousand times and I disagree with it profusely. After 30 years of it now on my part, I know we can and should do better. And that does mean reinventing it, not being resigned to the status quo.
- abnry 4y ago
  Care to give some examples? I am still too young in Unix years to be wise enough to judge what is good and what is bad.
  - wrldos 4y ago
    Top three bits of pain from me: 1. Constant weak reparsing of formats in shell pipelines. 2. Completely dangerous and unsafe API and principal programming language. 3. Trying to remember several different domain specific languages.
    - hdjjhhvvhga 4y ago
      > 2. Completely dangerous and unsafe API and principal programming language. Apart from the fact that these days you can hardly call C principal (yes, it's still used for the Linux and BSD kernels, but quite rarely for any new userland project at all), what are your main gripes with the API? It is well understood, stable, most people know its limitations, it's being used by most servers and clients on the Internet, so I'm curious why you call it "completely dangerous and unsafe".

akeck 4y ago

The post mentions using crontab for scheduling tasks. Recently, I needed to use systemd user account job scheduling. I found it pretty nice once I got the hang of it.