5 ms·

The Surreal Horror of PAM (2021)

81 points by popcalc 4y ago

dmitrygr 4y ago
> how do you debug PAM? > It ain’t easy. VMWare workstation has a built-in gdb stub that can debug guest kernel, or any Linux guest userspace process quite easily. There is also support for windows guests, and even a visual studio plugin for that. I am surprised this did not come up as a very simple option to do this.
- xena 4y ago
  I go into more detail about this in my followup talk: https://xeiaso.net/talks/rustconf-2022-sheer-terror-pam#:~:text=So%20the%20answer%20of%20how%20you%20debug%20stuff%20in%20PAM%20boils%20down%20to%20%22carefully%22.%20You%20debug%20it%20carefully.%20You%20know%20what%2C%20here%27s%20a%20story%20of%20my%20first%20time%20debugging%20a%20custom%20PAM%20module https://xeiaso.net/talks/rustconf-2022-sheer-terror-pam#:~:t.... It is not easy in the slightest. PAM has aged like bottled milk in a desert.
  - monocasa 4y ago
    Can confirm from being the maintainer of a PAM module in a past job. The only thing that was stable was debug builds that enabled awful fprintfs to a file in /tmp. It felt like how a freshman in college debugs their code.
  - tbrownaw 4y ago
    So what's a proper way to solve the problem that PAM solves? Instead of an in-process library, have it be a daemon that responds to DBUS requests? Aside from the config language, this sounds mostly like "it's a pre-login modular C library, and that's alien territory".
    - xena 4y ago
      It would probably need kernel support for this to happen, but I'd like to see a daemon in the system that runs authentication logic and then arbitrarily curses the requesting process with the UID/GID/whatever it needs to get the job done.
      - stevekemp 4y ago
        How would that work with, for example a single deamon supporting multiple (different) remote users - for example ftpd, or sshd? (I imagine you could say that sshd forks for each user, but some of the legacy systems like ftpd/ircd wouldn't have that as an option..)
    - kaba0 4y ago
      I think Android has a more saner system, but I’m a layperson when it comes to these — android runs every user process under a different user that has very restricted access (restricted with SELinux as well), and they do extensive IPC. There is a daemon that handles elevated requests through IPC, so that’s where dynamic access control happens.
- collegeburner 4y ago
  forget debugging, just configuring PAM is a freaking nightmare. oh and make a mistake? can't log in to fix it lmao. i got really good about keeping a root shell open after i made that mistake a few times.
nyanpasu64 4y ago
Tangentially related: when I run Gitea on a LAN server under a Tailscale hostname, and i have Tailscale SSH enabled, do I have to use Gitea's built-in SSH server with a custom port? When I try using port 22, even with PAM on (as requested by Gitea), sshing to gitea@hostname asks for my tailscale login then gives me a shell session (causing git push to fail). https://tailscale.com/kb/1193/tailscale-ssh/#advertise-ssh-on-the-host https://tailscale.com/kb/1193/tailscale-ssh/#advertise-ssh-o... says tailscale ssh "configures tailscaled to intercept all traffic from your tailnet that is routed to port 22 on the Tailscale IP address". I suppose I could set an ACL to only make Tailscale SSH intercept logins to nyanpasu64, and not other users like root or gitea (I haven't tested if Tailscale will reject other users or passthrough to gitea). I'm not sure if I should only approve my account, all nonroot, or "all but root and gitea" (unsure if possible with current syntax).
- Joel_Mckay 4y ago
  If I recall Gitea can run its owns libssh instance on a different port, and requires manually importing keys for each user. If you are using the https interface by default, than one may need to explicitly use the ssh:// uri. Gitea should not be exposed to the web, and a port knock sequence is highly recommended to keep private projects private. ;)
  - nyanpasu64 4y ago
    I'm running Gitea in my LAN only, and currently hiding behind Tailscale (only listening to the Tailscale IP for HTTP, forgot to set that up for SSH over a custom port but that's not really a vulnerability) rather than exposed to other LAN devices. I may open it up to my full LAN in the future, I haven't decided.
    - Joel_Mckay 4y ago
      Another issue with Gitea we found was memory requirements are somewhat unclear. So if you have to version large binary images in a source tree, the daemon can develop a number of hard to diagnose failure modes. It may sound ridiculous, but make sure you have at least 4GiB of free ram. Most of the time it will never be used, but it is far better than chasing ghost issue for that edge case where it does. ;)
orangepurple 4y ago
It's a ripe opportunity for a rewrite in Rust
jolop 4y ago
This doesn't sound so bad, I'm not really clear on what their major complaint about PAM is. Sounds about the same amount of hassle as developing anything deeply embedded in the operating system. I've seen worse.
- xena 4y ago
  Author of the post here. You are making a factual error by calling me "his". Can you please edit your comment to use the word "her" or "their"? Thanks!
  - jolop 4y ago
    Unfortunately the site isn't letting me edit the comment. Sorry about that, I heard a male voice on the video and assumed he was the author.
    - dang 4y ago
      I've edited it to say 'their' now. Sorry for getting to this late.
  - michaelmrose 4y ago
    Hacker News only allows editing for a few minutes then you are stuck with what you wrote typos mistakes and all. Because of this it makes little sense to ask anyone to edit anything as by the time you've noticed and they've noticed your reply that ship has sailed.
    - dang 4y ago
      You can always ask us to edit it later. Emailing hn@ycombinator.com is the best way to ask us things.
      - michaelmrose 4y ago
        That's nice but do you think the benefits of not allowing editing make sense on the overall and makes sense in the current configuration. Of the top of my head it promotes a continuous coherent narrative not full of annoying deletions and comments that don't differ wildly from original posts. No posting something incredibly inflammatory, waiting for the conflagration, and then replacing original post with something milder. On the other hand I have definitely posted something from my phone and cursed when I realized I couldn't fix a typo or decided something could have been communicated better. It feels like the edit window could at least be a little longer.
tbrownaw 4y ago
Identifying a (prospective) user by their network origin? This must be that "zero trust" I've been hearing about, yes?
- lmm 4y ago
  Meh, it's a VPN and they're apparently enforcing good authentication at that level. I still favour treating the network as public and doing real TLS (because your services want to do that anyway), but single sign on is definitely what you want, and if it were Kerberos or something in place of Tailscale then the PAM side of things would be more or less the same (other than not having to reinvent the wheel, because PAM for Kerberos has already been implemented).
  - themoonisachees 4y ago
    The overall idea isn't that bad because tailscale basically already authes you on the network level, and in the kinds of infras this is for, this means you are already behind like 5 VPNs. I agree that automatically authing without user input is a bit much (eg worms) but tailscale has support for recieving and handling a "are you sure you want to conect to x" confirmation on the authentication source, it's just a matter of actually using it.
PaulHoule 4y ago
That is the furriest furry blog I have seen in some time.
rkeene2 4y ago
Building on the point that PAM is simple, I wrote a blog post helping you write your very first PAM module! https://rkeene.org/projects/info/wiki/222 https://rkeene.org/projects/info/wiki/222
candiddevmike 4y ago
Everytime I delve into PAM I can't help but think it's carrying a ton of legacy tech debt. It seems like it tries to support a lot of things out of the box that may not be relevant these days, like mail and limits. The modules themselves are littered with arcane options, and the entire thing is just this commented mess of inheritance and options that most users will just use to compare a hash in a file. On a basic debian install, there's around 455 lines of configs in /etc/pam.d! Maybe systemd will streamline it someday.
- duskwuff 4y ago
  > Everytime I delve into PAM I can't help but think it's carrying a ton of legacy tech debt. It seems like it tries to support a lot of things out of the box that may not be relevant these days, like mail and limits. Beyond that, PAM has been saddled with a whole slew of responsibilities which go beyond user authentication, like supporting anonymous FTP (pam_ftp), maintaining /var/log/lastlog (pam_lastlog), or telling shell users that they have new mail when they log in (pam_mail). A lot of it is pretty specific to supporting traditional shell login workflows on shared machines, and is rarely used in most installs.
  - takeda 4y ago
    I do like the module that automatically creates a home directory on first login. It works well when you use ldap or similar mechanism for authentication.
    - bayindirh 4y ago
      And if you don't like that, you can disable it. It's both modular and flexible.
  - gnu8 4y ago
    > rarely used in most installs. I was going to object but you are absolutely right. Most Linux installs are instances of something that are created and destroyed without a human ever touching them. However, I wouldn't go so far as to characterize these features as technical debt. A fully functional Linux system should be able to tell users if they have mail, for instance.
Jedd 4y ago
It's about to get more surreal & horrible as several vendors are pushing privilege access management systems, and despite being in the auth[z|n] industry (they should know better) and having ~17k non-conflicting TLAs to choose from, are gleefully re-purposing PAM.
wolfgang42 4y ago
> /etc/passwd is world-readable by design for some arcane reason The reason is actually very straightforward: if you can’t read /etc/password, you don’t know who anyone is! Reading this file is (or was, prior to the introduction of nsswitch et al) the way you mapped a UID to a login name, so if you couldn’t read it then anything that needed to show a user (ps, who, ls -l, etc) would just show numbers instead. Putting the actual password hash in it was, in hindsight, a mistake[1]; this was corrected pretty quickly with /etc/shadow but the file had to stick around with the same name because of all the things that were reading it already. (This is the same reason the “GECOS” field stuck around long after Bell Labs stopped using a GE computer for their print spooler.) [1] I’m sure it made sense at the time—the fact that the password wasn’t just stored in plaintext was a novelty in its own right and nobody was going to spend that much CPU time trying to crack hashes when there were better things to do with the computer.
- bayindirh 4y ago
  Also, one of the "cons" has been noted as it "works" on a single machine. You can "fix" that by enabling LDAP on the hosts and authenticating everything via that thing called an LDAP server. We're doing this for a very long time on 500+ machines, and it just works. If you want to sync configs, you can do this with another "femtoservice" running somewhere. I understand the concerns, but just badmouthing things because they're evolving for decades instead of built from top to bottom every three years is bad to put it mildly.
  - kazinator 4y ago
    The problem of cross-machine passwd was solved before LDAP by something called NIS, also from Sun Microsystems. NIS was called Yellow Pages originally, so it has program names like ypbind.
    - bayindirh 4y ago
      I remember the name ypbind, but didn't know what it was. Thanks for the info, greatly appreciated.
- 4y ago
secabeen 4y ago
> password, the things that check passwords or do other kinds of validation like that (if you want to use Google Authenticator TOTP codes, you’d do that here) password is for workflows that change passwords. It is not used for TOTP, that is an `auth` module. From the Google Authenticator PAM module docs: > Add this line to your PAM configuration file: `auth required pam_google_authenticator.so no_increment_hotp`
dang 4y ago
Discussed at the time: The Surreal Horror of Pam - https://news.ycombinator.com/item?id=29167560 https://news.ycombinator.com/item?id=29167560 - Nov 2021 (137 comments)
EGreg 4y ago
I think PAM was very cute in the office. Maybe I just wasn't aware of all the details.
nigma1337 4y ago
I recently wrote my first PAM module, using pam-rs[1] so i could do it in rust. It was a learning experience, a rather enjoyable one though. While going down the rabbit hole, I feel even deeper when my module kept halting in openssh, due to a bug[2] from 2018. I didn't really feel the pain of debugging the author had, most likely just because my application was simple enough to be tested via log statements though. [1] https://github.com/anowell/pam-rs https://github.com/anowell/pam-rs [2] https://bugzilla.mindrot.org/show_bug.cgi?id=2876 https://bugzilla.mindrot.org/show_bug.cgi?id=2876
kaliszad 4y ago
For those who just don't get PAM, I can recommend PAM Mastery by Michael W Lucas https://mwl.io/nonfiction/tools#pam https://mwl.io/nonfiction/tools#pam. It is readable and has a very specific sysadmin kind of humor I like. It certainly helped me to understand at least something. (His other books are also quite helpful, SSH Mastery, ZFS Mastery and others to name a few.)
- dns_snek 4y ago
  Thank you, bookmarked. It's been a while since I last tried to understand what PAM actually is - or does, before giving up, knowing less than I did before, with the answer seemingly being "everything".
effnorwood 4y ago
deleted 4y ago
[deleted]

81 points by popcalc 4y ago

> how do you debug PAM? > It ain’t easy. VMWare workstation has a built-in gdb stub that can debug guest kernel, or any Linux guest userspace process quite easily. There is also support for windows guests, and even a visual studio plugin for that. I am surprised this did not come up as a very simple option to do this.

xena 4y ago
I go into more detail about this in my followup talk: https://xeiaso.net/talks/rustconf-2022-sheer-terror-pam#:~:text=So%20the%20answer%20of%20how%20you%20debug%20stuff%20in%20PAM%20boils%20down%20to%20%22carefully%22.%20You%20debug%20it%20carefully.%20You%20know%20what%2C%20here%27s%20a%20story%20of%20my%20first%20time%20debugging%20a%20custom%20PAM%20module https://xeiaso.net/talks/rustconf-2022-sheer-terror-pam#:~:t.... It is not easy in the slightest. PAM has aged like bottled milk in a desert.
- monocasa 4y ago
  Can confirm from being the maintainer of a PAM module in a past job. The only thing that was stable was debug builds that enabled awful fprintfs to a file in /tmp. It felt like how a freshman in college debugs their code.
- tbrownaw 4y ago
  So what's a proper way to solve the problem that PAM solves? Instead of an in-process library, have it be a daemon that responds to DBUS requests? Aside from the config language, this sounds mostly like "it's a pre-login modular C library, and that's alien territory".
  - xena 4y ago
    It would probably need kernel support for this to happen, but I'd like to see a daemon in the system that runs authentication logic and then arbitrarily curses the requesting process with the UID/GID/whatever it needs to get the job done.
    - stevekemp 4y ago
      How would that work with, for example a single deamon supporting multiple (different) remote users - for example ftpd, or sshd? (I imagine you could say that sshd forks for each user, but some of the legacy systems like ftpd/ircd wouldn't have that as an option..)
  - kaba0 4y ago
    I think Android has a more saner system, but I’m a layperson when it comes to these — android runs every user process under a different user that has very restricted access (restricted with SELinux as well), and they do extensive IPC. There is a daemon that handles elevated requests through IPC, so that’s where dynamic access control happens.
collegeburner 4y ago
forget debugging, just configuring PAM is a freaking nightmare. oh and make a mistake? can't log in to fix it lmao. i got really good about keeping a root shell open after i made that mistake a few times.

nyanpasu64 4y ago

Tangentially related: when I run Gitea on a LAN server under a Tailscale hostname, and i have Tailscale SSH enabled, do I have to use Gitea's built-in SSH server with a custom port? When I try using port 22, even with PAM on (as requested by Gitea), sshing to gitea@hostname asks for my tailscale login then gives me a shell session (causing git push to fail). https://tailscale.com/kb/1193/tailscale-ssh/#advertise-ssh-on-the-host https://tailscale.com/kb/1193/tailscale-ssh/#advertise-ssh-o... says tailscale ssh "configures tailscaled to intercept all traffic from your tailnet that is routed to port 22 on the Tailscale IP address". I suppose I could set an ACL to only make Tailscale SSH intercept logins to nyanpasu64, and not other users like root or gitea (I haven't tested if Tailscale will reject other users or passthrough to gitea). I'm not sure if I should only approve my account, all nonroot, or "all but root and gitea" (unsure if possible with current syntax).

Joel_Mckay 4y ago
If I recall Gitea can run its owns libssh instance on a different port, and requires manually importing keys for each user. If you are using the https interface by default, than one may need to explicitly use the ssh:// uri. Gitea should not be exposed to the web, and a port knock sequence is highly recommended to keep private projects private. ;)
- nyanpasu64 4y ago
  I'm running Gitea in my LAN only, and currently hiding behind Tailscale (only listening to the Tailscale IP for HTTP, forgot to set that up for SSH over a custom port but that's not really a vulnerability) rather than exposed to other LAN devices. I may open it up to my full LAN in the future, I haven't decided.
  - Joel_Mckay 4y ago
    Another issue with Gitea we found was memory requirements are somewhat unclear. So if you have to version large binary images in a source tree, the daemon can develop a number of hard to diagnose failure modes. It may sound ridiculous, but make sure you have at least 4GiB of free ram. Most of the time it will never be used, but it is far better than chasing ghost issue for that edge case where it does. ;)

orangepurple 4y ago

It's a ripe opportunity for a rewrite in Rust

jolop 4y ago

This doesn't sound so bad, I'm not really clear on what their major complaint about PAM is. Sounds about the same amount of hassle as developing anything deeply embedded in the operating system. I've seen worse.

xena 4y ago
Author of the post here. You are making a factual error by calling me "his". Can you please edit your comment to use the word "her" or "their"? Thanks!
- jolop 4y ago
  Unfortunately the site isn't letting me edit the comment. Sorry about that, I heard a male voice on the video and assumed he was the author.
  - dang 4y ago
    I've edited it to say 'their' now. Sorry for getting to this late.
- michaelmrose 4y ago
  Hacker News only allows editing for a few minutes then you are stuck with what you wrote typos mistakes and all. Because of this it makes little sense to ask anyone to edit anything as by the time you've noticed and they've noticed your reply that ship has sailed.
  - dang 4y ago
    You can always ask us to edit it later. Emailing hn@ycombinator.com is the best way to ask us things.
    - michaelmrose 4y ago
      That's nice but do you think the benefits of not allowing editing make sense on the overall and makes sense in the current configuration. Of the top of my head it promotes a continuous coherent narrative not full of annoying deletions and comments that don't differ wildly from original posts. No posting something incredibly inflammatory, waiting for the conflagration, and then replacing original post with something milder. On the other hand I have definitely posted something from my phone and cursed when I realized I couldn't fix a typo or decided something could have been communicated better. It feels like the edit window could at least be a little longer.

tbrownaw 4y ago

Identifying a (prospective) user by their network origin? This must be that "zero trust" I've been hearing about, yes?

lmm 4y ago
Meh, it's a VPN and they're apparently enforcing good authentication at that level. I still favour treating the network as public and doing real TLS (because your services want to do that anyway), but single sign on is definitely what you want, and if it were Kerberos or something in place of Tailscale then the PAM side of things would be more or less the same (other than not having to reinvent the wheel, because PAM for Kerberos has already been implemented).
- themoonisachees 4y ago
  The overall idea isn't that bad because tailscale basically already authes you on the network level, and in the kinds of infras this is for, this means you are already behind like 5 VPNs. I agree that automatically authing without user input is a bit much (eg worms) but tailscale has support for recieving and handling a "are you sure you want to conect to x" confirmation on the authentication source, it's just a matter of actually using it.

PaulHoule 4y ago

That is the furriest furry blog I have seen in some time.

rkeene2 4y ago

Building on the point that PAM is simple, I wrote a blog post helping you write your very first PAM module! https://rkeene.org/projects/info/wiki/222 https://rkeene.org/projects/info/wiki/222

candiddevmike 4y ago

Everytime I delve into PAM I can't help but think it's carrying a ton of legacy tech debt. It seems like it tries to support a lot of things out of the box that may not be relevant these days, like mail and limits. The modules themselves are littered with arcane options, and the entire thing is just this commented mess of inheritance and options that most users will just use to compare a hash in a file. On a basic debian install, there's around 455 lines of configs in /etc/pam.d! Maybe systemd will streamline it someday.

duskwuff 4y ago
> Everytime I delve into PAM I can't help but think it's carrying a ton of legacy tech debt. It seems like it tries to support a lot of things out of the box that may not be relevant these days, like mail and limits. Beyond that, PAM has been saddled with a whole slew of responsibilities which go beyond user authentication, like supporting anonymous FTP (pam_ftp), maintaining /var/log/lastlog (pam_lastlog), or telling shell users that they have new mail when they log in (pam_mail). A lot of it is pretty specific to supporting traditional shell login workflows on shared machines, and is rarely used in most installs.
- takeda 4y ago
  I do like the module that automatically creates a home directory on first login. It works well when you use ldap or similar mechanism for authentication.
  - bayindirh 4y ago
    And if you don't like that, you can disable it. It's both modular and flexible.
- gnu8 4y ago
  > rarely used in most installs. I was going to object but you are absolutely right. Most Linux installs are instances of something that are created and destroyed without a human ever touching them. However, I wouldn't go so far as to characterize these features as technical debt. A fully functional Linux system should be able to tell users if they have mail, for instance.

Jedd 4y ago

It's about to get more surreal & horrible as several vendors are pushing privilege access management systems, and despite being in the auth[z|n] industry (they should know better) and having ~17k non-conflicting TLAs to choose from, are gleefully re-purposing PAM.

wolfgang42 4y ago

> /etc/passwd is world-readable by design for some arcane reason The reason is actually very straightforward: if you can’t read /etc/password, you don’t know who anyone is! Reading this file is (or was, prior to the introduction of nsswitch et al) the way you mapped a UID to a login name, so if you couldn’t read it then anything that needed to show a user (ps, who, ls -l, etc) would just show numbers instead. Putting the actual password hash in it was, in hindsight, a mistake[1]; this was corrected pretty quickly with /etc/shadow but the file had to stick around with the same name because of all the things that were reading it already. (This is the same reason the “GECOS” field stuck around long after Bell Labs stopped using a GE computer for their print spooler.) [1] I’m sure it made sense at the time—the fact that the password wasn’t just stored in plaintext was a novelty in its own right and nobody was going to spend that much CPU time trying to crack hashes when there were better things to do with the computer.

bayindirh 4y ago
Also, one of the "cons" has been noted as it "works" on a single machine. You can "fix" that by enabling LDAP on the hosts and authenticating everything via that thing called an LDAP server. We're doing this for a very long time on 500+ machines, and it just works. If you want to sync configs, you can do this with another "femtoservice" running somewhere. I understand the concerns, but just badmouthing things because they're evolving for decades instead of built from top to bottom every three years is bad to put it mildly.
- kazinator 4y ago
  The problem of cross-machine passwd was solved before LDAP by something called NIS, also from Sun Microsystems. NIS was called Yellow Pages originally, so it has program names like ypbind.
  - bayindirh 4y ago
    I remember the name ypbind, but didn't know what it was. Thanks for the info, greatly appreciated.
4y ago

secabeen 4y ago

> password, the things that check passwords or do other kinds of validation like that (if you want to use Google Authenticator TOTP codes, you’d do that here) password is for workflows that change passwords. It is not used for TOTP, that is an `auth` module. From the Google Authenticator PAM module docs: > Add this line to your PAM configuration file: `auth required pam_google_authenticator.so no_increment_hotp`

dang 4y ago

Discussed at the time: The Surreal Horror of Pam - https://news.ycombinator.com/item?id=29167560 https://news.ycombinator.com/item?id=29167560 - Nov 2021 (137 comments)

EGreg 4y ago

I think PAM was very cute in the office. Maybe I just wasn't aware of all the details.

nigma1337 4y ago

I recently wrote my first PAM module, using pam-rs[1] so i could do it in rust. It was a learning experience, a rather enjoyable one though. While going down the rabbit hole, I feel even deeper when my module kept halting in openssh, due to a bug[2] from 2018. I didn't really feel the pain of debugging the author had, most likely just because my application was simple enough to be tested via log statements though. [1] https://github.com/anowell/pam-rs https://github.com/anowell/pam-rs [2] https://bugzilla.mindrot.org/show_bug.cgi?id=2876 https://bugzilla.mindrot.org/show_bug.cgi?id=2876

kaliszad 4y ago

For those who just don't get PAM, I can recommend PAM Mastery by Michael W Lucas https://mwl.io/nonfiction/tools#pam https://mwl.io/nonfiction/tools#pam. It is readable and has a very specific sysadmin kind of humor I like. It certainly helped me to understand at least something. (His other books are also quite helpful, SSH Mastery, ZFS Mastery and others to name a few.)

dns_snek 4y ago
Thank you, bookmarked. It's been a while since I last tried to understand what PAM actually is - or does, before giving up, knowing less than I did before, with the answer seemingly being "everything".

effnorwood 4y ago

deleted 4y ago

[deleted]