11 ms·

Unauthenticated Remote Code Execution in Erlang/OTP SSH

199 points by kimi 1y ago

deleted 1y ago
[deleted]
aftbit 1y ago
As I understand it, this is talking about an SSH server built into Erlang/OTP, not e.g. OpenSSH on a server with Erlang installed. >Any service using Erlang/OTP's SSH library for remote access such as those used in OT/IoT devices, edge computing devices are susceptible to exploitation. https://thehackernews.com/2025/04/critical-erlangotp-ssh-vulnerability.html https://thehackernews.com/2025/04/critical-erlangotp-ssh-vul...
- kimi 1y ago
  Yes - one of the many things that you can find in OTP is a programmable SSH/SCP client and server. The vulnerability is in the server component. See for example https://blog.differentpla.net/blog/2022/11/01/erlang-ssh/ https://blog.differentpla.net/blog/2022/11/01/erlang-ssh/
  - davidw 1y ago
    Erlang, because of its architecture, has something of a habit of people rewriting various protocols in Erlang itself, rather than calling out to some C library. This has pros and cons.
    - innocentoldguy 1y ago
      This is probably because C NIFs run in the same process as the Erlang scheduler. If you have a long-running or blocking NIF, it can starve the scheduler and cause significant performance degradation across the system.
      - natrys 1y ago
        I think they now have "dirty" NIFs that use a separate scheduler for this.
        throwawaymaths 1y ago
        yes, but there is a finite number of them, by default equal to the number of available cores. If your connection stays in c-land for too long you might run into trouble, if more than one connection are desired.
      - hinkley 1y ago
        I wonder if there's space for a libuv inspired solution now.
        toast0 1y ago
        libuv is more or less abstraction around an event loop for async i/o right? The BEAM is also more or less an abstraction around an event loop for async i/o. If you want async i/o in nifs, I think you want to integrate with BEAM's event loop. Inside NIFs, I think you want to use enif_select [1] (and friends), available since OTP 20 originally from 2017-06-21. In a port driver, you'd use driver_select [2] which I think has been around forever --- there's mentions of changes in R13 which I think was mostly release 2009-11-20 (that may have been R13B though). [1] https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_select https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_selec... [2] https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_select https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_...
        hinkley 1y ago
        It uses different threads in order to make infrequent blocking calls look like they are asynchronous. When we (or at least some quantity of “we”) want is infrequent native calls to be able to fail without taking the BEAM down. The problem with doing it with threads though is that a bad thread can still vomit all over working memory, still causing a panic even if it itself doesn’t panic.
    - toast0 1y ago
      Writing protocol code in Erlang is nice, because the parsing is so easy and clear. And if you want to do something that's not so easy by spawning a command, then you may as well build it in Erlang. And it's fun and symmetric to build both a server and a client... I've not looked at OTP SSH code, but I'd assume the ciphering is still calls to external c libraries, as it is in the OTP TLS code. Of course, easy protocol parsing doesn't do the whole job; state management is required too (and was missed here, clearly).
- rollcat 1y ago
  This is why I generally do not rely on SSH servers other than OpenSSH. It's (by far) the most widely deployed implementation, thoroughly battle-tested, etc. It's also hard to actually get pwned; the OpenBSD[1] guys believe in security as the default. There's some value in avoiding a monoculture, or choosing different trade-offs (e.g. binary size, memory usage). But as exemplified by this incident, any incentives must be carefully weighted against the risks. SSH is your final line of defence. [1]: https://www.openbsd.org/donations.html https://www.openbsd.org/donations.html
  - whizzter 1y ago
    There's a huge difference here, historically that was because many C codebases were vulnerable due to inherent C flaws and ssh daemons due to their age was C based. OpenBSD folks stances on coding and system design avoids pitfalls. This is an Erlang daemon, thus written in a managed language without buffer overflows,etc, but it seems like someone left a huge gaping logic hole to drive a bus through. SSH or not, this could've equally well been a logic hole in a base webserver,etc. I'd say this is more akin to the Log4j debacle, a perfectly safe language but bad design makes it vulnerable to something trivial.
  - PhilipRoman 1y ago
    I also have this principle, although I make an exception for https://tinyssh.org https://tinyssh.org
  - VWWHFSfQ 1y ago
    OpenSSH has actually been "pwned" numerous times though. It's a very desirable target.
    - throwawaymaths 1y ago
      yeah and iirc erlang's ssl was one of the only ssl implementations not affected by heaetbleed since erlang is memory safe
      - toast0 1y ago
rramadass 1y ago
For folks interested in the Security aspects of Erlang/BEAM languages the guidelines from Security Working Group of the Erlang Ecosystem Foundation are a good resource - https://security.erlef.org/ https://security.erlef.org/ and https://erlef.org/wg/security https://erlef.org/wg/security
formerly_proven 1y ago
If I interpret the patch correctly the issue seems to be that you could just ask for a channel and do a request_exec before authenticating. The regression test is: {send, hello}, {send, ssh_msg_kexinit}, {match, #ssh_msg_kexinit{_='_'}, receive_msg}, {send, SshMsgChannelOpen}, {send, SshMsgChannelRequest}, {match, disconnect(), receive_msg} https://github.com/erlang/otp/commit/6eef04130afc8b0ccb63c9a0d8650209cf54892f https://github.com/erlang/otp/commit/6eef04130afc8b0ccb63c9a... edit: Ah, found by the people at RUB, they do a lot of research in verifying protocol implementations iirc.
- phoe-krk 1y ago
  CWE-306, Missing Authentication for Critical Function, linked in the report seems to suggest the same. The score of 10.0 is damn spicy, too - you just ask the server to execute something for you, and it does so, no questions asked.
  - tialaramex 1y ago
    Design bug here: Clearly we need to run code as somebody, so, there's no reason to have infrastructure which just executes user code with the current context or server (presumably? or maybe an actual zero, ie root) context. If we design the software this way, when we try to write the erroneous code we're caught - oh, wait, which user is authenticated? We need to... oh... we shouldn't be here without authenticating.
    - toast0 1y ago
      Afaik, if you were running the Erlang SSH daemon, when you connect as an authorized user (or just ask!), it drops you into the Erlang debug shell. There's no concept of different users in the debug shell. Erlang dist is kind of an almost single system image cluster, with no security boundaries once you're connected to the cluster. (Well, not exactly no boundaries, you can mark a process sensitive, and ets tables private, and do some other things to make introspection difficult [1], but you can likely still spawn an os process debugger and get to everything; you would need to modify the OTP sources to disable os process spawning) I didn't think anybody would actually run the Erlang SSH daemon, but there's evidence that some do. It makes more sense to run openssh, so you can debug BEAM failures etc, and you can load a debug shell from your OS shell easily. [1] https://security.erlef.org/secure_coding_and_deployment_hardening/sensitive_data https://security.erlef.org/secure_coding_and_deployment_hard...
throwawaymaths 1y ago
most Elixir deployments are probably unaffected (obviously, please please check to be sure), as SSH is turned off by default. https://paraxial.io/blog/erlang-ssh https://paraxial.io/blog/erlang-ssh
- giraffe_lady 1y ago
  This is also true of erlang right? This module is part of the stdlib but if you haven't implemented access using it it's "turned off" in the same way any other libraries you aren't using are.
- __jonas 1y ago
  I'm assuming the most likely affected Elixir projects would be those using Nerves with SSH enabled and exposed to the public internet, as nerves_ssh wraps the OTP SSH daemon. Don't think that's a very common thing to do, even in my hobby projects I would only access a Nerves device through a VPN.
qwertox 1y ago
How does this affect servers like ejabberd? I just noticed that they upgraded their server yesterday [0] and am wondering if it could contain some kind of fix for this, or would this be unrelated? [0] https://github.com/processone/ejabberd/releases https://github.com/processone/ejabberd/releases
- toast0 1y ago
  ejabberd doesn't start the Erlang SSH daemon; or at least codesearch on their github doesn't have any reference to ssh other than something unrelated trying to figure out if a url is a git repo. I didn't think anyone actually ran the Erlang SSH daemon (although there's evidence that some people do!). It makes for a fun demo, but a regular OS shell is more useful, and you can attach a debug shell to an existing BEAM process from there.
  - codetrotter 1y ago
    > It makes for a fun demo, but a regular OS shell is more useful, and you can attach a debug shell to an existing BEAM process from there. OTOH for example in Go people sometimes use the SSH protocol to provide access to cool things like an SSH based chat, instead of using it for shell access. I haven’t looked at what the Erlang SSH server provides, but maybe you could do something like that? Write a chat server in Erlang and use the Erlang SSH server to provide users access to that chat?
    - toast0 1y ago
      Yes, the Erlang debug shell is just the default option; you can replace the interface with whatever, if you wanted to build a chat/mud/ascii cinema/etc, it should all be possible (but you'd be advised to review the OTP ssh code to confirm it does what you want and nothing extra)
bilekas 1y ago
There’s something really strange and upsetting reading this on an archive site that wont be around for much longer..
- WillPostForFood 1y ago
  It isn't going away!
aposm 1y ago
Oops..... we are currently trying to sell an elixir-based greenfield project internally. This doesn't affect elixir by default as other commenters pointed out, but still might make our project a bit harder to pitch to management...
- jerf 1y ago
  If your organization is looking for "the language ecosystem that never has any security vulnerabilities", pack it in and close up shop because you're not going to find one. How many, how often, and how they are handled is far more important. While the Erlang/Elixir ecosystem won't stop you from writing a network server that takes in a string and just blithely passes it along to a shell without analysis, overall the Erlang/Elixir ecosystem is very strong and lacks most of the footguns like an "eval" statement that get people. Though I will ding it a point for the most obvious way to run a shell command [1] taking just a string that goes to a shell rather than an array of parameters to a shell command. It is on the higher end of secure languages to write a network server in.
  - toast0 1y ago
    > overall the Erlang/Elixir ecosystem is very strong and lacks most of the footguns like an "eval" statement that get people Erlang has erl_eval [1] if you're looking for more ability to shoot yourself in the foot. You can call that from Elixir, but I guess that'd be weird; I'm not an Elixir person, but I'd bet you can shoot yourself in the foot if you try! There's always fun with dist and proc_lib:spawn(Node, Fun) [2], which you can put in a list comprehension with erlang:nodes() [3] if you want to shot yourself in many feet rapidly ;) [1] https://www.erlang.org/doc/apps/stdlib/erl_eval.html https://www.erlang.org/doc/apps/stdlib/erl_eval.html [2] https://www.erlang.org/doc/apps/stdlib/proc_lib.html#spawn/2 https://www.erlang.org/doc/apps/stdlib/proc_lib.html#spawn/2 [3] https://www.erlang.org/doc/apps/erts/erlang.html#nodes/0 https://www.erlang.org/doc/apps/erts/erlang.html#nodes/0
    - keepamovin 1y ago
      All this foot shooting - this is the problem with permissive gun laws. We should ideally lock down all firearms to prevent any civilian doing harm. Only the select few government agents should own firearms. obligatory /S (for internet) also: 2 months ago I posted this Erlang SSH SFTP flaw - https://news.ycombinator.com/item?id=43126360 https://news.ycombinator.com/item?id=43126360
r3tr0 1y ago
you could probably write a custom XDP program to parse and check for this payload using a tool like yeet and XDP_DROP it. https://yeet.cx https://yeet.cx you can try our sandbox at https://yeet.cx/play https://yeet.cx/play
marioflach 1y ago
I wrote a GitHub „clone“ a while ago. Implementing Git’s wire and transfer protocol directly in Elixir. https://git-scm.com/docs/protocol-v2 https://git-scm.com/docs/protocol-v2 https://git-scm.com/book/ms/v2/Git-on-the-Server-The-Protocols https://git-scm.com/book/ms/v2/Git-on-the-Server-The-Protoco... Adding support for Git over SSH was very easy using Erlang built-in SSH libs. https://github.com/redrabbit/git.limo https://github.com/redrabbit/git.limo https://github.com/redrabbit/git.limo/blob/master/apps/gitgud/lib/gitgud/ssh_server.ex https://github.com/redrabbit/git.limo/blob/master/apps/gitgu...
curtisszmania 1y ago
[dead]

199 points by kimi 1y ago

deleted 1y ago

[deleted]

As I understand it, this is talking about an SSH server built into Erlang/OTP, not e.g. OpenSSH on a server with Erlang installed. >Any service using Erlang/OTP's SSH library for remote access such as those used in OT/IoT devices, edge computing devices are susceptible to exploitation. https://thehackernews.com/2025/04/critical-erlangotp-ssh-vulnerability.html https://thehackernews.com/2025/04/critical-erlangotp-ssh-vul...

kimi 1y ago
Yes - one of the many things that you can find in OTP is a programmable SSH/SCP client and server. The vulnerability is in the server component. See for example https://blog.differentpla.net/blog/2022/11/01/erlang-ssh/ https://blog.differentpla.net/blog/2022/11/01/erlang-ssh/
- davidw 1y ago
  Erlang, because of its architecture, has something of a habit of people rewriting various protocols in Erlang itself, rather than calling out to some C library. This has pros and cons.
  - innocentoldguy 1y ago
    This is probably because C NIFs run in the same process as the Erlang scheduler. If you have a long-running or blocking NIF, it can starve the scheduler and cause significant performance degradation across the system.
    - natrys 1y ago
      I think they now have "dirty" NIFs that use a separate scheduler for this.
      - throwawaymaths 1y ago
        yes, but there is a finite number of them, by default equal to the number of available cores. If your connection stays in c-land for too long you might run into trouble, if more than one connection are desired.
    - hinkley 1y ago
      I wonder if there's space for a libuv inspired solution now.
      - toast0 1y ago
        libuv is more or less abstraction around an event loop for async i/o right? The BEAM is also more or less an abstraction around an event loop for async i/o. If you want async i/o in nifs, I think you want to integrate with BEAM's event loop. Inside NIFs, I think you want to use enif_select [1] (and friends), available since OTP 20 originally from 2017-06-21. In a port driver, you'd use driver_select [2] which I think has been around forever --- there's mentions of changes in R13 which I think was mostly release 2009-11-20 (that may have been R13B though). [1] https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_select https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_selec... [2] https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_select https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_...
        hinkley 1y ago
        It uses different threads in order to make infrequent blocking calls look like they are asynchronous. When we (or at least some quantity of “we”) want is infrequent native calls to be able to fail without taking the BEAM down. The problem with doing it with threads though is that a bad thread can still vomit all over working memory, still causing a panic even if it itself doesn’t panic.
  - toast0 1y ago
    Writing protocol code in Erlang is nice, because the parsing is so easy and clear. And if you want to do something that's not so easy by spawning a command, then you may as well build it in Erlang. And it's fun and symmetric to build both a server and a client... I've not looked at OTP SSH code, but I'd assume the ciphering is still calls to external c libraries, as it is in the OTP TLS code. Of course, easy protocol parsing doesn't do the whole job; state management is required too (and was missed here, clearly).
rollcat 1y ago
This is why I generally do not rely on SSH servers other than OpenSSH. It's (by far) the most widely deployed implementation, thoroughly battle-tested, etc. It's also hard to actually get pwned; the OpenBSD[1] guys believe in security as the default. There's some value in avoiding a monoculture, or choosing different trade-offs (e.g. binary size, memory usage). But as exemplified by this incident, any incentives must be carefully weighted against the risks. SSH is your final line of defence. [1]: https://www.openbsd.org/donations.html https://www.openbsd.org/donations.html
- whizzter 1y ago
  There's a huge difference here, historically that was because many C codebases were vulnerable due to inherent C flaws and ssh daemons due to their age was C based. OpenBSD folks stances on coding and system design avoids pitfalls. This is an Erlang daemon, thus written in a managed language without buffer overflows,etc, but it seems like someone left a huge gaping logic hole to drive a bus through. SSH or not, this could've equally well been a logic hole in a base webserver,etc. I'd say this is more akin to the Log4j debacle, a perfectly safe language but bad design makes it vulnerable to something trivial.
- PhilipRoman 1y ago
  I also have this principle, although I make an exception for https://tinyssh.org https://tinyssh.org
- VWWHFSfQ 1y ago
  OpenSSH has actually been "pwned" numerous times though. It's a very desirable target.
  - throwawaymaths 1y ago
    yeah and iirc erlang's ssl was one of the only ssl implementations not affected by heaetbleed since erlang is memory safe
    - toast0 1y ago

rramadass 1y ago

For folks interested in the Security aspects of Erlang/BEAM languages the guidelines from Security Working Group of the Erlang Ecosystem Foundation are a good resource - https://security.erlef.org/ https://security.erlef.org/ and https://erlef.org/wg/security https://erlef.org/wg/security

formerly_proven 1y ago

If I interpret the patch correctly the issue seems to be that you could just ask for a channel and do a request_exec before authenticating. The regression test is: {send, hello}, {send, ssh_msg_kexinit}, {match, #ssh_msg_kexinit{_='_'}, receive_msg}, {send, SshMsgChannelOpen}, {send, SshMsgChannelRequest}, {match, disconnect(), receive_msg} https://github.com/erlang/otp/commit/6eef04130afc8b0ccb63c9a0d8650209cf54892f https://github.com/erlang/otp/commit/6eef04130afc8b0ccb63c9a... edit: Ah, found by the people at RUB, they do a lot of research in verifying protocol implementations iirc.

phoe-krk 1y ago
CWE-306, Missing Authentication for Critical Function, linked in the report seems to suggest the same. The score of 10.0 is damn spicy, too - you just ask the server to execute something for you, and it does so, no questions asked.
- tialaramex 1y ago
  Design bug here: Clearly we need to run code as somebody, so, there's no reason to have infrastructure which just executes user code with the current context or server (presumably? or maybe an actual zero, ie root) context. If we design the software this way, when we try to write the erroneous code we're caught - oh, wait, which user is authenticated? We need to... oh... we shouldn't be here without authenticating.
  - toast0 1y ago
    Afaik, if you were running the Erlang SSH daemon, when you connect as an authorized user (or just ask!), it drops you into the Erlang debug shell. There's no concept of different users in the debug shell. Erlang dist is kind of an almost single system image cluster, with no security boundaries once you're connected to the cluster. (Well, not exactly no boundaries, you can mark a process sensitive, and ets tables private, and do some other things to make introspection difficult [1], but you can likely still spawn an os process debugger and get to everything; you would need to modify the OTP sources to disable os process spawning) I didn't think anybody would actually run the Erlang SSH daemon, but there's evidence that some do. It makes more sense to run openssh, so you can debug BEAM failures etc, and you can load a debug shell from your OS shell easily. [1] https://security.erlef.org/secure_coding_and_deployment_hardening/sensitive_data https://security.erlef.org/secure_coding_and_deployment_hard...

throwawaymaths 1y ago

most Elixir deployments are probably unaffected (obviously, please please check to be sure), as SSH is turned off by default. https://paraxial.io/blog/erlang-ssh https://paraxial.io/blog/erlang-ssh

giraffe_lady 1y ago
This is also true of erlang right? This module is part of the stdlib but if you haven't implemented access using it it's "turned off" in the same way any other libraries you aren't using are.
__jonas 1y ago
I'm assuming the most likely affected Elixir projects would be those using Nerves with SSH enabled and exposed to the public internet, as nerves_ssh wraps the OTP SSH daemon. Don't think that's a very common thing to do, even in my hobby projects I would only access a Nerves device through a VPN.

qwertox 1y ago

How does this affect servers like ejabberd? I just noticed that they upgraded their server yesterday [0] and am wondering if it could contain some kind of fix for this, or would this be unrelated? [0] https://github.com/processone/ejabberd/releases https://github.com/processone/ejabberd/releases

toast0 1y ago
ejabberd doesn't start the Erlang SSH daemon; or at least codesearch on their github doesn't have any reference to ssh other than something unrelated trying to figure out if a url is a git repo. I didn't think anyone actually ran the Erlang SSH daemon (although there's evidence that some people do!). It makes for a fun demo, but a regular OS shell is more useful, and you can attach a debug shell to an existing BEAM process from there.
- codetrotter 1y ago
  > It makes for a fun demo, but a regular OS shell is more useful, and you can attach a debug shell to an existing BEAM process from there. OTOH for example in Go people sometimes use the SSH protocol to provide access to cool things like an SSH based chat, instead of using it for shell access. I haven’t looked at what the Erlang SSH server provides, but maybe you could do something like that? Write a chat server in Erlang and use the Erlang SSH server to provide users access to that chat?
  - toast0 1y ago
    Yes, the Erlang debug shell is just the default option; you can replace the interface with whatever, if you wanted to build a chat/mud/ascii cinema/etc, it should all be possible (but you'd be advised to review the OTP ssh code to confirm it does what you want and nothing extra)

bilekas 1y ago

There’s something really strange and upsetting reading this on an archive site that wont be around for much longer..

WillPostForFood 1y ago
It isn't going away!

aposm 1y ago

Oops..... we are currently trying to sell an elixir-based greenfield project internally. This doesn't affect elixir by default as other commenters pointed out, but still might make our project a bit harder to pitch to management...

jerf 1y ago
If your organization is looking for "the language ecosystem that never has any security vulnerabilities", pack it in and close up shop because you're not going to find one. How many, how often, and how they are handled is far more important. While the Erlang/Elixir ecosystem won't stop you from writing a network server that takes in a string and just blithely passes it along to a shell without analysis, overall the Erlang/Elixir ecosystem is very strong and lacks most of the footguns like an "eval" statement that get people. Though I will ding it a point for the most obvious way to run a shell command [1] taking just a string that goes to a shell rather than an array of parameters to a shell command. It is on the higher end of secure languages to write a network server in.
- toast0 1y ago
  > overall the Erlang/Elixir ecosystem is very strong and lacks most of the footguns like an "eval" statement that get people Erlang has erl_eval [1] if you're looking for more ability to shoot yourself in the foot. You can call that from Elixir, but I guess that'd be weird; I'm not an Elixir person, but I'd bet you can shoot yourself in the foot if you try! There's always fun with dist and proc_lib:spawn(Node, Fun) [2], which you can put in a list comprehension with erlang:nodes() [3] if you want to shot yourself in many feet rapidly ;) [1] https://www.erlang.org/doc/apps/stdlib/erl_eval.html https://www.erlang.org/doc/apps/stdlib/erl_eval.html [2] https://www.erlang.org/doc/apps/stdlib/proc_lib.html#spawn/2 https://www.erlang.org/doc/apps/stdlib/proc_lib.html#spawn/2 [3] https://www.erlang.org/doc/apps/erts/erlang.html#nodes/0 https://www.erlang.org/doc/apps/erts/erlang.html#nodes/0
  - keepamovin 1y ago
    All this foot shooting - this is the problem with permissive gun laws. We should ideally lock down all firearms to prevent any civilian doing harm. Only the select few government agents should own firearms. obligatory /S (for internet) also: 2 months ago I posted this Erlang SSH SFTP flaw - https://news.ycombinator.com/item?id=43126360 https://news.ycombinator.com/item?id=43126360

r3tr0 1y ago

you could probably write a custom XDP program to parse and check for this payload using a tool like yeet and XDP_DROP it. https://yeet.cx https://yeet.cx you can try our sandbox at https://yeet.cx/play https://yeet.cx/play

marioflach 1y ago

I wrote a GitHub „clone“ a while ago. Implementing Git’s wire and transfer protocol directly in Elixir. https://git-scm.com/docs/protocol-v2 https://git-scm.com/docs/protocol-v2 https://git-scm.com/book/ms/v2/Git-on-the-Server-The-Protocols https://git-scm.com/book/ms/v2/Git-on-the-Server-The-Protoco... Adding support for Git over SSH was very easy using Erlang built-in SSH libs. https://github.com/redrabbit/git.limo https://github.com/redrabbit/git.limo https://github.com/redrabbit/git.limo/blob/master/apps/gitgud/lib/gitgud/ssh_server.ex https://github.com/redrabbit/git.limo/blob/master/apps/gitgu...

curtisszmania 1y ago

[dead]