4 ms·

No team is capable of writing complex software without eventually introducing security bugs, and we (Zulip) are no exception. I'm disappointed that we introduce

No team is capable of writing complex software without eventually introducing security bugs, and we (Zulip) are no exception. I'm disappointed that we introduced this bug, and would like to apologize to our users. As a bit of commentary, my experience is that most SaaS vendors fix security bugs quietly. Others will do an brief internal investigation and only consider doing a public disclosure if they believe data privacy regulations require them to do so, because they discovered that the vulnerability had been in fact exploited. Usually public disclosure decisions are made by business executives with PR considerations in mind, so that shouldn't be surprising. See, for example, https://www.zdnet.com/article/slack-resets-passwords-for-1-of-its-users-because-of-2015-hack/ https://www.zdnet.com/article/slack-resets-passwords-for-1-o.... Zulip's core values include transparency, and our policy is to publicly disclose all security bugs that we discover with a formal CVE number, and this blog post is part of that practice. As noted in the blog post, we did an extensive audit and emailed every customer who our audit logs could not prove was not affected with enough detail to let them audit the situation. We did that despite our audit finding no evidence that this vulnerability was ever intentionally exploited, because we think it's the right thing to do. Some technical detail I can add for those interested is that our postmortem plans following this incident include further investment in the internal audit log system used for the investigation (`RealmAuditLog`, if any readers feel like grepping our codebase). Improvements we made in mid-2020 were very helpful in investigating whether this bug had in fact been exploited, and we take this as a sign that we should increase investment in that system.

tabbott 5y ago
As a side note, I'm surprised that this specific security disclosure ended up on the Hacker News homepage, while our security release about an RCE issue affecting self-hosted Zulip servers last month did not (https://blog.zulip.com/2022/01/25/zulip-server-4-9-security-release/ https://blog.zulip.com/2022/01/25/zulip-server-4-9-security-...). The core bug there an insecure secret generation algorithm in upstream Erlang/RabbitMQ, which seems more broadly notable to me! Despite our spending many hours composing detailed reports and lobbying security teams to get this problem fixed for everyone, not just Zulip, `apt install rabbitmq-server` continues to make a Debian/Ubuntu system vulnerable to RCE, and this fact has apparently been public for years. We did manage to get `rabbitmq-server` fixed in Debian testing, and it is scheduled to be included in the next Debian stable release, I would really have thought an RCE issue like this would result in an immediate security advisory. As a final remark, if you self-host Zulip, please subscribe to our announcements mailing list (https://groups.google.com/g/zulip-announce https://groups.google.com/g/zulip-announce) and make sure your installation is up to date, so that you benefit from the hard work our security team does to keep Zulip users safe.
- Ndymium 5y ago
  The blog post tries to paint the RabbitMQ cookie generation as a security issue, but the Erlang distribution cookie is NOT a security feature. Indeed, the Erlang documentation states: > "Security" here does not mean cryptographically secure, but rather security against accidental misuse, such as preventing a node from connecting to a cluster with which it is not intended to communicate. The purpose of the cookie is to guard against _accidental misuse_, such as a user connecting to the wrong node (prod vs testing?). The cookie is not meant to secure a system against unauthorised use. Any system relying on the cookie for security should already be considered broken. Thus I can see why there has not been such hurry to "fix" the issue.
  - tabbott 5y ago
    That argument would hold more water if there were an additional security mechanism in place preventing `apt install rabbitmq-server` from exposing a system to a remote code execution. It seems to be that the simplest fix for this problem is to make the cookie secure, since doing so is likely effective with minimal effort. Another option would be to have clustering disabled by default. If you'd like to summarize the upstream issue as "RabbitMQ as well as every distributors of it that we've investigated allows remote execution of arbitrary code on your server through having no effective access control mechanism for the distribution port, which is open by default after installing the software.", that summary is defensible. That still sounds to me like a critical security issue. One should be very disappointed in a vendor that had not patched years after its public disclosure, especially given the context that we're apparently not first project using RabbitMQ to be affected by these security choices by Erlang and RabbitMQ. See, for example, https://nvd.nist.gov/vuln/detail/CVE-2018-1279 https://nvd.nist.gov/vuln/detail/CVE-2018-1279, back in 2018. We definitely won't be the last, either. I hope nobody is going to argue that it's reasonable to publish server software with a "documented" RCE vulnerability, especially if you don't make any effort to highlight that detail. https://www.rabbitmq.com/access-control.html https://www.rabbitmq.com/access-control.html, the main page about access control in RabbitMQ, does not even mention this security concern; you have to read their clustering guide, which should be irrelevant to anyone not intending to use RabbitMQ clustering.

tabbott 5y ago

As a side note, I'm surprised that this specific security disclosure ended up on the Hacker News homepage, while our security release about an RCE issue affecting self-hosted Zulip servers last month did not (https://blog.zulip.com/2022/01/25/zulip-server-4-9-security-release/ https://blog.zulip.com/2022/01/25/zulip-server-4-9-security-...). The core bug there an insecure secret generation algorithm in upstream Erlang/RabbitMQ, which seems more broadly notable to me! Despite our spending many hours composing detailed reports and lobbying security teams to get this problem fixed for everyone, not just Zulip, `apt install rabbitmq-server` continues to make a Debian/Ubuntu system vulnerable to RCE, and this fact has apparently been public for years. We did manage to get `rabbitmq-server` fixed in Debian testing, and it is scheduled to be included in the next Debian stable release, I would really have thought an RCE issue like this would result in an immediate security advisory. As a final remark, if you self-host Zulip, please subscribe to our announcements mailing list (https://groups.google.com/g/zulip-announce https://groups.google.com/g/zulip-announce) and make sure your installation is up to date, so that you benefit from the hard work our security team does to keep Zulip users safe.

Ndymium 5y ago
The blog post tries to paint the RabbitMQ cookie generation as a security issue, but the Erlang distribution cookie is NOT a security feature. Indeed, the Erlang documentation states: > "Security" here does not mean cryptographically secure, but rather security against accidental misuse, such as preventing a node from connecting to a cluster with which it is not intended to communicate. The purpose of the cookie is to guard against _accidental misuse_, such as a user connecting to the wrong node (prod vs testing?). The cookie is not meant to secure a system against unauthorised use. Any system relying on the cookie for security should already be considered broken. Thus I can see why there has not been such hurry to "fix" the issue.
- tabbott 5y ago
  That argument would hold more water if there were an additional security mechanism in place preventing `apt install rabbitmq-server` from exposing a system to a remote code execution. It seems to be that the simplest fix for this problem is to make the cookie secure, since doing so is likely effective with minimal effort. Another option would be to have clustering disabled by default. If you'd like to summarize the upstream issue as "RabbitMQ as well as every distributors of it that we've investigated allows remote execution of arbitrary code on your server through having no effective access control mechanism for the distribution port, which is open by default after installing the software.", that summary is defensible. That still sounds to me like a critical security issue. One should be very disappointed in a vendor that had not patched years after its public disclosure, especially given the context that we're apparently not first project using RabbitMQ to be affected by these security choices by Erlang and RabbitMQ. See, for example, https://nvd.nist.gov/vuln/detail/CVE-2018-1279 https://nvd.nist.gov/vuln/detail/CVE-2018-1279, back in 2018. We definitely won't be the last, either. I hope nobody is going to argue that it's reasonable to publish server software with a "documented" RCE vulnerability, especially if you don't make any effort to highlight that detail. https://www.rabbitmq.com/access-control.html https://www.rabbitmq.com/access-control.html, the main page about access control in RabbitMQ, does not even mention this security concern; you have to read their clustering guide, which should be irrelevant to anyone not intending to use RabbitMQ clustering.