3 ms·

In a comment under the submission for Ruby Central's post, I said Arko changing the AWS password was an inexcusable ethical violation. This context does slight

In a comment under the submission for Ruby Central's post, I said Arko changing the AWS password was an inexcusable ethical violation. This context does slightly soften my view, especially the part about multiple 1Password accounts being in play. However there is a big thing still missing to me... Why would Arko not immediately notify RC that he had changed the password due to these concerns? If it was really a noble good faith action by the assigned on-call, giving a heads up to the remaining stakeholders would be the obligatory next step, no? According to RC's timeline, the password reset happened on September 19, but Arko did not disclose the issue to RC until September 30. From what I can tell, he has not refuted that timeline or explained the gap.

bigiain 1y ago
To me it looks like he wasn't sure who was trustworthy at the time. "The erratic and contradictory communication supplied by Marty Haught, and the complete silence from Shan and the board, made it impossible to tell exactly who had been authorized to take what actions. As this situation occurred, I was the primary on-call. My contractual, paid responsibility to Ruby Central was to defend the RubyGems.org service against potential threats. " and "Given Marty’s claims, the sudden permission deletions made no sense. Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers." and "Within a couple of days, Ruby Central made an (unsigned) public statement, and various board members agreed to talk directly to maintainers. At that point, I realized that what I thought might have been a malicious takeover was both legitimate and deliberate, and Marty would never “fix the permissions structure”, or “follow up more” as he said. Once I understood the situation, I backed off to let Ruby Central take care of their “security audit”. I left all accounts in a state where they could recover access." > According to RC's timeline, the password reset happened on September 19, but Arko did not disclose the issue to RC until September 30. The password reset happened on September 19, and "within a few days" he realised it was an intentional/malicious takeover, and he walked away knowing they had the means to recover their own access - no longer his monkeys, no longer his circus. The 30 Sep date was when he was asked by someone if he still had access, and he discovered he did, and let them know immediately. That all seems way more likely to be true and feels more plausible than anything Ruby Central has published over the last month or so...
- anon84873628 1y ago
  I realize I don't live in their world and none of this whole mishap makes sense if you're looking through the lens of a nominally functional organization. But this explanation still strains belief. 1) If you're actually concerned about phishing or some sort of hack, all the more reason to pick up the phone and hash things out. Arko knew his AWS access was revoked on the 18. You either believe that is an error and get to the bottom of it ASAP, or you step back and accept it for what it is. Resetting the root password silently the next day is like an inconsistent half measure. 2) As you quote, he had a "contractual, paid" relationship with Ruby Central. In contrast to the GitHib repos, there was never ambiguity that RC owned the AWS infrastructure. Therefore I don't buy any confusion about who the legit authority is... You talk to the person signing the checks. I understand this implication about some sort of internal coup, but the actions do not seem consistent with that being a serious belief at the time either.
  - philistine 1y ago
    Do they even have each other’s phone number?
- dismalaf 1y ago
  > To me it looks like he wasn't sure who was trustworthy at the time. It wasn't his decision to make. His consulting was terminated, he needed to hand over whatever credentials he had and wash his hands of it.
rys 1y ago
He does explain it in his blog post. He changed it after the erratic communication and actions of RC leadership, then after realising what they were really doing, left them to complete their “security audit”, assuming they’d discover it themselves and take appropriate action as part of that. That never happened (which is wild), so he let them know. They still don’t seem to be in complete control or understanding of the infrastructure they forcefully took control of.
- skywhopper 1y ago
  Also seems pretty obvious that there was no clear chain of command for the operators. The board themselves certainly aren’t deeply involved given the statement by the one board member about how they couldn’t be bothered to communicate with the community about what was happening because they are so busy in their day jobs. So who should Arko contact? The guy who’s his “boss” just suspended a bunch of access, twice, and emailed contradictory things. Given how sloppy the overall security situation clearly was and continues to be, I’m guessing no one really understands how AWS security works except for Andre anyway.
  - anon84873628 1y ago
    I appreciate these viewpoints. I still think Arko would have been better off communicating quickly and proactively to Haught any changes he made or security issues he discovered, despite however confused or contradictory Haught had been. As you say, RC is the "boss" in this relationship (they unambiguously own the AWS infrastructure and sign the consulting checks). So that is your duty as the professional in the room. And it would have at least protected his image when we now get to this point. Of course hindsight is 20/20. The whole debacle is a shame.
- james_marks 1y ago
  From Arko’s post I get the sense he actually cares about security. Seeing that he still has root, which means others may, changing root is the most benevolent thing he can do. It immediately means he has the only unauthorized access instead an unknown many, and that they’ll now cycle keys like they should have in the first place.

bigiain 1y ago

To me it looks like he wasn't sure who was trustworthy at the time. "The erratic and contradictory communication supplied by Marty Haught, and the complete silence from Shan and the board, made it impossible to tell exactly who had been authorized to take what actions. As this situation occurred, I was the primary on-call. My contractual, paid responsibility to Ruby Central was to defend the RubyGems.org service against potential threats. " and "Given Marty’s claims, the sudden permission deletions made no sense. Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers." and "Within a couple of days, Ruby Central made an (unsigned) public statement, and various board members agreed to talk directly to maintainers. At that point, I realized that what I thought might have been a malicious takeover was both legitimate and deliberate, and Marty would never “fix the permissions structure”, or “follow up more” as he said. Once I understood the situation, I backed off to let Ruby Central take care of their “security audit”. I left all accounts in a state where they could recover access." > According to RC's timeline, the password reset happened on September 19, but Arko did not disclose the issue to RC until September 30. The password reset happened on September 19, and "within a few days" he realised it was an intentional/malicious takeover, and he walked away knowing they had the means to recover their own access - no longer his monkeys, no longer his circus. The 30 Sep date was when he was asked by someone if he still had access, and he discovered he did, and let them know immediately. That all seems way more likely to be true and feels more plausible than anything Ruby Central has published over the last month or so...

anon84873628 1y ago
I realize I don't live in their world and none of this whole mishap makes sense if you're looking through the lens of a nominally functional organization. But this explanation still strains belief. 1) If you're actually concerned about phishing or some sort of hack, all the more reason to pick up the phone and hash things out. Arko knew his AWS access was revoked on the 18. You either believe that is an error and get to the bottom of it ASAP, or you step back and accept it for what it is. Resetting the root password silently the next day is like an inconsistent half measure. 2) As you quote, he had a "contractual, paid" relationship with Ruby Central. In contrast to the GitHib repos, there was never ambiguity that RC owned the AWS infrastructure. Therefore I don't buy any confusion about who the legit authority is... You talk to the person signing the checks. I understand this implication about some sort of internal coup, but the actions do not seem consistent with that being a serious belief at the time either.
- philistine 1y ago
  Do they even have each other’s phone number?
dismalaf 1y ago
> To me it looks like he wasn't sure who was trustworthy at the time. It wasn't his decision to make. His consulting was terminated, he needed to hand over whatever credentials he had and wash his hands of it.

rys 1y ago

He does explain it in his blog post. He changed it after the erratic communication and actions of RC leadership, then after realising what they were really doing, left them to complete their “security audit”, assuming they’d discover it themselves and take appropriate action as part of that. That never happened (which is wild), so he let them know. They still don’t seem to be in complete control or understanding of the infrastructure they forcefully took control of.

skywhopper 1y ago
Also seems pretty obvious that there was no clear chain of command for the operators. The board themselves certainly aren’t deeply involved given the statement by the one board member about how they couldn’t be bothered to communicate with the community about what was happening because they are so busy in their day jobs. So who should Arko contact? The guy who’s his “boss” just suspended a bunch of access, twice, and emailed contradictory things. Given how sloppy the overall security situation clearly was and continues to be, I’m guessing no one really understands how AWS security works except for Andre anyway.
- anon84873628 1y ago
  I appreciate these viewpoints. I still think Arko would have been better off communicating quickly and proactively to Haught any changes he made or security issues he discovered, despite however confused or contradictory Haught had been. As you say, RC is the "boss" in this relationship (they unambiguously own the AWS infrastructure and sign the consulting checks). So that is your duty as the professional in the room. And it would have at least protected his image when we now get to this point. Of course hindsight is 20/20. The whole debacle is a shame.
james_marks 1y ago
From Arko’s post I get the sense he actually cares about security. Seeing that he still has root, which means others may, changing root is the most benevolent thing he can do. It immediately means he has the only unauthorized access instead an unknown many, and that they’ll now cycle keys like they should have in the first place.