6 ms·

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing

• yargzeblog 1y ago
  IMO the only way to avoid doing a total rebuild is to have Andre Arko: 1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime. If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.
  • ctoth 1y ago
    So you would expect people to accept that the entire root chain of custody for the Ruby supply chain is attested by ... A guy saying he didn't do anything bad? I have a cool cryptocurrency you might wanna check out that I definitely don't have a backdoor to!
  • LamaOfRuin 1y ago
    If Andre doing that was criminal, it seems quite possible that their original takeover of the github organization was also criminal? I have been waiting to hear if there would be any civil action on it since it's not at all clear they had any rights to do most of what they did.
    • tptacek 1y ago
      No, that's not at all clear. Ruby Central owns the AWS account for which Arko is (pretty clearly) being accused of changing the AWS root account password after having his access revoked. I don't think for a second Arko will be charged, but there isn't a "nuh-uh, you did this gross thing in our open source community" defense for 18 USC 1030.
      • LamaOfRuin 1y ago
        I didn't say it was clear, and I never said there was a defense. I implied that the wronged party in one case might want to be careful about raising the specter of liability or criminality.
  • SAI_Peregrinus 1y ago
    Ruby Central isn't capable of giving clemency. They could refuse to testify in any prosecution, but they don't get to pick whether a relevant attorney general or district attorney decides to prosecute.
    • dragonwriter 1y ago
      > They could refuse to testify in any prosecution Legally, they can state a preference not to testify, but they couldn't (legally) refuse to if issued a subpoena. (And, even more emphatically, they couldn't accept a good or service from the person who might be charged in exchange for not reporting the crime, or for refusing testimony.)
  • dragonwriter 1y ago
    In the US, at least, private parties can not grant immunity from prosecution for a crime (only public prosecutors of the jurisdiction against whose laws the crime was committed can do that), and they may face legal jeopardy in agreeing, or even offering, not to report a crime in exchange for some good or service of value, as that is the definition of blackmail.
  • florkbork 1y ago
    I hate to ask; but - Account created 14 hours ago. - Posts article crammed full of accusations - Has a strong well formed opinion about "it's a crime", but didn't? read the content where the subject of the accusations has.... Already disclosed they had access in both private and public. My account is also very new, because I have opted to discard my previous ones. I have used it to comment predominantly on this topic, as I sympathise with the maintainers. So in the interests of making a similar disclosure is there any chance you are affiliated with RubyCentral through a business relationship with them, their legal counsel, a marketing or PR agency or anything of that nature?
• ctoth 1y ago
  Thinking about this a bit more... it sure is interesting that around the time of a competing project launch that something just happens which might reasonably completely compromise trust in the previous incumbent, isn't it? Odd!
  • dmix 1y ago
    That was intentional according the Joel Drapper who leaked this incident, he wanted to make Ruby Central look bad https://www.reddit.com/r/ruby/comments/1o2bxol/comment/ninly3w/ https://www.reddit.com/r/ruby/comments/1o2bxol/comment/ninly... >> Why did Joel give so little time of advance notice before publishing his post revealing Andre’s production access? That struck me as irresponsible disclosure, but I may have missed something. > I decided to publish when I did because I knew that Ruby Central had been informed and I wanted the world to be informed about how sloppy Ruby Central were with security, despite their security posturing as an excuse to take over open source projects. > What I revealed changed nothing about Ruby Central’s security, since André had access whether I revealed that he did or not. When you have security information that impacts lots of people, you publish it so they can take precautions. That is responsible disclosure.
  • picadi 1y ago
    how can you trust gem.coop isn't already mining request logs + IPs to try and monetize lists of companies using specific packages & versions — besides the privacy/ethical concerns it is super useful data for hackers looking for vulnerable apps no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences (as Andre seems to have done while showing off he still had the login to rubygem's AWS root account while in Japan)
    • deleted 1y ago
      [deleted]
• akerl_ 1y ago
  Given the context of the post, it seems like "Enabled AWS CloudTrail, GuardDuty, and DataDog alerting" means "enabled alerts via CloudTrail, GuardDuty, and Datadog", not "enabled Cloudtrail logging". Otherwise the comment about reviewing Cloudtrail wouldn't make sense.
  • ctoth 1y ago
    So the attacker turns logging off (was log file validation enabled? usually isn't in Terraform ) which does not fire an alert because there is no alerting. Then does their bad stuff ... Then modifies the logs (which are in an S3 bucket on the compromised account, remember!) Then they turn logging on? The whole point is alerts go outside AWS. They go to like, your inbox or pagerduty or whatever. If they had no alerts then what use are their logs, which could have been modified? Do you think they set up cross-account logging or had enable_log_file_validation set to true?
• shevy-java 1y ago
  You are actually right, I didn't think of this before. How can they ensure that nobody else did any tampering? It seems RubyCentral did not think this through completely.
  • blibble 1y ago
    > It seems RubyCentral did not think this through completely. this is the problem when you fire all the maintainers who do anything
  • busterarm 1y ago
    You can set up EC2 instances in a way that that just having AWS root access doesn't give you ssh/console access to the instances. You can still do things like Run Command but that leaves a very obvious trail (although even this is preventable with enough effort). Also you can enable cloudtrail log validation which can ensure you know if you're looking at tampered logs or not. Really it all depends on how their accounts are set up. Unless you know the operational details you can't make a call here. I've run a multi-million dollar/year AWS Org for the last decade or so and setting things up this way is kind of brass tacks.
    • ctoth 1y ago
      So you almost certainly know that a lot of IaC tooling has terrible defaults like enable_log_file_validation being set to false. Based on the quality of their credential management and what else you can read from this blog post, how much would you wanna wager they did it right?
      • busterarm 1y ago
        I assume everyone is professional until they prove otherwise. Based on how things have been described on both sides, it actually sounds like they do a pretty good job. Oversights happen -- we're all human -- and this access was already limited to a small single-digit number of people. Given the history, it's reasonable that Arko would have had this high of a level of access and the oversight was in forgetting that when removing him. Also it's reasonable to assume that people with that access wouldn't do something criminal/malicious, and if they did, while annoying, the situation is very easily recoverable. Especially if you're using IaC tooling as you mentioned. If you're already taking the position that Ruby Central are "the bad guys" it's easy to assume that they're doing everything wrong, but that would be a mistake.
• arianvanp 1y ago
  You can't enable or disable AWS Cloud Trail as far I know? You can enable the persistent storage of trails. But you can always access 90 days of events regardless of that being enabled
  • frenchtoast8 1y ago
    This was my understanding as well, but earlier I couldn't find any documentation to prove this so I never wrote a comment. CloudTrail can be configured to save logs to S3 or CloudWatch Logs, but I think that even if you were to disable, delete, or tamper with these logs, you can still search and download unaltered logs directly from AWS using the CloudTrail Events page.
  • ctoth 1y ago
    Only management events, see https://news.ycombinator.com/item?id=45532772 https://news.ycombinator.com/item?id=45532772
• placardloop 1y ago
  CloudTrail logs for the last 90 days are enabled by default, cannot be turned off, and are immutable, even by root. If you view this “event” as starting when Arko was supposed to have their access terminated, that’s within the 90 day window and you can indeed trust the logs from that period.
  • ctoth 1y ago
    CloudTrail's 90-day immutable Event History only logs management events (IAM changes, instance launches, bucket creation). It does NOT log: * S3 object reads/writes (GetObject, PutObject) - these are "data events" requiring explicit configuration[0] * SSH/RDP to EC2 instances - CloudTrail only captures AWS API calls, not OS-level activity[1] With root access for 11 days, someone could modify gem files in S3, backdoor packages, SSH into build servers - none of it would appear in the logs they reviewed. Correct? [0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l... [1] https://repost.aws/questions/QUVsPRWwclS0KbWOYXvSla3w/cloud-trail-configuration-options-help https://repost.aws/questions/QUVsPRWwclS0KbWOYXvSla3w/cloud-...
    • placardloop 1y ago
      SSH is totally irrelevant here. Having AWS root account access doesn’t give you any ability to SSH to or otherwise access running instances. You could access data on those instances by cloning the EBS volumes or modifying build pipelines or changing network access or similar, but these would all show up in CloudTrail even without data events enabled. For S3 objects, you don’t necessarily need data events to identify if tampering happened. S3 objects are immutable as well, so if any changed you would see that reflected in the creation date and new hashes that S3 attaches as tags, which you can correlate with application logs to see if they match up or not. It’s not as simple as data logging, sure. But you’re also missing the key component here that they did not say they only just enabled CloudTrail logs, they’re saying they just now enabled CloudTrail log alerting. We don’t have any idea if data events were enabled or not, or if things like flow logs were enabled or not, or what other investigation tools they have running at the application layer. However, even if none of existed, there’s still a lot more audit-ability of events that happen in an AWS account than you’re implying, even the root account.
• coredog64 1y ago
  Not going to bother reading the article, but will chime in here that the recommendation from AWS is to have a separate security account within your organization that only holds your CloudTrail logs. This does potentially double your cost, as you only get one CloudTrail for free, and it's very useful to have an in-account trail for debugging purposes. Organizations are also useful because you can attach SCPs to your accounts that deny broad classes of activities even to the root user.
• tptacek 1y ago
  Isn't the subtext of this post pretty clearly that the unauthorized actor was Andre Arko, who had until days prior all the same access to RubyGems.org already? The impression I have reading this is that they're going out of their way to make it clear they believe it was him, but aren't naming him because doing so would be accusing him of a criminal act.
  • phoronixrly 1y ago
    The other subtext is that they literally have no idea how to run rubygems securely... And what to do in case of a security incident...
    • JoshTriplett 1y ago
      The other other subtext is that this sure is an effective distraction from their governance problems, and muddies the waters. Given the utter lack of trust I have for anything the Ruby Central folks say at this point, given the amount of spin and misinformation they've spread already, my default assumption is that this is an excuse to malign someone who may well have had legitimate access, in the process of claiming that you're locking things down, which was always the excuse being made for kicking people out.
      • hitekker 1y ago
        At this point, it looks like everyone involved, not just RubyCentral, contributed to the governance problems over many years https://archive.md/SEzoV https://archive.md/SEzoV > Regarding Arko’s blog post about his removal, McQuaid [Homebrew Maintainer] told me it’s good that Arko is crediting other people for their contribution and that he’s following open source principles of community and transparency, but that “his ‘transparency’ here has been selective to things that benefit him/his narrative, he seems unwilling or unable to admit that he failed as a leader in being unwilling or unable to introduce a formal governance process long before this all went down or appoint a meaningful successor and step down amicably.”
      • JoshTriplett 1y ago
• deleted 1y ago
  [deleted]