6 ms·

There were no encrypted password dumps. No production secrets were leaked (according to the article). What's here is no more than what your average Twitch engin

by madrox 5y ago

There were no encrypted password dumps. No production secrets were leaked (according to the article). What's here is no more than what your average Twitch engineer has access to. Yes, that included payout data. Anyone with "staff" access to the site (which any employee can have) has access to any streamer's dashboard, which includes payout data. I don't think this was an attack. Based on the data so far I think it was a disgruntled engineer. Obviously if more gets leaked later I may revise that opinion.

twistedpair 5y ago
So much for information compartmentalization. Does the typical engineer need access to payment details for their daily work?
- deleted 5y ago
  [deleted]
- ABeeSea 5y ago
  Part of the appeal is working at a place like Amazon is having a voice in decision making in the product you’re building. Hard to make informed decisions or opinions without the data. Engineers in Amazon retail definitely have broad access to sales data.
- oconnor663 5y ago
  The tradeoffs for any individual piece of data are different from the tradeoffs of a company-wide policy. Siloing off one little thing (e.g. credit card info) usually doesn't inconvenience very many people, but at the same time it only provides marginal security. No front page headline has ever read "At Least The Credit Card Info Was Safe". On the other hand, a company-wide policy of siloing everything can have more of a security impact, but it also inconveniences everyone frequently. That's the tradeoff that many tech companies don't want to make.
  - zerkten 5y ago
    I don't see how this precludes just-in-time access. Even if people can re-up on their own, you can still observe the data access patterns and manage the risk. Further, when you see someone is getting blocked a lot you can improve the experience for them so they are unblocked, or have more efficient access to the data. This is just mature data and security management. Quality of life and developer experience are important topics in many ways, but should they really trump security consistently? It's always going to be dependent on people's risk assessment and comfort, but frequently it skews the wrong way because the people making the decisions know that they'll be gone.
    - myohmy 5y ago
      Implementing just-in-time access on legacy systems that pre-date just-in-time architectures is extremely expensive. Its cheaper to either give all info or no info. Which is what every legacy company does instead. My company can shut off my access to the all the databases when they stop asking me to troubleshoot any and all data issues. Which will never happen.
- nonameiguess 5y ago
  No, but doing least privilege, separation of privileges, and RBAC correctly is tedious and difficult and slows development velocity, so companies rarely do it well if they even bother trying unless some outside force compels them. I highly doubt it would be possible to do something like this at AWS, just because hosting multitenant infrastructure and working with the government forces you to implement security since you're being audited and awarded contracts on that basis. Twitch users don't give a crap about the security of the platform. They just want to monetize as quickly as they can, too. So I'm not hugely surprised that practices and culture would be different even if they have the same parent company, especially since Twitch was an acquisition. Even if not, though, I'd expect security at Prime to be better than Twitch but worse than Marketplace, Marketplace to be worse than AWS, etc. All speculation since I've never worked at any Amazon product, but that's what I would expect.
ljm 5y ago
Why would an intern at Twitch have access to data in production? Saying that no 'secrets' were leaked is effectively burying the lede.
- madrox 5y ago
  By secrets, I mean salts, password hashes, etc.
  - ihattendorf 5y ago
    > You also get access to every streamer's dashboard and their analytics I would classify that as access to production systems.
    - deleted 5y ago
      [deleted]
    - deleted 5y ago
      [deleted]
  - skilled 5y ago
    This statement makes no sense. The leak includes source code of multiple active websites and applications that are operated under the umbrella of Twitch/Amazon. Why would an intern have access to this data?
    - isbvhodnvemrwvn 5y ago
      In many companies source code for all products is available to each and single developer.
      - skilled 5y ago
        In what world does all this data in the leak would be stored together in a unified ecosystem? It makes absolutely no sense. If you're saying that Twitch runs their developer environment in a lousy manner (and you have proof of this), then please go ahead. But to imply that an intern/average developer would be given access to all this branching information is ignorant.
deleted 5y ago
[deleted]
ergerger 5y ago
I also worked for Twitch and can confirm what you're saying is true. These repo's any staff member had access to - including non-engineering staff. Revenue for the longest time was as simple as navigating to a streamers dashboard as staff, but they did finally gate that away from staff who don't need to see that info, however I am sure there are other ways to obtain revenue reporting info. I am assuming all data - including personal - has been compromised but so far, the data leaked is data that most staff would have access to in some way or another. Some may find that shocking, but this was not a "high level hack"
- madrox 5y ago
  I'm actually very happy to hear they finally added a flag for payout access. It's been years since I was there and my eyes bugged out when I saw what I had access to without needing it.
  - ogn3rd 5y ago
    Parent company was no different.
- realyashnag 5y ago
  Allegedly it also contains AWS access keys. I feel bad for the engineers who will have to answer for this.
- bengale 5y ago
  Why did non engineers have access to repos?
  - vkou 5y ago
    The better question is, why did random engineers have access to the financials of the streamers on the platform, without having to go through a break-glass, audited, emergency access escalation.
unethical_ban 5y ago
No one in IT should have access to business data. That's simply best practice. Worst case would be a database engineer who has access to backups or some prod data for troubleshooting, and even that should be under tight control with good access accounting.
- hamburglar 5y ago
  Welcome to devops. Ask Mike down the hall to add you to the “admin” group. Tell him you’re a new dev so you need everything. (This is a joke but also, at many companies, it’s not. Twitch was once small and grew. Who knows what ancient all-access switches are still critical to running the systems, marked “tech debt” in someone’s backlog)
  - unethical_ban 5y ago
    The whole point of devops is to automate everything according to best practices, so fuckups are a thing of the past! The only fuckups, of course, will be Terraform state issues.
    - syshum 5y ago
      No the whole point of devops is to get rid of those terrible sysadmins always keeping the devs from doing anything... Once you have "DevOps" the devs are ops, your head count drops, and all that pesky security and other things those dirty sysadmins wanted are gone kinda /s sometimes I think that is really want managers think about devops
    - hamburglar 5y ago
      As it turns out, the entire industry doesn't quite agree on "the whole point of devops."
- cube00 5y ago
  Until the business raises a priority one incident that their monthly reports are not looking right and you need to dive into the data to find out why some other API back end decided to present its numbers this month divided by 1000 for ease of display to their own users. I know, I know, service contacts but my point is sometimes engineers need at least temporary access to provide support at times.
popotamonga 5y ago
I worked for a multi billion company and even 6 month contractors had access to basically everything with little effort.
weaksauce 5y ago
Could have been a hack of a twitch engineer's laptop or something like that.
- bdreadz 5y ago
  This is what I thought of as well. Maybe just an engineer was hacked.
syshum 5y ago
Sounds like someone in Twitch Security needs to take a course on Least Privileged Access then

by madrox 5y ago

twistedpair 5y ago

So much for information compartmentalization. Does the typical engineer need access to payment details for their daily work?

deleted 5y ago
[deleted]
ABeeSea 5y ago
Part of the appeal is working at a place like Amazon is having a voice in decision making in the product you’re building. Hard to make informed decisions or opinions without the data. Engineers in Amazon retail definitely have broad access to sales data.
oconnor663 5y ago
The tradeoffs for any individual piece of data are different from the tradeoffs of a company-wide policy. Siloing off one little thing (e.g. credit card info) usually doesn't inconvenience very many people, but at the same time it only provides marginal security. No front page headline has ever read "At Least The Credit Card Info Was Safe". On the other hand, a company-wide policy of siloing everything can have more of a security impact, but it also inconveniences everyone frequently. That's the tradeoff that many tech companies don't want to make.
- zerkten 5y ago
  I don't see how this precludes just-in-time access. Even if people can re-up on their own, you can still observe the data access patterns and manage the risk. Further, when you see someone is getting blocked a lot you can improve the experience for them so they are unblocked, or have more efficient access to the data. This is just mature data and security management. Quality of life and developer experience are important topics in many ways, but should they really trump security consistently? It's always going to be dependent on people's risk assessment and comfort, but frequently it skews the wrong way because the people making the decisions know that they'll be gone.
  - myohmy 5y ago
    Implementing just-in-time access on legacy systems that pre-date just-in-time architectures is extremely expensive. Its cheaper to either give all info or no info. Which is what every legacy company does instead. My company can shut off my access to the all the databases when they stop asking me to troubleshoot any and all data issues. Which will never happen.
nonameiguess 5y ago
No, but doing least privilege, separation of privileges, and RBAC correctly is tedious and difficult and slows development velocity, so companies rarely do it well if they even bother trying unless some outside force compels them. I highly doubt it would be possible to do something like this at AWS, just because hosting multitenant infrastructure and working with the government forces you to implement security since you're being audited and awarded contracts on that basis. Twitch users don't give a crap about the security of the platform. They just want to monetize as quickly as they can, too. So I'm not hugely surprised that practices and culture would be different even if they have the same parent company, especially since Twitch was an acquisition. Even if not, though, I'd expect security at Prime to be better than Twitch but worse than Marketplace, Marketplace to be worse than AWS, etc. All speculation since I've never worked at any Amazon product, but that's what I would expect.

ljm 5y ago

Why would an intern at Twitch have access to data in production? Saying that no 'secrets' were leaked is effectively burying the lede.

madrox 5y ago
By secrets, I mean salts, password hashes, etc.
- ihattendorf 5y ago
  > You also get access to every streamer's dashboard and their analytics I would classify that as access to production systems.
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
- skilled 5y ago
  This statement makes no sense. The leak includes source code of multiple active websites and applications that are operated under the umbrella of Twitch/Amazon. Why would an intern have access to this data?
  - isbvhodnvemrwvn 5y ago
    In many companies source code for all products is available to each and single developer.
    - skilled 5y ago
      In what world does all this data in the leak would be stored together in a unified ecosystem? It makes absolutely no sense. If you're saying that Twitch runs their developer environment in a lousy manner (and you have proof of this), then please go ahead. But to imply that an intern/average developer would be given access to all this branching information is ignorant.

deleted 5y ago

[deleted]

ergerger 5y ago

I also worked for Twitch and can confirm what you're saying is true. These repo's any staff member had access to - including non-engineering staff. Revenue for the longest time was as simple as navigating to a streamers dashboard as staff, but they did finally gate that away from staff who don't need to see that info, however I am sure there are other ways to obtain revenue reporting info. I am assuming all data - including personal - has been compromised but so far, the data leaked is data that most staff would have access to in some way or another. Some may find that shocking, but this was not a "high level hack"

madrox 5y ago
I'm actually very happy to hear they finally added a flag for payout access. It's been years since I was there and my eyes bugged out when I saw what I had access to without needing it.
- ogn3rd 5y ago
  Parent company was no different.
realyashnag 5y ago
Allegedly it also contains AWS access keys. I feel bad for the engineers who will have to answer for this.
bengale 5y ago
Why did non engineers have access to repos?
- vkou 5y ago
  The better question is, why did random engineers have access to the financials of the streamers on the platform, without having to go through a break-glass, audited, emergency access escalation.

unethical_ban 5y ago

No one in IT should have access to business data. That's simply best practice. Worst case would be a database engineer who has access to backups or some prod data for troubleshooting, and even that should be under tight control with good access accounting.

hamburglar 5y ago
Welcome to devops. Ask Mike down the hall to add you to the “admin” group. Tell him you’re a new dev so you need everything. (This is a joke but also, at many companies, it’s not. Twitch was once small and grew. Who knows what ancient all-access switches are still critical to running the systems, marked “tech debt” in someone’s backlog)
- unethical_ban 5y ago
  The whole point of devops is to automate everything according to best practices, so fuckups are a thing of the past! The only fuckups, of course, will be Terraform state issues.
  - syshum 5y ago
    No the whole point of devops is to get rid of those terrible sysadmins always keeping the devs from doing anything... Once you have "DevOps" the devs are ops, your head count drops, and all that pesky security and other things those dirty sysadmins wanted are gone kinda /s sometimes I think that is really want managers think about devops
  - hamburglar 5y ago
    As it turns out, the entire industry doesn't quite agree on "the whole point of devops."
cube00 5y ago
Until the business raises a priority one incident that their monthly reports are not looking right and you need to dive into the data to find out why some other API back end decided to present its numbers this month divided by 1000 for ease of display to their own users. I know, I know, service contacts but my point is sometimes engineers need at least temporary access to provide support at times.

popotamonga 5y ago

I worked for a multi billion company and even 6 month contractors had access to basically everything with little effort.

weaksauce 5y ago

Could have been a hack of a twitch engineer's laptop or something like that.

bdreadz 5y ago
This is what I thought of as well. Maybe just an engineer was hacked.

syshum 5y ago

Sounds like someone in Twitch Security needs to take a course on Least Privileged Access then