3 ms·
As always, these disclosures are an opportunity for all of us to learn something without having to pay the substantial reputational price for doing so. I'd enc
by patio11 11y ago
As always, these disclosures are an opportunity for all of us to learn something without having to pay the substantial reputational price for doing so. I'd encourage everyone on HN to read the How We Ended Up Here parts carefully and take simple remediation steps in your own projects/companies.
The unauthorized access was confirmed to have taken place ... via a debug version of our website that was visible to the public.
This is a ridiculously common way for attackers to get into your network. You both want to have an accurate inventory of the apps/boxes your organization has (highly non-trivial at scale) and you don't want dev/staging/etc to be publicly accessible. The simplest way is to firewall off traffic to them.
If you need them to be accessible for e.g. callbacks from public services, default-deny everything and control access at the network or frontend web server level before things get to your applications (which are likely going to be your vulnerable underbelly).
In most cases, the dev servers will not themselves have really interesting things on them. They will instead be used by attackers to pivot to other, more interesting servers on your network. Patreon's understanding is that the dev servers lacked SSH keys to the production servers -- that's a good thing, but after the bad guy has a dev server with local network access to interesting machines, they're going to do things like e.g. probe for exposed admin consoles to network appliances, outdated RedMine instances, internal tools, etc etc, and they will eventually own every machine that talks to any machine that your dev server can talk to. You will eventually lose the entire deployment if the bad guy wants you to.
The development server included a snapshot of our production database, which included encrypted data.
Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)
- voltagex_ 11y ago>>The development server included a snapshot of our production database, which included encrypted data. >Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.) Would you use a migration script that "scrambled" certain columns on the way back from Production to Development? That's the only way I've seen it done properly at $EMPLOYER.
- rhizome 11y agoSeeding development with production data is just lazy, unnecessary, and, as we see here, dangerous. Spend some resources on a proper seeding setup.
- voltagex_ 11y agoWhat would you suggest for such a setup? Also, I did not say seed it with production data - I mean jumble PII and then import a section of production data into dev. Sometimes this is necessary.
- thaumaturgy 11y agoA while back a fake DB data generator was posted to HN: https://news.ycombinator.com/item?id=8761953 https://news.ycombinator.com/item?id=8761953 Further back, there's also this one for Python: https://news.ycombinator.com/item?id=7102268 https://news.ycombinator.com/item?id=7102268 And this one for PHP: https://github.com/fzaninotto/Faker https://github.com/fzaninotto/Faker (there were no HN comments on that thread) And this one for Ruby: https://news.ycombinator.com/item?id=86018 https://news.ycombinator.com/item?id=86018 (also no comments) These seem like a good starting point. Also, mining bug reports for data that causes problems in your application is a lot of fun. You can turn your development database into a collection of pathological edge cases.
- patio11 11y agoWould you use a migration script that "scrambled" certain columns on the way back from Production to Development? No, because you'll almost certainly fail to fully anonymize the data and if there is anything "interesting" in your possession it will probably still be recoverable. I wrote a comment on HN long ago about this, and still stand by it. https://news.ycombinator.com/item?id=1142326 https://news.ycombinator.com/item?id=1142326
- voltagex_ 11y agoWow, thank you. It took that kind of crystal clear example for me to understand the risk clearly.
- tallanvor 11y agoI've worked for organizations where you could be disciplined for not stripping out certain data from a database before using it for testing - primarily SSN and email addresses, but it's absolutely necessary at times where using real data will allow you to identify the cause of an issue whereas troubleshooting blindly on test data could take 10 times as long.