5 ms·

The Joy of Getting Hacked

86 points by kawera 11y ago

dijit 11y ago
I suspect there will be quite a lot more stories like this now that businesses are seeing "devops" as meaning a developer that must do operations/sysadmin activities. Not everyone can be versed in everything. And not every company can afford to learn from this mistake, they must learn from others.
- iolothebard 11y ago
  This is one of 20 reasons I just left my job. There were about 4 major points of failure that will eventually hit the IT dept. The owner doesn't give a shit (just work more hours). So I told them what they were lacking, the resources they need and they don't care.
- rm_-rf_slash 11y ago
  I believe the contrary. I work in higher education, where people are valued as investments. I have been instructed extensively in security, allowing me to create more secure applications for college/devops purposes. If I lacked the knowledge and they had simply brought on a security-focused developer, it would have cost more and could have resulted in a poorer suite of apps and standards, as I could overlook security flaws even a novice would spot.
- overgard 11y ago
  "devops" really terrifies me. I think the idea is that you have less friction if you have an operations person that also understands development, or vice versa. But! You're still asking one person to do two jobs. And they really are two very distinct jobs. Are they being paid double? To me, it just seems like cheaping out on having a proper operations person, which really is a full time job. It's understandable if you're a cash strapped startup.. but lets not pretend it isn't what it is.
  - olefoo 11y ago
    It's unfortunate but predictable that a movement that started off as an attempt to find a way out of the swamp has been coopted into the service of the standard ideology of extracting as much work from labor as possible. Fixing the issues with both development and operations that result in bad, insecure software and poor user experience all the way around is a worthwhile goal. Putting the team focus on delivering complete software systems that support traceability, manageability and testability is the whole point of "devops" if the term means anything at all. It's not about running Ruby scripts as root, or containerising every last script in your environment. It's about building better systems in a manner that's more humane to all concerned.
    - dijit 11y ago
      Except now recruiters and hiring managers don not see it this way, they see it as Devlopers can do Operations tasks. Which is what I was saying, and what the parent was mentioning. DevOps by itself is a noble cause, putting developers with operations in order to smooth a pipeline, but invariably it just means 1 person with both skills/disciplines. And like they say, a jack of all trades is a master of none. (but I suppose better than a master of one?)
      - olefoo 11y ago
        It's like every other broad movement in IT. It starts with very smart people taking a look at the processes and outcomes they are using and deciding they are broken. They come up with some solutions and some tools to support their new processes and start sharing them with their peers. The new techniques have some notable successes and several members of the original group find themselves drawn into teaching/evangelizing the new methods. To make communication easier catchphrases and buzzwords intended to be a shorthand for a suite of methods become popular. Buzzword compliance becomes a checkbox feature for groups, companies and individuals each of whom have varying levels of skill and understanding and the buzzwords become diluted and more closely associated with specific tooling. At this point the original group is crowded out by people who are serial evangelists, and enterprise sales become more important than sharing knowledge with peers. Job descriptions start to lose contact with reality. The movement becomes mainstream as a grotesque caricature of itself driven mostly by the greed-fueled hype train. At which point someone looks around and declares that the processes are broken... and the whole cycle repeats. See: object-orientation, agile, scrum, devops, etc.
ZenoArrow 11y ago
Perhaps there are good reasons why this hasn't happened yet, but I'm surprised that the bulk of Wordpress-based websites don't split the editing of content from the display of content. What do I mean by this? Imagine you have two servers. Server 1 has WordPress installed, and is where you manage your website content. Server 2 does not have WordPress installed, but instead displays a static copy of the WordPress content, which is updated by a script on Server 1 whenever new content is published. What are the advantages? First of all, Server 1 can be very heavily locked down. You can keep it accessible only over a VPN or ssh tunnel, you can keep it unlisted off search engines, you don't even need to register a domain name for it. Secondly, due to the static nature of the content served on Server 2, site performance and scalability is going to be excellent, and it'd be easy to manage a cluster of "Server 2's" if a single server isn't fast enough. What are the disadvantages? You have to rely on a third party solution like Disqus if you want to have comments on the content you create. Same for shopping cart functionality. Also, you may not be able to have a Wiki. I can't think of any other disadvantages.
- detaro 11y ago
  You can get the performance benefits with cache servers, and such a workflow isn't really integrated into Wordpress (e.g. can you easily generate a list of all pages affected by a change? Or do you want to crawl the entire site for each update?). Projects that are build with this in mind are probably a better choice.
  - ZenoArrow 11y ago
    Yes, you can get the performance benefits with caching solutions like Varnish, but what about the security benefits I outlined above, how do you get those?
    - detaro 11y ago
      You don't. Still, I'm not aware of any tooling around WordPress that makes this a viable strategy for the average WordPress customer, which I guess is the answer why nobody does it.
deckar01 11y ago
The same thing happened to me last week: - Giant shared server full of old PHP projects* gets owned. - Logs show an automated tool was brute forcing a vulnerable SQL injection vector for weeks before getting through. - Backed up server for analysis and removed malicious content. - Setup new ModSecurity rules, fail2ban, and fixed vulnerable code. - Started moving stuff to Digital Ocean. * Not my server or projects, but recently my responsibility.
kmfrk 11y ago
I get less and less comfortable using my Fever RSS reader (www.feedafever.com), which was made before FireSheep and HTTPS were part of the public conversation. I don't know of any services that come close, but although I don't have anything particularly incriminating in my feed, it's still a shitty feeling to know that you're a bored script kiddie away from all that getting owned. It also reminds me of the self-hosted fad a while back (around the time Docker came around) where people still didn't want to fork over the money to get SSL for all their personal health data and whatnot.
- dexterdog 11y ago
  There have been free ssl options for a long time. Now there is letsencrypt so there is really no excuse.
zwetan 11y ago
a bit surprised by the move to digital ocean as a magic silver bullet that solve all the problems no, you just moved the problem "away" to "oh it's a virtual instance, so if anything goes wrong I can restore from backup" I don't see how this protect from being hacked ? if you run a server, not maintaining it is what make it hackable. so yeah reading the digital ocean tutorials can be a good start, like reading the ubuntu server guide https://help.ubuntu.com/lts/serverguide/ https://help.ubuntu.com/lts/serverguide/ , but it will never replace the time you invest in your server, eg. doing sysadmin. It does not have to be hard it just have to be done and on a regular basis. It's like a car, motorcycle, bicycle, etc. you need to spend the time to change the oil, check the tire pressure, and all those little things that are simple but necessary ... otherwise it get rotten with time.
- jnbiche 11y ago
  > It's like a car, motorcycle, bicycle, etc. you need to spend the time to change the oil, check the tire pressure, and all those little things that are simple but necessary ... otherwise it get rotten with time. Agreed. Or else pay significantly more for a managed WordPress server, but even those aren't immune to security issues, as we saw recently with WP Engine. The most secure option by far if you don't serve dynamic content (that would require a login, for example) is to use a static site generator and serve it via Github, S3, or Netlify. Or even your own Nginx (only slightly less secure as long as you understand Nginx and SSH and how to mitigate any potential security issues).
- deckar01 11y ago
  For me the decision to move to Digital Ocean was not expected to solve any of the underlying security issues. I chose them because they encourage better security practices through documentation and make isolating instances cheaper and easier. The previous host did not give very good advice and kept band-aiding the side effect of bad infrastructure decisions.
- 3pt14159 11y ago
  What I like about Digital Ocean is that I can separate out my "risky" servers (you know, when a client wants to use Wordpress, or when they mandate an out of date version of a library because it works with some of their existing codebase) onto $5 instances. I never put full Github keys on them and with regular backups, the damage is mitigated. When you go with a single server any one hack can take out your entire database, like what happened here.
deleted 11y ago
[deleted]

86 points by kawera 11y ago

dijit 11y ago

I suspect there will be quite a lot more stories like this now that businesses are seeing "devops" as meaning a developer that must do operations/sysadmin activities. Not everyone can be versed in everything. And not every company can afford to learn from this mistake, they must learn from others.

iolothebard 11y ago
This is one of 20 reasons I just left my job. There were about 4 major points of failure that will eventually hit the IT dept. The owner doesn't give a shit (just work more hours). So I told them what they were lacking, the resources they need and they don't care.
rm_-rf_slash 11y ago
I believe the contrary. I work in higher education, where people are valued as investments. I have been instructed extensively in security, allowing me to create more secure applications for college/devops purposes. If I lacked the knowledge and they had simply brought on a security-focused developer, it would have cost more and could have resulted in a poorer suite of apps and standards, as I could overlook security flaws even a novice would spot.
overgard 11y ago
"devops" really terrifies me. I think the idea is that you have less friction if you have an operations person that also understands development, or vice versa. But! You're still asking one person to do two jobs. And they really are two very distinct jobs. Are they being paid double? To me, it just seems like cheaping out on having a proper operations person, which really is a full time job. It's understandable if you're a cash strapped startup.. but lets not pretend it isn't what it is.
- olefoo 11y ago
  It's unfortunate but predictable that a movement that started off as an attempt to find a way out of the swamp has been coopted into the service of the standard ideology of extracting as much work from labor as possible. Fixing the issues with both development and operations that result in bad, insecure software and poor user experience all the way around is a worthwhile goal. Putting the team focus on delivering complete software systems that support traceability, manageability and testability is the whole point of "devops" if the term means anything at all. It's not about running Ruby scripts as root, or containerising every last script in your environment. It's about building better systems in a manner that's more humane to all concerned.
  - dijit 11y ago
    Except now recruiters and hiring managers don not see it this way, they see it as Devlopers can do Operations tasks. Which is what I was saying, and what the parent was mentioning. DevOps by itself is a noble cause, putting developers with operations in order to smooth a pipeline, but invariably it just means 1 person with both skills/disciplines. And like they say, a jack of all trades is a master of none. (but I suppose better than a master of one?)
    - olefoo 11y ago
      It's like every other broad movement in IT. It starts with very smart people taking a look at the processes and outcomes they are using and deciding they are broken. They come up with some solutions and some tools to support their new processes and start sharing them with their peers. The new techniques have some notable successes and several members of the original group find themselves drawn into teaching/evangelizing the new methods. To make communication easier catchphrases and buzzwords intended to be a shorthand for a suite of methods become popular. Buzzword compliance becomes a checkbox feature for groups, companies and individuals each of whom have varying levels of skill and understanding and the buzzwords become diluted and more closely associated with specific tooling. At this point the original group is crowded out by people who are serial evangelists, and enterprise sales become more important than sharing knowledge with peers. Job descriptions start to lose contact with reality. The movement becomes mainstream as a grotesque caricature of itself driven mostly by the greed-fueled hype train. At which point someone looks around and declares that the processes are broken... and the whole cycle repeats. See: object-orientation, agile, scrum, devops, etc.

ZenoArrow 11y ago

Perhaps there are good reasons why this hasn't happened yet, but I'm surprised that the bulk of Wordpress-based websites don't split the editing of content from the display of content. What do I mean by this? Imagine you have two servers. Server 1 has WordPress installed, and is where you manage your website content. Server 2 does not have WordPress installed, but instead displays a static copy of the WordPress content, which is updated by a script on Server 1 whenever new content is published. What are the advantages? First of all, Server 1 can be very heavily locked down. You can keep it accessible only over a VPN or ssh tunnel, you can keep it unlisted off search engines, you don't even need to register a domain name for it. Secondly, due to the static nature of the content served on Server 2, site performance and scalability is going to be excellent, and it'd be easy to manage a cluster of "Server 2's" if a single server isn't fast enough. What are the disadvantages? You have to rely on a third party solution like Disqus if you want to have comments on the content you create. Same for shopping cart functionality. Also, you may not be able to have a Wiki. I can't think of any other disadvantages.

detaro 11y ago
You can get the performance benefits with cache servers, and such a workflow isn't really integrated into Wordpress (e.g. can you easily generate a list of all pages affected by a change? Or do you want to crawl the entire site for each update?). Projects that are build with this in mind are probably a better choice.
- ZenoArrow 11y ago
  Yes, you can get the performance benefits with caching solutions like Varnish, but what about the security benefits I outlined above, how do you get those?
  - detaro 11y ago
    You don't. Still, I'm not aware of any tooling around WordPress that makes this a viable strategy for the average WordPress customer, which I guess is the answer why nobody does it.

deckar01 11y ago

The same thing happened to me last week: - Giant shared server full of old PHP projects* gets owned. - Logs show an automated tool was brute forcing a vulnerable SQL injection vector for weeks before getting through. - Backed up server for analysis and removed malicious content. - Setup new ModSecurity rules, fail2ban, and fixed vulnerable code. - Started moving stuff to Digital Ocean. * Not my server or projects, but recently my responsibility.

kmfrk 11y ago

I get less and less comfortable using my Fever RSS reader (www.feedafever.com), which was made before FireSheep and HTTPS were part of the public conversation. I don't know of any services that come close, but although I don't have anything particularly incriminating in my feed, it's still a shitty feeling to know that you're a bored script kiddie away from all that getting owned. It also reminds me of the self-hosted fad a while back (around the time Docker came around) where people still didn't want to fork over the money to get SSL for all their personal health data and whatnot.

dexterdog 11y ago
There have been free ssl options for a long time. Now there is letsencrypt so there is really no excuse.

zwetan 11y ago

a bit surprised by the move to digital ocean as a magic silver bullet that solve all the problems no, you just moved the problem "away" to "oh it's a virtual instance, so if anything goes wrong I can restore from backup" I don't see how this protect from being hacked ? if you run a server, not maintaining it is what make it hackable. so yeah reading the digital ocean tutorials can be a good start, like reading the ubuntu server guide https://help.ubuntu.com/lts/serverguide/ https://help.ubuntu.com/lts/serverguide/ , but it will never replace the time you invest in your server, eg. doing sysadmin. It does not have to be hard it just have to be done and on a regular basis. It's like a car, motorcycle, bicycle, etc. you need to spend the time to change the oil, check the tire pressure, and all those little things that are simple but necessary ... otherwise it get rotten with time.

jnbiche 11y ago
> It's like a car, motorcycle, bicycle, etc. you need to spend the time to change the oil, check the tire pressure, and all those little things that are simple but necessary ... otherwise it get rotten with time. Agreed. Or else pay significantly more for a managed WordPress server, but even those aren't immune to security issues, as we saw recently with WP Engine. The most secure option by far if you don't serve dynamic content (that would require a login, for example) is to use a static site generator and serve it via Github, S3, or Netlify. Or even your own Nginx (only slightly less secure as long as you understand Nginx and SSH and how to mitigate any potential security issues).
deckar01 11y ago
For me the decision to move to Digital Ocean was not expected to solve any of the underlying security issues. I chose them because they encourage better security practices through documentation and make isolating instances cheaper and easier. The previous host did not give very good advice and kept band-aiding the side effect of bad infrastructure decisions.
3pt14159 11y ago
What I like about Digital Ocean is that I can separate out my "risky" servers (you know, when a client wants to use Wordpress, or when they mandate an out of date version of a library because it works with some of their existing codebase) onto $5 instances. I never put full Github keys on them and with regular backups, the damage is mitigated. When you go with a single server any one hack can take out your entire database, like what happened here.

deleted 11y ago

[deleted]