5 ms·

4800 Aussie sites evaporate after hack

46 points by xhtml_weaver 15y ago

ChuckMcM 15y ago
"Customers hit the Whirlpool forums to complain that Distribute.IT had not adequately responded with information about the break-in and that the hack 'has probably killed my business'." I'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on. So is there a web opportunity to add a value "over" AWS/S3 which is the equivalent of managing a strip mall property in the real world?
- elithrar 15y ago
  > I'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on. Having worked for a hosting company in Australia, I can say that most of the customers who suffer these problems are 1) small business owners and 2) not tech savvy. Most of their sites are pre-packaged e-commerce solutions, and the idea of doing a weekly DB dump + /www tar is just beyond them. They would probably be better off with something like Shopify, but many of them also want local support & like the idea of "DIY". Unfortunately for them, doing it yourself means you're handling all the risks.
  - ChuckMcM 15y ago
    Cool, thanks for the direct perspective. So it does sound like there is perhaps some space for s service which a company could provide which looked to the shop owner as a 'DIY' service but was implemented as setting up various values in standard templates on the back end? Sort of like 'App Engine' for small business where you "design" it but the modules you use are just front ends for the underlying cloud services?
    - elithrar 15y ago
      > Cool, thanks for the direct perspective. So it does sound like there is perhaps some space for s service which a company could provide which looked to the shop owner as a 'DIY' service but was implemented as setting up various values in standard templates on the back end? Sort of like 'App Engine' for small business where you "design" it but the modules you use are just front ends for the underlying cloud services? I still think many people "DIY'ing" shouldn't. The sites often look horrible, have no way to track stats (conversions), etc. What would be better is it if the hosted e-commerce players really pushed to market to these people who think DIY'ing is the better way.
      - ChuckMcM 15y ago
        "I still think many people "DIY'ing" shouldn't." Chuckle. Agreed 100% however folks want to be independent, even when they aren't as good at it as they might think they are. From the psychology perspective if you can make a system that "feels" DIY but really isn't so DIY that the final product looks DIY, that is the value proposition in this market. You sell people the notion they can "do it themselves, no need for techno-nerds telling you how you're doing it wrong!" and then you give them a design system on rails for which the 'exit' points are all decent web sites.
Dramatize 15y ago
In Australia I've found that businesses are still scared about having their websites hosted overseas.
- jacques_chester 15y ago
  Apart from FBI raids, hosting the USA is a much better option. I run a blog network (http://ozblogistan.com.au http://ozblogistan.com.au) on a pair of Linode VPSes out of their Fremont centre. I periodically reprice hosting in Australia, but basically bandwidth charges make it prohibitive to host locally for the amount of traffic I get. I'm talking $100+/mth differences. Still, independent backups are vital. I use tarsnap.
  - dorian-graph 15y ago
    What are the good Australian hosts? We have a bunch of the common shared hosting but..
    - jacques_chester 15y ago
      Sorry, I haven't looked for a while.
    - dools 15y ago
      I have VPSs with networkpresence.com.au (and we actually share rackspace at Equinix).
    - dpig 15y ago
      We have a couple of VPSs with intervolve.com.au and have found them to be good
- boyter 15y ago
  Its changing slowly. A lot of my clients are happy to host Overseas these days since the prices are a LOT more competitive (my preference being Atum in Canada). It was certainly true 5-10 years ago, and may still be true for older people, but my experience has been that you get better support overseas now anyway. The biggest issue for me has been that scheduled downtimes are usually in the middle of my day, but you can work around that.
josephcooney 15y ago
"The significant data loss has raised questions from backup experts as to why Distribute.IT did not appear to have offsite backups of customer data." Bingo
calloc 15y ago
What I find completely amazing is that business's don't have adequate backups of their websites and information and are not able to move to another hosting provider and get set back up. I have multiple copies of my own personal websites in all kinds of different locations, stored both at home, as well as on remote servers, and backed up within the cloud. Any eCommerce company needs to realise that their livelihood is at stake and that if they can't get back online within a day then maybe they are in the wrong business.
- dorian-graph 15y ago
  Unfortunately many if not most basic businesses who start operating online just hire 'some guy' who uses the bare minimum tools like Dreamweaver, templates and common CMSes, to throw together something like what the client wants/needs and then abandons them and moves on to their next victim. He never tells them about correct backup handling or much else, really.
  - elithrar 15y ago
    Yep, spot on. That, or they learn how to get a basic ZenCart or OSCommerce install running themselves. I agree that if you don't know how to get your business back online within a day or so, that you don't deserve to be running a business and that you've taken that risk on by trying to cut costs/corners elsewhere. Still, my idealising doesn't stop people from doing it.
thaumaturgy 15y ago
Just as a point of contrast between what I would consider minimum best practices, and what they were doing: We not only back up all of our hosted websites every day, on behalf of our clients, and make those backups available to our clients going back 120 days so that they can restore any files they accidentally lose on their own, but we also back up all mail accounts every hour and can restore individual mail messages if necessary. All of the backups are stored in a private location separate from all of our servers, which are scattered across the country. To have 4000 customers and not have even a whiff of that kind of architecture is, to me, completely and totally inexcusable.
- parallel 15y ago
  I don't think it's accurate to say they didn't have a whiff of this architecture. The hackers accessed and deleted their backups. If your employees can do this then you're vulnerable.
  - thaumaturgy 15y ago
    The backup system can't be accessed from the servers, and individuals can't delete backups, for exactly this reason. The backup system uses a public key ssh login to the servers and has access only to a pre-configured rsync command through sudo. The backup file system is mounted noexec to prevent any other goofiness.
  - akronim 15y ago
    Having backups that are accessible if you get hacked doesn't really cut it. I'm guessing there wasn't a huge amount of data involved here, and if there was a set of tapes in a safe somewhere a lot of people would be much happier right now. Maybe this isn't viable for budget hosting. But even the host's website is unavailable, obviously that is critical for their business and it appears it itself wasn't securely backed up. And this is hardly a totally unexpected scenario, it should have come up pretty quickly in a "what could go wrong" stage of their backup planning.
    - parallel 15y ago
mcbarry 15y ago
As an ex-developer of a company who had servers co-located with them, I'm feeling mighty smug about using our own backup system instead of their in-house one. As mentioned earlier, bandwidth in Australia is prohibitively expensive so the choice wasn't as simple as "use both".
uses 15y ago
It's absolutely terrifying that there exists in the world the type of individual who would plan and carry out such vast and permanent destruction of unique information. I can't imagine a motive that would rationalize something like this. At least hold the data for ransom or something.
kristianp 15y ago
I can imagine businesses evaluating hosting providers based on the 'professional' look of their website, which has no mention of security or backups. http://web.archive.org/web/20100525225803/http://www.distributeit.com.au/ http://web.archive.org/web/20100525225803/http://www.distrib... Does anyone know of a security auditing service that customers could look for when evaluating hosting?
jmitcheson 15y ago
The ignorance level of some of those comments is amazing (not the comments here, the ones on the article). "I am a apple fan, but you can be assured that I will NOT be using iCloud for this very reason. Always have a back up and NEVER rely on cyber space." "Moral of the storyis GET YOUR HEAD AND EVERYTHING ELSE OUT OPF THE CLOUD(S)" "Like others on here, I feel that the rush to the "latest and greatest" IT thing of the cloud has serious implications - when you are not in direct possession or responsibility of your data, and something happens to it, what do you do?" "This is why cloud computing is destined to fail. 'Cloud Shocks' like this will force people to rethink the whole concept and write cloud computing off as just a 'fad'. This isn't the first cloud shock and will not be the last." The company in the article wasn't even a cloud provider...and further more, if they were a cloud provider they would have more chance of getting their data back. Some people just don't know the difference between the internet and cloud computing..
- mambodog 15y ago
  This is an article on a mainstream newspaper's website. Despite what they might think of themselves, the reader base mainly consists of people who know rather little about how computers and the internet work.
  - neckbeard 15y ago
    The general level of reader commentary on the SMH website is pretty pathetic, no matter what the subject matter is. This isn't helped by the current trend for their articles to be link bait or scraped off a real tech source, rather than anything approximating journalism.
    - mambodog 15y ago
      I would expand that: The general level of reader commentary on newspaper websites is pretty pathetic, no matter what the subject matter is.
    - alastairpat 15y ago
      I think Fairfax as a whole is fairly lax when it comes to comment moderation. As much as I despise The Herald Sun and News Ltd., they do the whole online community thing better and the site is better for it.

46 points by xhtml_weaver 15y ago

"Customers hit the Whirlpool forums to complain that Distribute.IT had not adequately responded with information about the break-in and that the hack 'has probably killed my business'." I'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on. So is there a web opportunity to add a value "over" AWS/S3 which is the equivalent of managing a strip mall property in the real world?

elithrar 15y ago
> I'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on. Having worked for a hosting company in Australia, I can say that most of the customers who suffer these problems are 1) small business owners and 2) not tech savvy. Most of their sites are pre-packaged e-commerce solutions, and the idea of doing a weekly DB dump + /www tar is just beyond them. They would probably be better off with something like Shopify, but many of them also want local support & like the idea of "DIY". Unfortunately for them, doing it yourself means you're handling all the risks.
- ChuckMcM 15y ago
  Cool, thanks for the direct perspective. So it does sound like there is perhaps some space for s service which a company could provide which looked to the shop owner as a 'DIY' service but was implemented as setting up various values in standard templates on the back end? Sort of like 'App Engine' for small business where you "design" it but the modules you use are just front ends for the underlying cloud services?
  - elithrar 15y ago
    > Cool, thanks for the direct perspective. So it does sound like there is perhaps some space for s service which a company could provide which looked to the shop owner as a 'DIY' service but was implemented as setting up various values in standard templates on the back end? Sort of like 'App Engine' for small business where you "design" it but the modules you use are just front ends for the underlying cloud services? I still think many people "DIY'ing" shouldn't. The sites often look horrible, have no way to track stats (conversions), etc. What would be better is it if the hosted e-commerce players really pushed to market to these people who think DIY'ing is the better way.
    - ChuckMcM 15y ago
      "I still think many people "DIY'ing" shouldn't." Chuckle. Agreed 100% however folks want to be independent, even when they aren't as good at it as they might think they are. From the psychology perspective if you can make a system that "feels" DIY but really isn't so DIY that the final product looks DIY, that is the value proposition in this market. You sell people the notion they can "do it themselves, no need for techno-nerds telling you how you're doing it wrong!" and then you give them a design system on rails for which the 'exit' points are all decent web sites.

Dramatize 15y ago

In Australia I've found that businesses are still scared about having their websites hosted overseas.

jacques_chester 15y ago
Apart from FBI raids, hosting the USA is a much better option. I run a blog network (http://ozblogistan.com.au http://ozblogistan.com.au) on a pair of Linode VPSes out of their Fremont centre. I periodically reprice hosting in Australia, but basically bandwidth charges make it prohibitive to host locally for the amount of traffic I get. I'm talking $100+/mth differences. Still, independent backups are vital. I use tarsnap.
- dorian-graph 15y ago
  What are the good Australian hosts? We have a bunch of the common shared hosting but..
  - jacques_chester 15y ago
    Sorry, I haven't looked for a while.
  - dools 15y ago
    I have VPSs with networkpresence.com.au (and we actually share rackspace at Equinix).
  - dpig 15y ago
    We have a couple of VPSs with intervolve.com.au and have found them to be good
boyter 15y ago
Its changing slowly. A lot of my clients are happy to host Overseas these days since the prices are a LOT more competitive (my preference being Atum in Canada). It was certainly true 5-10 years ago, and may still be true for older people, but my experience has been that you get better support overseas now anyway. The biggest issue for me has been that scheduled downtimes are usually in the middle of my day, but you can work around that.

josephcooney 15y ago

"The significant data loss has raised questions from backup experts as to why Distribute.IT did not appear to have offsite backups of customer data." Bingo

calloc 15y ago

What I find completely amazing is that business's don't have adequate backups of their websites and information and are not able to move to another hosting provider and get set back up. I have multiple copies of my own personal websites in all kinds of different locations, stored both at home, as well as on remote servers, and backed up within the cloud. Any eCommerce company needs to realise that their livelihood is at stake and that if they can't get back online within a day then maybe they are in the wrong business.

dorian-graph 15y ago
Unfortunately many if not most basic businesses who start operating online just hire 'some guy' who uses the bare minimum tools like Dreamweaver, templates and common CMSes, to throw together something like what the client wants/needs and then abandons them and moves on to their next victim. He never tells them about correct backup handling or much else, really.
- elithrar 15y ago
  Yep, spot on. That, or they learn how to get a basic ZenCart or OSCommerce install running themselves. I agree that if you don't know how to get your business back online within a day or so, that you don't deserve to be running a business and that you've taken that risk on by trying to cut costs/corners elsewhere. Still, my idealising doesn't stop people from doing it.

thaumaturgy 15y ago

Just as a point of contrast between what I would consider minimum best practices, and what they were doing: We not only back up all of our hosted websites every day, on behalf of our clients, and make those backups available to our clients going back 120 days so that they can restore any files they accidentally lose on their own, but we also back up all mail accounts every hour and can restore individual mail messages if necessary. All of the backups are stored in a private location separate from all of our servers, which are scattered across the country. To have 4000 customers and not have even a whiff of that kind of architecture is, to me, completely and totally inexcusable.

parallel 15y ago
I don't think it's accurate to say they didn't have a whiff of this architecture. The hackers accessed and deleted their backups. If your employees can do this then you're vulnerable.
- thaumaturgy 15y ago
  The backup system can't be accessed from the servers, and individuals can't delete backups, for exactly this reason. The backup system uses a public key ssh login to the servers and has access only to a pre-configured rsync command through sudo. The backup file system is mounted noexec to prevent any other goofiness.
- akronim 15y ago
  Having backups that are accessible if you get hacked doesn't really cut it. I'm guessing there wasn't a huge amount of data involved here, and if there was a set of tapes in a safe somewhere a lot of people would be much happier right now. Maybe this isn't viable for budget hosting. But even the host's website is unavailable, obviously that is critical for their business and it appears it itself wasn't securely backed up. And this is hardly a totally unexpected scenario, it should have come up pretty quickly in a "what could go wrong" stage of their backup planning.
  - parallel 15y ago

mcbarry 15y ago

As an ex-developer of a company who had servers co-located with them, I'm feeling mighty smug about using our own backup system instead of their in-house one. As mentioned earlier, bandwidth in Australia is prohibitively expensive so the choice wasn't as simple as "use both".

uses 15y ago

It's absolutely terrifying that there exists in the world the type of individual who would plan and carry out such vast and permanent destruction of unique information. I can't imagine a motive that would rationalize something like this. At least hold the data for ransom or something.

kristianp 15y ago

I can imagine businesses evaluating hosting providers based on the 'professional' look of their website, which has no mention of security or backups. http://web.archive.org/web/20100525225803/http://www.distributeit.com.au/ http://web.archive.org/web/20100525225803/http://www.distrib... Does anyone know of a security auditing service that customers could look for when evaluating hosting?

jmitcheson 15y ago

The ignorance level of some of those comments is amazing (not the comments here, the ones on the article). "I am a apple fan, but you can be assured that I will NOT be using iCloud for this very reason. Always have a back up and NEVER rely on cyber space." "Moral of the storyis GET YOUR HEAD AND EVERYTHING ELSE OUT OPF THE CLOUD(S)" "Like others on here, I feel that the rush to the "latest and greatest" IT thing of the cloud has serious implications - when you are not in direct possession or responsibility of your data, and something happens to it, what do you do?" "This is why cloud computing is destined to fail. 'Cloud Shocks' like this will force people to rethink the whole concept and write cloud computing off as just a 'fad'. This isn't the first cloud shock and will not be the last." The company in the article wasn't even a cloud provider...and further more, if they were a cloud provider they would have more chance of getting their data back. Some people just don't know the difference between the internet and cloud computing..

mambodog 15y ago
This is an article on a mainstream newspaper's website. Despite what they might think of themselves, the reader base mainly consists of people who know rather little about how computers and the internet work.
- neckbeard 15y ago
  The general level of reader commentary on the SMH website is pretty pathetic, no matter what the subject matter is. This isn't helped by the current trend for their articles to be link bait or scraped off a real tech source, rather than anything approximating journalism.
  - mambodog 15y ago
    I would expand that: The general level of reader commentary on newspaper websites is pretty pathetic, no matter what the subject matter is.
  - alastairpat 15y ago
    I think Fairfax as a whole is fairly lax when it comes to comment moderation. As much as I despise The Herald Sun and News Ltd., they do the whole online community thing better and the site is better for it.