5 ms·

Public servants sit idle as Australian Tax Office's IT melts down

85 points by tankenmate 10y ago

tyingq 10y ago
This old press release[1] suggests the issue might be with HP StorageWorks P9500 storage arrays.[2] [1] http://www8.hp.com/us/en/hp-news/press-release.html?id=948957#.WE_8iIgrJhE http://www8.hp.com/us/en/hp-news/press-release.html?id=94895... [2]http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c02602824 http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr... Edit: A newer press release (Nov/2016), showing the HPE 3PAR deployment. If the comments below are correct, and it was 3PAR that failed, they sure weren't in for long. http://www.crn.com.au/news/ato-ripping-out-emc-storage-and-moving-to-hpe-3par-441262 http://www.crn.com.au/news/ato-ripping-out-emc-storage-and-m... "The new solution introduces many improvements, including faster processing, reduced costs and significant simplification to the storage environment." Ouch.
- Namidairo 10y ago
  Other sources are suggesting that it was a HPE 3PAR that failed.[1][2] HP are apparently calling it the first unit in the world to fail, but apparently one in Kings College London kicked the bucket too. [3] [1] http://www.itnews.com.au/news/hpe-storage-crash-killed-ato-online-services-444490 http://www.itnews.com.au/news/hpe-storage-crash-killed-ato-o... [2] https://www.hpe.com/au/en/storage/3par.html https://www.hpe.com/au/en/storage/3par.html [3] http://www.theregister.co.uk/2016/10/19/kcls_strand_data_centre_down/ http://www.theregister.co.uk/2016/10/19/kcls_strand_data_cen...
  - senectus1 10y ago
    It failed for a large mining company in WA as well... They dumped the product.
  - module0000 10y ago
    I've experienced a HPE 3PAR failure, but HP insisted we can't call it "data loss"(or make a claim as such), since the data wasn't lost/damaged. Instead it simply stopped exporting our LUNS, and the management interface became unreachable. Ie..."your data isn't lost, you just can't get to it!" would paraphrase their response to us. Ended up returning to a large keepalived+iscsi+softraid linux storage cluster. edit: typo
    - dekhn 10y ago
      Availability and durability are two distinct things. HPE maintains staff that can recover the data through extreme measures, so they were technically correct.
      - lb1lf 10y ago
        ...'recover data through extreme measures' reminded me of this little nugget from the comic strip User Friendly: http://www.userfriendly.org/cartoons/archives/97nov/uf17a199.gif http://www.userfriendly.org/cartoons/archives/97nov/uf17a199... Sometime, somewhere, someone's had to resort to this recovery method, I am sure.
        dekhn 10y ago
        When I worked at $BIGCORP we had filers from $FILER_COMPANY. $FOUNDER of $FILER_COMPANY lived in hawaii, and if the filer was bad enough, you pulled the plug, mailed the drives to hawaii, and eventually would get a tape back with the data. The founder knew the file layout and could recover the data by manually walking the metadata.
        mikestew 10y ago
        Sometime, somewhere, someone's had to resort to this recovery method, I am sure. Right before they went out of business. Years ago (like 15-20) I heard a stat about companies that suffered a catastrophic data loss (lost the drives, no usable backup). I don't recall the exact number, but something like 75% of them were out of business within 12 months.
  - Diederich 10y ago
    > HP are apparently calling it the first unit in the world to fail ... Hah...we had a catastrophic 3par failure at a company I worked for.
cyberferret 10y ago
A million gigabytes of data lost and NO taxpayer information was lost? I'll call their bluff. Still, impressive that they can restore that inside of two days (according to their estimates).
- shakna 10y ago
  If HP have actually followed the ATOs data retention policies, and if it was just two SANs going down, then theoretically this could just be a hardware failure where all the data is safe. But I doubt HP did follow the policies to the letter. I hope the ATO are incentivised by the loss of revenue to push for high penalties, but I see them paying HP to fix something that they likely could have anticipated: back pressure on the backup machine.
amarant 10y ago
Never did like HP, this article sits well with my confirmation bias...
- krona 10y ago
  I was expecting it to be about IBM. Imagine my surprise.
  - godmodus 10y ago
    right now all major hardware companies bar nvidia and AMD are in utter shite and on a shit producing roll. maybe they'll learn from this.
    - tyingq 10y ago
      I'm not sure it's their fault. Demand for "enterprisey stuff" fell dramatically once the data centers started buying mostly commodity things. High end storage doesn't have quite the margins that it used to.
sundvor 10y ago
Ah, the beauty of mission critical IT outsourcing... 1) http://www.intermedium.com.au/article/hp-enterprise-services-wins-final-ato-outsourcing-arrangement-contract http://www.intermedium.com.au/article/hp-enterprise-services... 2) https://delimiter.com.au/2012/06/08/hp-starts-cutting-aussie-staff-caspari-ascends/ https://delimiter.com.au/2012/06/08/hp-starts-cutting-aussie... 3) http://www.theage.com.au/national/public-service/thousands-of-public-servants-sit-idle-as-atos-it-melts-down-20161213-gt9xfd.html http://www.theage.com.au/national/public-service/thousands-o... This might be a bit of a cheap shot, not knowing the exact failure, however my bias is that they are related.
- godmodus 10y ago
  nah, tbh i think you're spot on. outsourcing isn't "cheaper" it's also shitty quality most of the time, which enables those cheap prices. it's pretty irresponsible and downright dumb.
  - nickff 10y ago
    I think outsourcing was a very intelligent move; the bureaucrats kept their jobs, got more money, and less responsibility.
fencepost 10y ago
"Lost" is probably headline overstatement. "Loses access to" would make much more sense and would mesh with a single point of failure. Since the article indicates that the failover system was also affected they may have triggered a bug in the storage systems or it could be something completely different like "someone let the magic smoke out of our bespoke networking equipment and we didn't realize the closest spare was disassembled in HP's network engineering lab in California."
- manicdee 10y ago
  The data was corrupted on storage. It was lost. Then when switching over to the backup hardware it got corrupted too. No taxpayer's private data was compromised though, since third parties can not compromise the data because it is corrupted.
dx034 10y ago
At least losing in this context doesn't mean that someone else has the data now.
m-p-3 10y ago
Let's all drink and hope for the best for these sysadmins hard at work on this.
warrenm 10y ago
Got to "HP" and stopped reading
unknown_apostle 10y ago
Anybody any idea on how the ATO has managed to collect petabytes? Must include audit logs and disk images etcetera, but even then...
- cmurf 10y ago
  Scanned information returns? They're many orders of magnitude larger than their declarative (e.g. PDF) or plain text equivalent.
- spangry 10y ago
  I used to work there. The scanned returns probably make up a bit, but they simply have lots of data (e.g. last 10 years of tax returns for every taxpayer, bunch of data matching DBs, enormous maze-like website, maps to buried treasure etc. etc.) And you're correct, their audit programmes all produce shitloads of data, all of which must be retained for legal record keeping (Archives Act). EDIT: come to think of it, the biggest space eater would be scans of documents sent in by folks who have been audited...
spangry 10y ago
I just pray they don't outsource their voice signature database. Imagine if someone got their hands on that...
nthcolumn 10y ago
The headline makes it sound like they are doing nothing about it but in fact it refers to the end-users.
technion 10y ago
There are several examples of major 3PAR incidents in recent firmware: http://3parug.com/viewtopic.php?f=18&t=2309 http://3parug.com/viewtopic.php?f=18&t=2309 http://3parug.com/viewtopic.php?f=18&t=1975&p=9092 http://3parug.com/viewtopic.php?f=18&t=1975&p=9092 Let me put this to you: If a "firmware update" involves copy pasting a several KB bash script into a putty session, how surprised are you to see software bugs?

85 points by tankenmate 10y ago

This old press release[1] suggests the issue might be with HP StorageWorks P9500 storage arrays.[2] [1] http://www8.hp.com/us/en/hp-news/press-release.html?id=948957#.WE_8iIgrJhE http://www8.hp.com/us/en/hp-news/press-release.html?id=94895... [2]http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c02602824 http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr... Edit: A newer press release (Nov/2016), showing the HPE 3PAR deployment. If the comments below are correct, and it was 3PAR that failed, they sure weren't in for long. http://www.crn.com.au/news/ato-ripping-out-emc-storage-and-moving-to-hpe-3par-441262 http://www.crn.com.au/news/ato-ripping-out-emc-storage-and-m... "The new solution introduces many improvements, including faster processing, reduced costs and significant simplification to the storage environment." Ouch.

Namidairo 10y ago
Other sources are suggesting that it was a HPE 3PAR that failed.[1][2] HP are apparently calling it the first unit in the world to fail, but apparently one in Kings College London kicked the bucket too. [3] [1] http://www.itnews.com.au/news/hpe-storage-crash-killed-ato-online-services-444490 http://www.itnews.com.au/news/hpe-storage-crash-killed-ato-o... [2] https://www.hpe.com/au/en/storage/3par.html https://www.hpe.com/au/en/storage/3par.html [3] http://www.theregister.co.uk/2016/10/19/kcls_strand_data_centre_down/ http://www.theregister.co.uk/2016/10/19/kcls_strand_data_cen...
- senectus1 10y ago
  It failed for a large mining company in WA as well... They dumped the product.
- module0000 10y ago
  I've experienced a HPE 3PAR failure, but HP insisted we can't call it "data loss"(or make a claim as such), since the data wasn't lost/damaged. Instead it simply stopped exporting our LUNS, and the management interface became unreachable. Ie..."your data isn't lost, you just can't get to it!" would paraphrase their response to us. Ended up returning to a large keepalived+iscsi+softraid linux storage cluster. edit: typo
  - dekhn 10y ago
    Availability and durability are two distinct things. HPE maintains staff that can recover the data through extreme measures, so they were technically correct.
    - lb1lf 10y ago
      ...'recover data through extreme measures' reminded me of this little nugget from the comic strip User Friendly: http://www.userfriendly.org/cartoons/archives/97nov/uf17a199.gif http://www.userfriendly.org/cartoons/archives/97nov/uf17a199... Sometime, somewhere, someone's had to resort to this recovery method, I am sure.
      - dekhn 10y ago
        When I worked at $BIGCORP we had filers from $FILER_COMPANY. $FOUNDER of $FILER_COMPANY lived in hawaii, and if the filer was bad enough, you pulled the plug, mailed the drives to hawaii, and eventually would get a tape back with the data. The founder knew the file layout and could recover the data by manually walking the metadata.
      - mikestew 10y ago
        Sometime, somewhere, someone's had to resort to this recovery method, I am sure. Right before they went out of business. Years ago (like 15-20) I heard a stat about companies that suffered a catastrophic data loss (lost the drives, no usable backup). I don't recall the exact number, but something like 75% of them were out of business within 12 months.
- Diederich 10y ago
  > HP are apparently calling it the first unit in the world to fail ... Hah...we had a catastrophic 3par failure at a company I worked for.

cyberferret 10y ago

A million gigabytes of data lost and NO taxpayer information was lost? I'll call their bluff. Still, impressive that they can restore that inside of two days (according to their estimates).

shakna 10y ago
If HP have actually followed the ATOs data retention policies, and if it was just two SANs going down, then theoretically this could just be a hardware failure where all the data is safe. But I doubt HP did follow the policies to the letter. I hope the ATO are incentivised by the loss of revenue to push for high penalties, but I see them paying HP to fix something that they likely could have anticipated: back pressure on the backup machine.

amarant 10y ago

Never did like HP, this article sits well with my confirmation bias...

krona 10y ago
I was expecting it to be about IBM. Imagine my surprise.
- godmodus 10y ago
  right now all major hardware companies bar nvidia and AMD are in utter shite and on a shit producing roll. maybe they'll learn from this.
  - tyingq 10y ago
    I'm not sure it's their fault. Demand for "enterprisey stuff" fell dramatically once the data centers started buying mostly commodity things. High end storage doesn't have quite the margins that it used to.

sundvor 10y ago

Ah, the beauty of mission critical IT outsourcing... 1) http://www.intermedium.com.au/article/hp-enterprise-services-wins-final-ato-outsourcing-arrangement-contract http://www.intermedium.com.au/article/hp-enterprise-services... 2) https://delimiter.com.au/2012/06/08/hp-starts-cutting-aussie-staff-caspari-ascends/ https://delimiter.com.au/2012/06/08/hp-starts-cutting-aussie... 3) http://www.theage.com.au/national/public-service/thousands-of-public-servants-sit-idle-as-atos-it-melts-down-20161213-gt9xfd.html http://www.theage.com.au/national/public-service/thousands-o... This might be a bit of a cheap shot, not knowing the exact failure, however my bias is that they are related.

godmodus 10y ago
nah, tbh i think you're spot on. outsourcing isn't "cheaper" it's also shitty quality most of the time, which enables those cheap prices. it's pretty irresponsible and downright dumb.
- nickff 10y ago
  I think outsourcing was a very intelligent move; the bureaucrats kept their jobs, got more money, and less responsibility.

fencepost 10y ago

"Lost" is probably headline overstatement. "Loses access to" would make much more sense and would mesh with a single point of failure. Since the article indicates that the failover system was also affected they may have triggered a bug in the storage systems or it could be something completely different like "someone let the magic smoke out of our bespoke networking equipment and we didn't realize the closest spare was disassembled in HP's network engineering lab in California."

manicdee 10y ago
The data was corrupted on storage. It was lost. Then when switching over to the backup hardware it got corrupted too. No taxpayer's private data was compromised though, since third parties can not compromise the data because it is corrupted.

dx034 10y ago

At least losing in this context doesn't mean that someone else has the data now.

m-p-3 10y ago

Let's all drink and hope for the best for these sysadmins hard at work on this.

warrenm 10y ago

Got to "HP" and stopped reading

unknown_apostle 10y ago

Anybody any idea on how the ATO has managed to collect petabytes? Must include audit logs and disk images etcetera, but even then...

cmurf 10y ago
Scanned information returns? They're many orders of magnitude larger than their declarative (e.g. PDF) or plain text equivalent.
spangry 10y ago
I used to work there. The scanned returns probably make up a bit, but they simply have lots of data (e.g. last 10 years of tax returns for every taxpayer, bunch of data matching DBs, enormous maze-like website, maps to buried treasure etc. etc.) And you're correct, their audit programmes all produce shitloads of data, all of which must be retained for legal record keeping (Archives Act). EDIT: come to think of it, the biggest space eater would be scans of documents sent in by folks who have been audited...

spangry 10y ago

I just pray they don't outsource their voice signature database. Imagine if someone got their hands on that...

nthcolumn 10y ago

The headline makes it sound like they are doing nothing about it but in fact it refers to the end-users.

technion 10y ago

There are several examples of major 3PAR incidents in recent firmware: http://3parug.com/viewtopic.php?f=18&t=2309 http://3parug.com/viewtopic.php?f=18&t=2309 http://3parug.com/viewtopic.php?f=18&t=1975&p=9092 http://3parug.com/viewtopic.php?f=18&t=1975&p=9092 Let me put this to you: If a "firmware update" involves copy pasting a several KB bash script into a putty session, how surprised are you to see software bugs?