7 ms·

WooHoo!! They hosted their website on AWS. And totally ignored privacy sensitive nature of census. Also deployed Google Analytics. And as with anything these da

by iamshs 10y ago

WooHoo!! They hosted their website on AWS. And totally ignored privacy sensitive nature of census. Also deployed Google Analytics. And as with anything these days, added in a healthy dose of Trump. Neat work, guys. BTW, this looks like neat program name, "Bernd is studying Creative Industries and Information Technology".

mistermann 10y ago
AWS isn't capable of supporting confidential information? I'd think I'd have read about that somewhere because it seems like a fairly serious limitation.
- sixhobbits 10y ago
  AWS is certainly capable of supporting confidential information. Are two Junior engineers capable of producing an actually scalable, secure, fault tolerant system with adequate consistency, integrity, availability? Probably not. In a couple of days? Almost certainly not. Could it have been done in less time and for less cost than a government did it for? Almost certainly yes.
- taspeotis 10y ago
  I believe AWS and Azure can support government workloads. For example, Azure is part of IRAP which means the Australian Signals Directorate has signed off on them for certain types of non-classified workloads. https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL > Microsoft Azure and Microsoft Office 365 were among the first cloud services to achieve this certification for the storage and processing of unclassified (DLM) data. The certification recognizes the successful completion, review, and acceptance of a comprehensive assessment undertaken by an Information Security Registered Assessor. This certification can be leveraged by all Australian and New Zealand government agencies. > ...certification provides assurance to public sector customers in government and their partners that Microsoft has appropriate and effective security controls in place for the processing, storage, and transmission of unclassified sensitive data (the majority of government, healthcare, and education data in Australia).
- Gatsky 10y ago
  It's more that if you don't have exclusive access to the hardware then you have no security. Fine for most things, but storing an entire population's details not so much.
- deleted 10y ago
  [deleted]
- shearnie 10y ago
  We've managed to get confidential data stored in Azure for a large financial organisation, all compliances they required passed. A server is a server right? Problem could be cultural and generational. Some old school types like the idea of a server without redundancies and fire prevention and fault tolerance and high security clearance in a data center. They want it to be blinking lights at them in a cupboard on premises. They feel it is safe due to physical proximity, even though their firewalls are like cottage cheese. We're noticing that issue with health data for medical practises. Older people are suspicious of cloud. The younger generation are more savvy.
  - viraptor 10y ago
    I know this is a popular way to look at things around here, but no. It's not about cloud / no cloud. People with sensitive data welcome VMs and cloud deployment - as long as the cloud is in a very separate datacentre with only local people handling it. Look at the actual rules and what risks they want to protect from: http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3ed4b2562bb00121564/170fd5a4b684aa3eca257f1e0021a392/$FILE/ABS%20Privacy%20Impact%20Assessment%202016%20Census.pdf http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3... Here are some of them: 2. an audited linking environment, involving staff activity being logged, monitored and, if inappropriate activity is found, investigated. Any misuse would result in immediate termination of access for the staff member, with further sanctions imposed if necessary; 3. ABS staff and in-posted officers sign legally binding Undertakings of Fidelity and Secrecy to ensure they are aware of their obligation to protect confidential information, and the consequences of disclosure (which include criminal penalties); 4. enforcement of the clear desks and clear screen policy; 5. access on a ‘need to know’ basis; How do you enforce those rules on AWS / Azure / whatever provider's staff? How do you make sure only people with appropriate clearance access the servers? 7. Vulnerability Assessments are carried out on all new IT Systems by specialised staff in IT Security trained in the field of Ethical Hacking; How do you get those providers to agree to internal pentest exercises? At that point, it's just easier to stuff some more servers in the government datacentre - completely isolated from other projects. And whether it's cloud or not is a completely separate thing - it may as well be on OpenStack. > Older people are suspicious of cloud. The younger generation are more savvy. So no, this is complete bullshit. (and so is putting this as us-vs-them - I'm part of a fairly young generation in the CS, working on cloud infrastructure, and I still would not want to see census data processed anywhere outside of "no phones in the building" government environment)
- viraptor 10y ago
  > AWS isn't capable of supporting confidential information? It's not a yes/no question. It depends on the information: https://aws.amazon.com/compliance/ https://aws.amazon.com/compliance/ But as far as census goes, no. I don't expect any country to be stupid enough to trust a foreign third-party with census information. Even in the US, AWS has separate zones for federal applications. This includes people who specifically can touch the data involved.
viraptor 10y ago
You beat me to it by a minute. Yes. It would take them considerably more to add data verification, audit trail, full logging, etc. It would also kill part of that performance. And all of that needs to be done allowing for the extra data storage requirements. No AWS for you. I'm not saying their overall architecture wasn't better than what the official census had, but the solution is not comparable without knowing all the requirements.
- Hbthegreat 10y ago
  The requirements of the census should be obvious to anyone using it. Not the requirements of some suits sitting around blowing money because they can't think of scalable ways to host a web form. The system has one main purpose and that is data collection. A concise list of requirements could be: - Scalable to the entire population of Australia (not less than 1 million submissions per hour) - Significant attempts to combat expected DDoS attacks (the original census site decided not to take the offered help from upstream DDoS protection services) - Data Security needs to be paramount (seems like the one thing they did right as nothing has been leaked yet) - Just because it is a government contract doesn't mean the cost is irrelevant. [$9.8m + (testing costs exceeding .5m and .5m of OFFICE plants)] These students achieved this in a caffeine filled weekend hackathon for around $500 and it achieves at least my first point as they benchmarked it at 4mil req/h. Bantering on about how it isn't an adequate solution just isn't productive here. The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours. Credit where it is due guys.
  - iamshs 10y ago
    You think IT guys in ABS do not know about AWS? It is about what the management thinks. And management needs code.gov like initiative desperately. Otherwise we will keep bleeding money to IBM time and time again.
  - ACow_Adonis 10y ago
    A benchmark doesn't mean anything unless it's benchmarked with all features and requirements accounted for. I have all sympathy for the argument that you could do things better and cheaper than the government and IBM, but while banter isn't productive, neither is a benchmark of a bare hosted web form. Edit: and as another poster below me pointed out, lower downs in the ABS are well aware of AWS. Inability to get new tools and methods in was one of the reasons I left (also the philosophy that tech pay basically tops out at the APS six level, and they don't want tech experts at EL1 / EL2 levels, they want "managers". Also, the government has been pretty explicit that they don't value or respect the ABS: efficiency dividends, job cuts, left without a head for a long time, moving jobs too Geelong!?!, and low levels of pay for technical experts). Edit 2: also I just want to stare that I in no way think AWS is necessarily the right platform on which to be conducting a national census.
skeletonjelly 10y ago
Yeah this is what I don't get. Yes they solved the scale issue by not having anywhere near as many questions, using plain html, drastically reduced javascript complexity, but they failed on the issue that dominated the headlines for weeks prior which was data security. Build it again using software that can be hosted in a government datacentre.
codeka 10y ago
Pretty much any half decent software developer could have done what they did over a weekend. This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN.
- bootload 10y ago
  "This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN." Karl Stefanovic on HN, really? ~ https://news.ycombinator.com/submitted?id=codeka https://news.ycombinator.com/submitted?id=codeka
deleted 10y ago
[deleted]
technion 10y ago
It's obvious there's a lot more to a Government project than stands out. A project I was involved with had burnt more than 54 hours just in meetings and project management before anyone wrote a line of code. But you can extrapolate a lot of the build into a more "Government approved" environment, and you still come in demonstrating that a modern type of environment can be done a lot cheaper than census, which was their goal. They've said a lot to the average non-tech about what you can achieve.

by iamshs 10y ago

mistermann 10y ago

AWS isn't capable of supporting confidential information? I'd think I'd have read about that somewhere because it seems like a fairly serious limitation.

sixhobbits 10y ago
AWS is certainly capable of supporting confidential information. Are two Junior engineers capable of producing an actually scalable, secure, fault tolerant system with adequate consistency, integrity, availability? Probably not. In a couple of days? Almost certainly not. Could it have been done in less time and for less cost than a government did it for? Almost certainly yes.
taspeotis 10y ago
I believe AWS and Azure can support government workloads. For example, Azure is part of IRAP which means the Australian Signals Directorate has signed off on them for certain types of non-classified workloads. https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL > Microsoft Azure and Microsoft Office 365 were among the first cloud services to achieve this certification for the storage and processing of unclassified (DLM) data. The certification recognizes the successful completion, review, and acceptance of a comprehensive assessment undertaken by an Information Security Registered Assessor. This certification can be leveraged by all Australian and New Zealand government agencies. > ...certification provides assurance to public sector customers in government and their partners that Microsoft has appropriate and effective security controls in place for the processing, storage, and transmission of unclassified sensitive data (the majority of government, healthcare, and education data in Australia).
Gatsky 10y ago
It's more that if you don't have exclusive access to the hardware then you have no security. Fine for most things, but storing an entire population's details not so much.
deleted 10y ago
[deleted]
shearnie 10y ago
We've managed to get confidential data stored in Azure for a large financial organisation, all compliances they required passed. A server is a server right? Problem could be cultural and generational. Some old school types like the idea of a server without redundancies and fire prevention and fault tolerance and high security clearance in a data center. They want it to be blinking lights at them in a cupboard on premises. They feel it is safe due to physical proximity, even though their firewalls are like cottage cheese. We're noticing that issue with health data for medical practises. Older people are suspicious of cloud. The younger generation are more savvy.
- viraptor 10y ago
  I know this is a popular way to look at things around here, but no. It's not about cloud / no cloud. People with sensitive data welcome VMs and cloud deployment - as long as the cloud is in a very separate datacentre with only local people handling it. Look at the actual rules and what risks they want to protect from: http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3ed4b2562bb00121564/170fd5a4b684aa3eca257f1e0021a392/$FILE/ABS%20Privacy%20Impact%20Assessment%202016%20Census.pdf http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3... Here are some of them: 2. an audited linking environment, involving staff activity being logged, monitored and, if inappropriate activity is found, investigated. Any misuse would result in immediate termination of access for the staff member, with further sanctions imposed if necessary; 3. ABS staff and in-posted officers sign legally binding Undertakings of Fidelity and Secrecy to ensure they are aware of their obligation to protect confidential information, and the consequences of disclosure (which include criminal penalties); 4. enforcement of the clear desks and clear screen policy; 5. access on a ‘need to know’ basis; How do you enforce those rules on AWS / Azure / whatever provider's staff? How do you make sure only people with appropriate clearance access the servers? 7. Vulnerability Assessments are carried out on all new IT Systems by specialised staff in IT Security trained in the field of Ethical Hacking; How do you get those providers to agree to internal pentest exercises? At that point, it's just easier to stuff some more servers in the government datacentre - completely isolated from other projects. And whether it's cloud or not is a completely separate thing - it may as well be on OpenStack. > Older people are suspicious of cloud. The younger generation are more savvy. So no, this is complete bullshit. (and so is putting this as us-vs-them - I'm part of a fairly young generation in the CS, working on cloud infrastructure, and I still would not want to see census data processed anywhere outside of "no phones in the building" government environment)
viraptor 10y ago
> AWS isn't capable of supporting confidential information? It's not a yes/no question. It depends on the information: https://aws.amazon.com/compliance/ https://aws.amazon.com/compliance/ But as far as census goes, no. I don't expect any country to be stupid enough to trust a foreign third-party with census information. Even in the US, AWS has separate zones for federal applications. This includes people who specifically can touch the data involved.

viraptor 10y ago

You beat me to it by a minute. Yes. It would take them considerably more to add data verification, audit trail, full logging, etc. It would also kill part of that performance. And all of that needs to be done allowing for the extra data storage requirements. No AWS for you. I'm not saying their overall architecture wasn't better than what the official census had, but the solution is not comparable without knowing all the requirements.

Hbthegreat 10y ago
The requirements of the census should be obvious to anyone using it. Not the requirements of some suits sitting around blowing money because they can't think of scalable ways to host a web form. The system has one main purpose and that is data collection. A concise list of requirements could be: - Scalable to the entire population of Australia (not less than 1 million submissions per hour) - Significant attempts to combat expected DDoS attacks (the original census site decided not to take the offered help from upstream DDoS protection services) - Data Security needs to be paramount (seems like the one thing they did right as nothing has been leaked yet) - Just because it is a government contract doesn't mean the cost is irrelevant. [$9.8m + (testing costs exceeding .5m and .5m of OFFICE plants)] These students achieved this in a caffeine filled weekend hackathon for around $500 and it achieves at least my first point as they benchmarked it at 4mil req/h. Bantering on about how it isn't an adequate solution just isn't productive here. The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours. Credit where it is due guys.
- iamshs 10y ago
  You think IT guys in ABS do not know about AWS? It is about what the management thinks. And management needs code.gov like initiative desperately. Otherwise we will keep bleeding money to IBM time and time again.
- ACow_Adonis 10y ago
  A benchmark doesn't mean anything unless it's benchmarked with all features and requirements accounted for. I have all sympathy for the argument that you could do things better and cheaper than the government and IBM, but while banter isn't productive, neither is a benchmark of a bare hosted web form. Edit: and as another poster below me pointed out, lower downs in the ABS are well aware of AWS. Inability to get new tools and methods in was one of the reasons I left (also the philosophy that tech pay basically tops out at the APS six level, and they don't want tech experts at EL1 / EL2 levels, they want "managers". Also, the government has been pretty explicit that they don't value or respect the ABS: efficiency dividends, job cuts, left without a head for a long time, moving jobs too Geelong!?!, and low levels of pay for technical experts). Edit 2: also I just want to stare that I in no way think AWS is necessarily the right platform on which to be conducting a national census.

skeletonjelly 10y ago

Yeah this is what I don't get. Yes they solved the scale issue by not having anywhere near as many questions, using plain html, drastically reduced javascript complexity, but they failed on the issue that dominated the headlines for weeks prior which was data security. Build it again using software that can be hosted in a government datacentre.

codeka 10y ago

Pretty much any half decent software developer could have done what they did over a weekend. This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN.

bootload 10y ago
"This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN." Karl Stefanovic on HN, really? ~ https://news.ycombinator.com/submitted?id=codeka https://news.ycombinator.com/submitted?id=codeka

deleted 10y ago

[deleted]

technion 10y ago

It's obvious there's a lot more to a Government project than stands out. A project I was involved with had burnt more than 54 hours just in meetings and project management before anyone wrote a line of code. But you can extrapolate a lot of the build into a more "Government approved" environment, and you still come in demonstrating that a modern type of environment can be done a lot cheaper than census, which was their goal. They've said a lot to the average non-tech about what you can achieve.