6 ms·
Two Australian University Students make a better Census in 54 hours
- iamshs 10y agoWooHoo!! They hosted their website on AWS. And totally ignored privacy sensitive nature of census. Also deployed Google Analytics. And as with anything these days, added in a healthy dose of Trump. Neat work, guys. BTW, this looks like neat program name, "Bernd is studying Creative Industries and Information Technology".
- mistermann 10y agoAWS isn't capable of supporting confidential information? I'd think I'd have read about that somewhere because it seems like a fairly serious limitation.
- sixhobbits 10y agoAWS is certainly capable of supporting confidential information. Are two Junior engineers capable of producing an actually scalable, secure, fault tolerant system with adequate consistency, integrity, availability? Probably not. In a couple of days? Almost certainly not. Could it have been done in less time and for less cost than a government did it for? Almost certainly yes.
- taspeotis 10y agoI believe AWS and Azure can support government workloads. For example, Azure is part of IRAP which means the Australian Signals Directorate has signed off on them for certain types of non-classified workloads. https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL > Microsoft Azure and Microsoft Office 365 were among the first cloud services to achieve this certification for the storage and processing of unclassified (DLM) data. The certification recognizes the successful completion, review, and acceptance of a comprehensive assessment undertaken by an Information Security Registered Assessor. This certification can be leveraged by all Australian and New Zealand government agencies. > ...certification provides assurance to public sector customers in government and their partners that Microsoft has appropriate and effective security controls in place for the processing, storage, and transmission of unclassified sensitive data (the majority of government, healthcare, and education data in Australia).
- Gatsky 10y agoIt's more that if you don't have exclusive access to the hardware then you have no security. Fine for most things, but storing an entire population's details not so much.
- deleted 10y ago[deleted]
- shearnie 10y agoWe've managed to get confidential data stored in Azure for a large financial organisation, all compliances they required passed. A server is a server right? Problem could be cultural and generational. Some old school types like the idea of a server without redundancies and fire prevention and fault tolerance and high security clearance in a data center. They want it to be blinking lights at them in a cupboard on premises. They feel it is safe due to physical proximity, even though their firewalls are like cottage cheese. We're noticing that issue with health data for medical practises. Older people are suspicious of cloud. The younger generation are more savvy.
- viraptor 10y agoI know this is a popular way to look at things around here, but no. It's not about cloud / no cloud. People with sensitive data welcome VMs and cloud deployment - as long as the cloud is in a very separate datacentre with only local people handling it. Look at the actual rules and what risks they want to protect from: http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3ed4b2562bb00121564/170fd5a4b684aa3eca257f1e0021a392/$FILE/ABS%20Privacy%20Impact%20Assessment%202016%20Census.pdf http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3... Here are some of them: 2. an audited linking environment, involving staff activity being logged, monitored and, if inappropriate activity is found, investigated. Any misuse would result in immediate termination of access for the staff member, with further sanctions imposed if necessary; 3. ABS staff and in-posted officers sign legally binding Undertakings of Fidelity and Secrecy to ensure they are aware of their obligation to protect confidential information, and the consequences of disclosure (which include criminal penalties); 4. enforcement of the clear desks and clear screen policy; 5. access on a ‘need to know’ basis; How do you enforce those rules on AWS / Azure / whatever provider's staff? How do you make sure only people with appropriate clearance access the servers? 7. Vulnerability Assessments are carried out on all new IT Systems by specialised staff in IT Security trained in the field of Ethical Hacking; How do you get those providers to agree to internal pentest exercises? At that point, it's just easier to stuff some more servers in the government datacentre - completely isolated from other projects. And whether it's cloud or not is a completely separate thing - it may as well be on OpenStack. > Older people are suspicious of cloud. The younger generation are more savvy. So no, this is complete bullshit. (and so is putting this as us-vs-them - I'm part of a fairly young generation in the CS, working on cloud infrastructure, and I still would not want to see census data processed anywhere outside of "no phones in the building" government environment)
- viraptor 10y ago> AWS isn't capable of supporting confidential information? It's not a yes/no question. It depends on the information: https://aws.amazon.com/compliance/ https://aws.amazon.com/compliance/ But as far as census goes, no. I don't expect any country to be stupid enough to trust a foreign third-party with census information. Even in the US, AWS has separate zones for federal applications. This includes people who specifically can touch the data involved.
- viraptor 10y agoYou beat me to it by a minute. Yes. It would take them considerably more to add data verification, audit trail, full logging, etc. It would also kill part of that performance. And all of that needs to be done allowing for the extra data storage requirements. No AWS for you. I'm not saying their overall architecture wasn't better than what the official census had, but the solution is not comparable without knowing all the requirements.
- Hbthegreat 10y agoThe requirements of the census should be obvious to anyone using it. Not the requirements of some suits sitting around blowing money because they can't think of scalable ways to host a web form. The system has one main purpose and that is data collection. A concise list of requirements could be: - Scalable to the entire population of Australia (not less than 1 million submissions per hour) - Significant attempts to combat expected DDoS attacks (the original census site decided not to take the offered help from upstream DDoS protection services) - Data Security needs to be paramount (seems like the one thing they did right as nothing has been leaked yet) - Just because it is a government contract doesn't mean the cost is irrelevant. [$9.8m + (testing costs exceeding .5m and .5m of OFFICE plants)] These students achieved this in a caffeine filled weekend hackathon for around $500 and it achieves at least my first point as they benchmarked it at 4mil req/h. Bantering on about how it isn't an adequate solution just isn't productive here. The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours. Credit where it is due guys.
- iamshs 10y agoYou think IT guys in ABS do not know about AWS? It is about what the management thinks. And management needs code.gov like initiative desperately. Otherwise we will keep bleeding money to IBM time and time again.
- ACow_Adonis 10y agoA benchmark doesn't mean anything unless it's benchmarked with all features and requirements accounted for. I have all sympathy for the argument that you could do things better and cheaper than the government and IBM, but while banter isn't productive, neither is a benchmark of a bare hosted web form. Edit: and as another poster below me pointed out, lower downs in the ABS are well aware of AWS. Inability to get new tools and methods in was one of the reasons I left (also the philosophy that tech pay basically tops out at the APS six level, and they don't want tech experts at EL1 / EL2 levels, they want "managers". Also, the government has been pretty explicit that they don't value or respect the ABS: efficiency dividends, job cuts, left without a head for a long time, moving jobs too Geelong!?!, and low levels of pay for technical experts). Edit 2: also I just want to stare that I in no way think AWS is necessarily the right platform on which to be conducting a national census.
- skeletonjelly 10y agoYeah this is what I don't get. Yes they solved the scale issue by not having anywhere near as many questions, using plain html, drastically reduced javascript complexity, but they failed on the issue that dominated the headlines for weeks prior which was data security. Build it again using software that can be hosted in a government datacentre.
- codeka 10y agoPretty much any half decent software developer could have done what they did over a weekend. This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN.
- bootload 10y ago"This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN." Karl Stefanovic on HN, really? ~ https://news.ycombinator.com/submitted?id=codeka https://news.ycombinator.com/submitted?id=codeka
- deleted 10y ago[deleted]
- technion 10y agoIt's obvious there's a lot more to a Government project than stands out. A project I was involved with had burnt more than 54 hours just in meetings and project management before anyone wrote a line of code. But you can extrapolate a lot of the build into a more "Government approved" environment, and you still come in demonstrating that a modern type of environment can be done a lot cheaper than census, which was their goal. They've said a lot to the average non-tech about what you can achieve.
- hoodoof 10y agoNot much more was needed than an editable Adobe PDF form and a web server that accepts POST data.
- rbobby 10y ago> How would it cope with a Denial of Service attack though? “Fine” – “it would have racked up a bill, but it would have survived” Uhm... so an unbound AWS bill... that doesn't sound "fine" to me.
- bootload 10y agoA bit of context for foreign readers. Every 5 years a mandatory national census is undertaken by Australia's premier data gathering agency, the Australian Bureau of Statistics (ABS). For over 100 years the census has been completed on paper forms delivered and collected by government employees. Political interference by prior governments who cut funding and left the department without a boss for 12 months. Change of leadership, no re-funding and the newly appointed ABS leader over promises savings while implementing a new online census without a backup plan. At the same time (this is the real story) decides to collect names and addresses of every person completing the census to link these previously anonymous data gathering. New code, untested systems and an expected audience of 15 million people logging on, most likely after dinner around 1930 EST. The system experiences load, the site admins panic, make mistakes then pull the server off-line. At the same time politicians and other stake holders deride ordinary Austrians who question the name-gathering issue. As the service goes off-line and people try to enter the census, the site is down, the PM reports on twitter everything is good just before 1930. The ABS continues to auto-tweet the service is working. Australian gold medal swimmer at the Olympics derides convicted Chinese swimmer of drug use. Australian politicians blame Chinese for hacking. [0] 72 hours later the service was still down. The census requires around 95% completion to be useful. 10M AUD is spent and we have around a 50% completion. The Australian security services investigated the alleged hack and found the administration and contractor (IBM) to be at most fault. An all round cluster-f*ck. I saw this claim yesterday (https://twitter.com/peterrenshaw/status/764991292828815360 https://twitter.com/peterrenshaw/status/764991292828815360) and had a quick chat to Austin (SWE) and Adam (Organiser/AI & Neural networks) asking for them to write a summary of what they did addressing both non-technical for the benefit of very non-technical press and political wonks as well as a technical article that explains the assumptions made and addressing how the ABS may have technical reasons for not using third-party computing services. As far as I can tell this article is the general non-technical article by Adam. [0] check my twitter feed for examples of articles explaining the buildup: https://twitter.com/peterrenshaw https://twitter.com/peterrenshaw
- 6nf 10y agoGood summary. The only thing I wanted to add is that the site was load tested at 1 million submissions per hour, which if obviously not enough for the evening peak time on census night.
- foxylad 10y agoNice demonstration of smart system design, but the first item on the Australian government's specification would be the use of a private Australia-based system. The bureaucrats who wrote the spec will have limited understanding of data security, and will believe building their own local system costing an order of magnitude more than a cloud-based system must be an order of magnitude more secure. On a much smaller scale, we provide a school interview booking system used by thousands of Australian schools, built on Appengine. Sadly Queensland's state government has woken up to the fact that the data (names, email addresses and booking times - not hugely sensitive information) is stored overseas, and we're having to jump through all sort of hoops to avoid a blanket ban. Basically if your customers are not in the US, you'll be well aware of this issue.