7 ms·
‘We feel really terrible,’ says chief executive whose app roiled the Iowa caucus
- thr483900 7y agoGeorge Soros is pissed off
- deleted 7y ago[deleted]
- mmcclure 7y agoIf someone with mod powers sees this (or OP), the title has a typo in it. It's intended to be "says CEO whose app roiled Iowa", not "rolled Iowa."
- op00to 7y agoTo “roll” is to mug or to rob someone. Seems fitting?
- borkt 7y agoAgreed, I thought it was an intentional backhanded comment to the CEO.
- AndrewBissell 7y agoA funny typo and one I wouldn't mind leaving too much, but I fixed it. :)
- pasquinelli 7y agoI've never heard that, but i've heard of being picked up by police being referred to as being rolled.
- alpb 7y agoSeriously curious, there were only 1,765 precincts. Couldn't they just create a Google Forms link share it to the field people sending the results back? It would be only 1765 lines in a Google Sheets doc, and would be practically free. (Not to mention more secure than a custom implementation by semi-incapable development contractors like in this case.)
- whalesalad 7y agoWhere is the fun in that? Gotta throw an elastic kafka beanstalk at it behind a series of microservices that serve the frontend as multiple single page apps composed into one umbrella via iframes.
- insane_dreamer 7y agoIf you don’t at least have a kubernetes cluster, you’re doing it wrong.
- outworlder 7y ago> Where is the fun in that? Gotta throw an elastic kafka beanstalk at it behind a series of microservices that serve the frontend as multiple single page apps composed into one umbrella via iframes. Had they done any of that, it might even have worked. I don't think their stack is as exciting as that.
- secabeen 7y agoThey legitimately wanted more security than that, with two-factor and other similar systems to prevent hostile actors inserting bad data.
- zamadatix 7y agoGoogle probably does 2 factor better than $smallapp. I'd probably take more than 1 form per report though (I think that falls under "similar systems" stuff you mentioned).
- 7y ago
- taurath 7y agoI really don't see a lot of accountability in any of his statements. It just seems like "well this thing went wrong, somehow, and its nobodies fault and hey nothing was lost". I find that to be absolutely abhorrant. Think about if you're one of their junior developers - you'd probably be blaming yourself for whatever bug that happened to get in. That the CEO wouldn't take any responsibility is a complete lack of leadership. To those devs, jr or otherwise - every step that was supposed to protect against a failure in the app seems to have failed here, and you should not feel bad. Mistakes happen and should be expected, but the damage that they are allowed to wreck is the blame of poor management, not you.
- alpb 7y agoEvery single executive in this little software company used to work at Clinton's 2016 campaign [1] so they have political connections. Not to mention, political parties aren't transparently run in many cases like we've seen in 2016 Democratic National Committee email leak. Most likely, these people are feeding off of their political connections and harvesting donation money from not just the party, but also the candidates [2]. This is also why I'm no longer donating to a political party directly, or a candidate who would in turn would do the same. It seems like the only independent candidate running under democratic party is keeping his campaign money in-house. [1] https://www.crn.com/slide-shows/applications-os/who-is-shadow-inc-the-company-behind-the-iowa-caucus-voting-app-/2 https://www.crn.com/slide-shows/applications-os/who-is-shado... [2] https://twitter.com/BustinTrudeau/status/1224697566182498306 https://twitter.com/BustinTrudeau/status/1224697566182498306
- zomg 7y agoi love a smug, passive-aggressive apology. just OWN up to it, acknowledge the implications of the issue you caused and provide an actual apology. unreal. “I’m really disappointed that some of our technology created an issue that made the caucus difficult,” said Gerard Niemira, chief executive of political technology company Shadow Inc., in his first interview after the caucus. “We feel really terrible about that.”
- op00to 7y agoYou can totally visualize them deeply sniffing their own farts between the lines here.
- kyleblarson 7y agoWell, take a look at whose campaign they worked for prior to this.
- Aperocky 7y agoSounds like it’s all the interns fault. Wonder how much tech this person understands
- mxcrossb 7y ago> just OWN up to it That’s how you end up getting sued.
- ggm 7y agoIts not like they can avoid being sued by not doing this. It just changes the cost point to be sued successfully, and actually denying impact and consequence probably increases their burden. Owning up, being sued but showing contrition has an upside.
- cryptozeus 7y agoBasically someone didn’t do the load testing before prod release.
- 6nf 7y agoPeople getting trained on the app last week reported manhy issues with it but the reply was 'don't worry you can always phone in the results too'
- runawaybottle 7y agoThe company leadership basically didn’t take the Iowa caucus seriously. It’s obvious to me they wanted to bang out a quick frontend mobile app with a skeleton backend (Skeleton frontend for that matter) as a proof of concept to say ‘hey we did it! Now give us all the other primary contracts’, at which point perhaps they’d invest more heavily on a more comprehensive tech team/platform. They did this with a minimal viable product approach for what should have been a critical production ready/tested app.
- yellow_postit 7y agoWasn’t even a prod release. They used a Test Flight like distribution platform to bypass the app stores because the app wasn’t ready in time to go through App Store approval processes.
- maxlybbert 7y agoThat’s what I thought originally, but there really shouldn’t have been much load (ignoring the load of downloading the app to begin with). There were fewer than 2000 precincts, and each precinct had to upload around 100 bytes of data. That shouldn’t have been hard. One news story I read quoted an error message about a “missing protocol.” I believe that the “coding error” was really a configuration error: somebody forgot to include https:// https:// in the web service URL. The app recorded the data, but couldn’t upload it. Load testing would have found this, but so would a single integration test/smoke test.
- runawaybottle 7y ago
- shadykiller 7y agoFrom the article here - https://www.bostonherald.com/2020/02/04/companies-behind-iowa-caucuses-app-failure-had-ties-to-hillary-clinton-pete-buttigieg-campaigns/ https://www.bostonherald.com/2020/02/04/companies-behind-iow... Shadow Inc. — the tech company behind the app — launched in 2019 with backing from political nonprofit ACRONYM, according to a statement from the nonprofit. ACRONYM founder and CEO Tara McGowan, a Newport, R.I., native, said on Twitter that Shadow is “an independent company ACRONYM invested in. McGowan, who lists herself as a former journalist, married then-Hillary Clinton campaign staffer and current Buttigeig strategist Michael Halle in 2015, according to a Providence Journal marriage announcement and Halle’s Twitter account. Not at all suspicious !
- goatherders 7y agoIt's not really. That's how most software projects happen. "We need XYZ app/site/solution? Hm...I worked with these guys back in 2016 and they did good work. I'll make an introduction...."
- pasquinelli 7y agoBut that first step, "we need xyz"... they didn't.
- thedance 7y agoAlso "they did good work" can't possibly be true, since Clinton 2016 was a bad campaign, with bad data, that lost.
- ncallaway 7y agoYour logic is there can be no individual person who did good work on the campaign, if the overall campaign was bad?
- thedance 7y agoWe're talking about the person who _led_ the software and data efforts for Clinton in 2016. Clinton's data was bad and misleading and stale, which contributed to the candidate taking victory tours a week before losing four states the campaign thought they would easily win. Yes, I think every high-ranking official of that campaign should be unemployable in the same field.
- iamaelephant 7y ago> The problem was caused by a bug in the code that transmits results data into the state party’s data warehouse. The amount of data we're talking about here would fit comfortably in an Excel spreadsheet. Why on Earth was there a data warehouse involved at all?
- 6nf 7y agoThey could given restricted access to a shared Google Sheet at this scale.
- runawaybottle 7y agoFrom what I read on various articles linked in this thread and others, the votes were being checked against the warehouse (who knows where the warehouse is getting data, if that’s the one true source, why even have this app). It failed to check against it properly (they were getting back an error). I guess election night was the first time they tested this validation phase , as they were not able to account for it and the app simply failed.
- glofish 7y agoCEO claims the "The app was sound and good" but there was a "transmission bug" in the code... Other way to say it, if you don't count the mistakes - then the app is excellent.
- Aperocky 7y ago> the app was sound and good, the problem was a transmission code ... This guy doesn’t know what he’s talking about is he? Is this same person who wondered why he could copy the google frontend and it would look literally the same? What use is a browser with faulty internet ‘transmission code’
- runawaybottle 7y agoYeah, like what does that mean? Their post requests had bad data from the frontend? How were they validating what they were sending?
- quocble 7y agoMade by shadow, inc. No joke
- quocble 7y agoElection software made by Shadow, inc. No joke
- itqwertz 7y agoDoes anyone have the source code or packages for the apps? I think the HN community could have a field day with this!
- AndrewBissell 7y agoVice obtained a copy of the app and has published it. They wrote an article about the conclusions from some experts: https://news.ycombinator.com/item?id=22251787 https://news.ycombinator.com/item?id=22251787
- akimball 7y agoNot open source ==> not auditable
- jiveturkey 7y agonot open source ==> not auditable by the public
- itqwertz 7y agoHere's the React Native JS Code: https://raw.githubusercontent.com/jfbguy/iowa-caucus-app-decompiled/master/resources/assets/index.android.bundle https://raw.githubusercontent.com/jfbguy/iowa-caucus-app-dec... We still need the index.android.bundle.map to faithfully deminify and audit.
- itqwertz 7y agoShadow team, if you're in the shadows, please come into the light and post this file for us!
- supernova87a 7y agoA typical non-apology. Making it sound like there were just minor subtle errors that they could never have foreseen and were caught off-guard by. They were deploying an app to a very inexperienced user group, via developer certificate enterprise sandbox because they couldn't get the app done in time for App Store review and approval. They didn't do the requisite amount of live user testing and were sending out instructions up to 3pm on the day of voting. They had a chance to evaluate whether they should take on this job and do it right or not. They could've advised and warned the IDP that they were going to get in trouble. But they thought they could handle it based on their past inconsequential apps. And they wanted the business and publicity of doing this job. They were wrong, and now playing it off like it was some minor mistake. It wasn't. Just a typical story of overly confident people getting in over their heads, and now passing it off as no big deal when in fact it is a big deal. Maybe if such behavior was fined or penalized there would be a lesson learned...
- war1025 7y agoJust saw this article from ProPublica [1] that basically implies they sent all the data over http instead of https. So much for "improved security" [1] https://www.propublica.org/article/the-iowa-caucuses-app-had-another-problem-it-could-have-been-hacked https://www.propublica.org/article/the-iowa-caucuses-app-had...
- thedance 7y agoThis guy is quoted in Vice magazine throwing shade on Vertica, a database written by a Turing Award recipient and used in many demanding roles. Then he hired some Starbucks prep cooks (literally, check LinkedIn) with no education and no industry experience and created a disaster. These events should replace whatever is currently in the literature as an example of the consequences of the Dunning-Kruger effect.
- wheelerwj 7y agoI really don't think its okay to be criticizing the job history of people. Everyone did something before they were a developer. You used to shit yourself daily, but we don't bring it up during code review. So i don't see how a previous position that lead towards being a developer is grounds for derogatory commentary.
- thedance 7y agoYou can teach anybody to do anything. It isn't the fault of the employees at this company that they ended up here. The failure in this case is the CEO's. It was their hubris to think they could found a company with no skilled labor whatsoever and produce a better solution than the legendary giants of our field.
- SideburnsOfDoom 7y agoIt's Ok to have new / inexperienced devs on the team. In fact it's good for the industry to bring them in. The problems come about when there are _nothing but_ inexperienced devs on the team.
- forkexec 7y ago"... that we got caught manipulating elections for our buddies."
- munk-a 7y ago> Niemira also confirmed Acronym’s role as majority investor in Shadow but declined to name the company’s other investors. He also declined to name the members of Shadow’s board of directors. Hey guys, you done messed up - no more lack of disclosure. Your company will probably be dissolved due to this hilariously well publicized failure. > The company’s mission is to help advance Democratic causes and candidates, and its employees were excited to have an important role in Iowa’s historic presidential caucus. That sounds noble and all, but also pretty silly - how was your solution ever going to help "advance Democratic causes and candidates" purely by serving as a drop-box for everyone to upload voter tallies to. The employees at your company can feel excited to be working to support the DNC - but it's really a stretch to claim that your vote tallying solution was advancing Democratic causes. It's like claiming, as a car manufacturer, that you're excited to help advance the cause of addressing back problems because some of the people who might buy your car end up driving to staples and buying a new office chair.
- jiveturkey 7y agoWith all the furor around election tampering, why aren't there regulations around vote counting hardware/software? Is the CMM still a thing? https://en.wikipedia.org/wiki/Capability_Maturity_Model https://en.wikipedia.org/wiki/Capability_Maturity_Model "The CMM was originally intended as a tool to evaluate the ability of government contractors to perform a contracted software project." For something as important as elections, developers (organizations) should be required to adhere to CMM 5 and use a very tight waterfall + TDD practices. Further, the product and testing requirements should be public. Not this shadow development crap.
- tengbretson 7y agoAs far as I know there is no legal basis for a vote having to happen at all, let alone be on the up-and-up for things like party primaries.
- Boxbot 7y agothe dnc and associated state organizations are not government organizations and there are no rules or regulations on whatever processes they use to select a candidate for the presidential election. they can literally do whatever the hell they want. for that matter there is no (federal level) law or regulation requiring a public vote for the president at all. the constitution establishes the electoral college but the methods by which electors are chosen and vote are left to the state themselves.
- magwa101 7y agoWhat a den of incompetence, thanks DNC. Trying to rig Iowa and failed so badly is laughable.
- craftinator 7y agoMost comments here seem to be about "what Shadow should have done", making it obvious, given the distribution of HN users, that Shadow really should have used a different systems approach. But I'd like to focus on a different aspect: they used the systems approach that they did (given budget and deadline), and the outcome was poor. Is it difficult, using their approach, to accomplish what they accomplished? Seems like a single function call is the likely culprit. What exactly caused this... A race condition, database sharding? What's the cause of this issue, at a technical level?
- lasky 7y agoI remember hearing on NPR how annoyed people were with the DNC’s lack of transparency on this app, and their reasoning was “we don’t want to let the Russian hackers know how we’re doing this”. The more this saga continues the more the democrat gang seems like a troop of out of touch hippie professors recruiting freshman to do their “technology” work.
- mercer 7y agoOr, few rungs up the ladder, it's a convenient way to get in Sanders' way while claiming incompetence!
- GoodJokes 7y agoAll you need to know is that they could have done this in any number of ways and an app was definitely the most complicated way. It is fishy.
- ramblerman 7y agoFirst line in their job application: "Shadow is a technology company dedicated to building power within the progressive movement." Can some things remain non political these days? What a weird sentiment. Are we moving to a future where I need to check the political leanings of software providers before they can agree to do business with me? I'm looking for a professional to do a job, you can keep your politics at home...
- skinnymuch 7y agoThey aren’t progressive. So caring about the statement is pointless. North Korea’s official name doesn’t mean shit.
- JohnClark1337 7y agoThey were formed specifically to aid the Democratic party in the election. I think there's a lot of former Hillary campaign people on the team. Most tech companies try to hold the appearance of being neutral.
- kraigie 7y agoBut not terrible enough to issue a refund for the taxpayers.