13 ms·
The U.S. Air Force learned to code and saved the Pentagon millions
- Shivetya 8y agoI enjoyed that, there mere mention of Shaw AFB brought back memories of serving there in the late 80s leaving just before the first Gulf "war" broke out. anecdotal part goes here. Being in IT back then was fascinating, the secure comm side systems were Burroughs of the generation just after tube computers went out of style. Complete with paper tape booting or switches and lights. The main system which I was on was a Sperry 1100/70 using cards as the main input from on base customers. Those transitioned into images on a 360k diskette when I was leaving. The system data was mostly stored on tape (autoloading reel) and there were three disk packs including one we joked was kick start (seriously, if you swapped packs and it didn't spin you faced away with your butt on the edge and swift flat to the side hit it with your boot). It was that diskette that got me into programming. I started with Turbo C and Turbo Pascal settling on the second. The tool we were provided by Sperry would upload the card images one line at a time. On our Sperry branded PCs (same as IBM/XT) it could take longer to upload a deck than to run one through a card reader and the personelle group had the equivalent to three boxes. I ended up digging into the emulator software, the arcane manuals we had, and whooped up something in TP that did it in a tenth of the time. From that point on it was off to the races for my desire to program.
- tabtab 8y agoI once interned at a military base that used punched card readers. Some jobs (processes) would refuse to run if any card in the deck was detected as being bad (using counts and check-sums), such as misaligned holes. Being analog, different cards often were "bad" on repeat reads. It drove everybody crazy. Experienced card people knew all kinds of tricks to reduce such a problems. I learned to respect experience.
- deleted 8y ago[deleted]
- bdavis_ 8y agoWhat a wonderful story. It all comes down to risk and reward. Would Northrup go all 'agiley' and get this done quickly? Sure if you paid them Time and Material for the work and let them off the hook for any operational failures. No-one wants to do the work for 5% profit. No-one wants to hear the USAF want compensation when your program gives them a wrong answer and you lose a plane. So, we write a spec, and all agree. The we do a design, and all agree, Then we deliver and requirements have changed. The nature of the world we live in.
- w8rbt 8y agoWhen theory meets reality, requirements change. We thought this approach would work, and our research suggested it would. Then the real world came along.
- Nexxxeh 8y agoNo plan survives first contact with the enemy?
- tabtab 8y agoThe more general version: "No plan survives first contact with reality." Somebody once also said that planning is still very useful even if the plan changes later because it helps those involved prepare for and understand the changes better when they do come.
- mlb_hn 8y agoSounds like more to do with not taking battlefield conditions/training into consideration. When push comes to shove, if you give people cool tech, unless it's easy to mindlessly pick up and use they won't use it. When I was in, I had boxes and boxes of crap that was on the books that we never used (and didn't know how to use but had to inventory), then the stuff we actually used.
- bogle 8y ago"Everyone has a plan 'till they get punched in the mouth." - Mike Tyson
- forapurpose 8y agoStories like this one should be understood not in isolation, but as one point in a pendulum system: Endpoint A: These rules are all "red tape" stopping us from getting things done. Let's just get the job done and save lots of money, time, and overhead. Endpoint B: Our systems are crap; they don't meet specifications - we don't even have specifications so we don't know what's needed; they are unreliable; they don't interoperate; there's no competition for the work, it just goes to incompetent people who have friends in the right places and rip off taxpayers; and military officers know nothing about procurement, bidding, or complex system development. This is ridiculous and unprofessional: We need specifications, competitive bidding, quality requirements, QA, and professional procurement for our $600 billion budget. .... The problem isn't the endpoints, it's the pendulum. If this is your organization, you need to get off the pendulum and make expert, intelligent decisions about trade-offs. EDIT: Some clarifying edits
- tabtab 8y agoCompetent internal teams can produce relatively simple solutions within a reasonable budget. However, it's hard to formalize and replicate such teams and/or team habits. That's why the more formal specification & contract approach is often used instead: It's a BIT more predictable. The first approach may get a grade from A to F, while the second ranges mostly from B- to D-. Predictable semi-crap is often easier to accept than something that might easily be an A, F, and all between. The first's probability curve is mostly flat from A to F, while the second peaks around C-, and thus it's somewhat more predictable, and that's why managers and bureaucrats often prefer it.
- forapurpose 8y ago> Competent internal teams can produce relatively simple solutions within a reasonable budget. However, it's hard to formalize and replicate such teams and/or team habits. Tangentially, that is exactly the military's central management challenge (per my limited understanding): Combat depends on replicating competence in internal teams, under extreme stress and extremely unpredictable situations, and with unpredictable turnover of key personnel, including management (they die). The U.S. military, for example, has to accomplish this across hundreds of thousands of personnel. Imagine you get a new manager the week before a major project is due, with lots of critical decisions to be made. Now imagine you get a new manager the day that person is supposed to lead you on an assault up an enemy hill, where a less-than-optimal decision might kill you.
- totaldick 8y agoThere's a few people like Will Roper (former MIT prof) and Raj Shah in the military that are introducing agile thinking into the military, which it desperately needs. Other countries are catching up and it's fast becoming the case that software is the only thing that really matters in warfare. http://www.foxnews.com/tech/2018/06/07/f-35-to-2070-air-force-says-software-will-decide-who-wins-future-wars.html http://www.foxnews.com/tech/2018/06/07/f-35-to-2070-air-forc...
- notarealaccount 8y agoI call BS on this [0] [0] https://blog.codinghorror.com/boyds-law-of-iteration/ https://blog.codinghorror.com/boyds-law-of-iteration/
- yazaddaruvala 8y agoI've heard that the salaries of government employees are capped. I'm very curious how this will play out. The Air Force can easily afford to pay Silicon Valley salaries (sure the officers who are being taught to code can be paid less), but if the Air Force is artificially limited by red-tape to offer competitive salaries, wouldn't these software developers choose to leave as soon as they've learned? The other issue is paying these officers competitive salaries, would have quite the ripple effect. Higher ranking officers would all need to get paid more, or software developers would need inflated ranks, but with some designation where rank did not determine command. I have zero insight and am just speculating. If anyone has more insight I'd love to hear it.
- nkassis 8y agoThe cap should be based on the private sector equivalent of a job. Having an absolute cap is counter productive.
- slededit 8y agoI don't think the public is quite ready for just how much developers get paid. I doubt many would believe it.
- dfjliasjg 8y agoWe can't even afford fresh college graduates. But we can afford Oracle.
- bigiain 8y agoI'm not even sure all _developers_ are quite ready for how much they (can) get paid - if they choose the most lucrative jobs. https://www.recode.net/2018/4/30/17301264/how-much-twitter-google-amazon-highest-paying-salary-tech https://www.recode.net/2018/4/30/17301264/how-much-twitter-g... Facebook's _median_ salary is apparently over $240k...
- toomanybeersies 8y ago
- danso 8y agoSpeaking of stories of people coding in the military and in other U.S. bureaucracies, these came to mind: How I got a medal from the Army for writing code (2014): https://news.ycombinator.com/item?id=7950190 https://news.ycombinator.com/item?id=7950190 How learning to code kept me sane when I was a diplomat (2014): https://news.ycombinator.com/item?id=8810589 https://news.ycombinator.com/item?id=8810589
- black6 8y agoThe Army has Jyn Erso. The Air Force has the Kessel Run Experimentation Lab. Is Disney sponsoring the DoD now?
- jacques_chester 8y agoA few months ago some Kessel Run folks gave (really fantastic) presentations[0][1] at CF Summit which are worth watching for deeper dives on the experience. Disclosure: I work for Pivotal, though in R&D. [0] https://www.youtube.com/watch?v=V8w4jf5clJk https://www.youtube.com/watch?v=V8w4jf5clJk [1] https://www.youtube.com/watch?v=lCgxdzoJGqE https://www.youtube.com/watch?v=lCgxdzoJGqE
- maxxxxx 8y ago"“It hasn’t been peacetime in a long time,” I was told." If we don't have peace right now, how can we ever have peace? Is eternal war finally there?
- borski 8y agoThis is the current war: https://en.wikipedia.org/wiki/Operation_Inherent_Resolve https://en.wikipedia.org/wiki/Operation_Inherent_Resolve
- UncleEntity 8y agoSeems like NASA's europa-pso code is right up the alley for what the tanker scheduling needs and is already out there collecting bit-rot: https://github.com/nasa/europa https://github.com/nasa/europa Think there was a follow-on project but can't remember the name of it.
- inamberclad 8y ago> To do so, a person called “the Gonker” entered data into an Excel spreadsheet known as “the Gonkulator,” while “the Planner” arranged magnetic pucks and plastic laminated cards on the whiteboard to indicate how long planes could stay in the air. I love the names and vocab that comes out of unofficial military slang.
- AndrewKemendo 8y agoIt came from Hogans Heroes and was generally adopted by the military as a "black box" that nobody understands. In my experience, it's predominantly used in the Air Force.
- eriktrautman 8y agoIt’s a very interesting look behind the curtain at what’s wrong with the procurement and development process but also reads like a Pivotal Labs puff piece.
- trumped 8y agooverall, did the Air Force spend more or less?
- jonhendry18 8y agoSoon: "This should be contracted out, we'd save a lot of taxpayer money". Later: Contractors take over costing much more.
- fulafel 8y agoInteresting there is no discussion about Pivotal and the ethics of optimizing war machines, given that only a couple of weeks ago there was a big discussion about Google employees successfully stopping the company from working on drones.
- bogle 8y agoPossibly because the applications are qualitatively different; possibly because the software is being produced by military personnel (aided by civilian advisors).
- kerpele 8y agoFrom the article: 'In addition to the name, the team chose a hashtag for its project that is a nod to both agile software development, and the fact that Kessel Run’s culture is more startup than strictly military: #agileAF “The AF stands for Air Force,” Furtado assured me.' I wonder where the reporter's scepticism comes from...
- blahedo 8y ago> “Just because you followed the software requirements spec doesn’t mean you met the capability need,” Kroger said. Sage words. I do wonder, though: becoming more agile (or agile AF as the case may be) is one thing, but is it healthy for an actual military branch to be moving in the direction of an industry whose unofficial motto is "move fast and break things"? That's one thing when the bad outcome might be that someone gets delivered the wrong thing or gets charged twice and you have to refund them. But broken things have much graver consequences when they involve guns and bombs and airplanes.
- Arwill 8y ago> "move fast and break things" Doesn't mean that you deliver broken code, it means that you should not be lazy to rewrite old code.
- breakingcups 8y agoThat's not what that quote means. It literally means it's okay to not get it right, to make mistakes, as long as you can deliver quickly [1]. Not that weird for an industry where time to market is everything. [1] https://www.sfgate.com/news/article/Mark-Zuckerberg-Moving-Fast-And-Breaking-Things-2461393.php https://www.sfgate.com/news/article/Mark-Zuckerberg-Moving-F...
- bogle 8y agoI would have thought "move fast and break things" could almost be an Air Force motto!
- jacques_chester 8y ago> in the direction of an industry whose unofficial motto is "move fast and break things"? At Pivotal we have a different version: "go fast, forever". I believe Facebook updated theirs from "move fast and break things" to "move fast". > But broken things have much graver consequences when they involve guns and bombs and airplanes. We aren't to my knowledge participating in weapons systems or flight avionics (and I hope we never get involved in weapons systems). Our work demonstrates a principle discovered in other industries a while ago: relentless focus on improvement breaks the iron triangle. Fast, affordable, good: pick any three. But you can't do this by mouthing some slogans, skimming a book and then burping up code as fast as you can type. It takes a genuine and sustained discipline to keep to the core practices: pairing, TDD, CI/CD, IPMs, retrospectives, user-centred design, lean product management, balanced teams etc. But keep to it and the payoff in sustainable pace is remarkable. Disclosure: I work for Pivotal in R&D.
- mattlondon 8y agoWhat is most interesting about this article is the comments here considering what we've recently seen RE Google. Where are the calls from HN commenters suggesting that the engineers refuse to work on this? Where are the commenters questioning the morals and ethics of the engineers working on this? Where are the comments on here calling for the engineers working on this to quit in protest, or be deemed a collaborator? Or does that line of thought only apply when it is everyone's favourite bogeyman Google that is contracted to replace manual work with automated systems? If its not Google, then is it A-Okay to work on military systems? Both this and the Google project are IT systems that save manual, error-prone, and time-consuming work to enable the US military to more efficiently kill people. I know I'll get down-voted to hell on this, but please if you're going to try and take the moral high ground at least do so consistently.
- bogle 8y agoYou should qualify which engineers you are refering to. From the article it's clear that the majority of the coding is "by a fast-moving cadre of Air Force engineers". They are Forces personnel, not civilians. Yes, Pivotal are involved, equally it seems if they are practicing XP, perhaps now only as Agile Coaches. I also believe the applications this team are creating are substantively different in intent to those the Google employees were complaining about.
- Glide 8y agoHey, I work for Pivotal and I'm on an Kessel Run Air Force engagement. We teach things by doing them. That's why we're effective. So we are involved heavily in the making of software, we've always been. Our main schtick is that we try to make sure that we aren't necessary to the software development process when we roll off. There are actually a couple of things that are the reason why a lot of people don't think it's too much of a problem (some people do and I work out of an office that does a lot of government work so we hear more of this than others do). The main reason is that there isn't pressure for a pivot to work on a project. If the project makes you feel uncomfortable request to be on something else. Leadership has been very thoughtful in addressing how people feel about working on military and government projects (especially with the new administration). The other reason is that the applications are pretty different in intent, from what I understand, to what was complained by Google employees. It honestly feels like very standard enterprise issues. We're helping them cut through a lot of the same issues anyone would find at large enterprises.
- NiteCoder 8y agoThis story brings back a lot of memories. Right before I separated from the USAF in 1997, I was working with a tanker squadron deployed in support of operations in Bosnia. Crews would spend hours parsing the Air Tasking Orders to figure out when to take off, and with how much fuel. I had just gotten in to programming and wrote a simple VB6 app to parse the huge file and produce a simple report with the details they were after. It saved the crews hours, and I was still getting feature requests on the app for a year after I separated. It's simply amazing it took another 20 years for them to solve the problem.
- Jtsummers 8y agoThere is an, unfortunate, lack of continuity in many offices like that in the military. This is due (as you're aware, this statement is more for others) to the way that personnel get moved within the military. Officers, in particular, as the ones who are typically the "decision makers" for selecting what projects to work on will be in their job for 2-3 years at a time. If they don't have a chief or a civilian staffer who acts as a continuity officer, projects like what you did get forgotten. "Why are we still using this VB6 thing after 15 years? We should put out a contract and have it done properly." says some colonel or major who doesn't know better. And so it happens, a contract is awarded and it's a mess. I saw it more from the civilian side. Without strong continuity officers, when the military leadership changed over the direction and intent of things is forgotten. Additionally, as this article points out, the military is set on Waterfall-styled contracts and projects, given to the lowest bidders. Which is why it becomes a mess once a contract is awarded. Waterfall only works under a few circumstances, and I guarantee your lowest bidder won't satisfy them all (expertise with the implementation technology, expertise in the problem domain, expertise with how the system is actually used, ideally a project that's been done 100x before by the same team).
- coldcode 8y agoNo one commented yet on the title "... saved the Pentagon millions". Compared to 700B a year (budgeted, more is spent), millions is sort of less inspirational.
- exabrial 8y agoI had a literal Laugh out Loud at the #AgileAF (Agile Air Force) hashtag.