5 ms·

Harper Reed and Scott VanDenPlas did a talk / Q&A at the Chicago AWS meetup group a few months ago. My hazy recollection of the salient points of the talk: 1)

by minimax 14y ago

Harper Reed and Scott VanDenPlas did a talk / Q&A at the Chicago AWS meetup group a few months ago. My hazy recollection of the salient points of the talk: 1) This represents a heterogenous set of services where the engineers in charge of each service were more-or-less free to use whatever technology stack they were most comfortable with. 2) Use puppet for provisioning everything. (There is some subtlety Scott went into about being able to bring up a bunch of servers quickly which I have since forgotten). 3) Everything shuts down the day after the election. Not much time spent future-proofing the code when it has a well known expiration date. 4) Testing testing testing. System had serious problems and 2008 and they didn't want to repeat in 2012. 5) Engineers and sysadmins had a lot of latitude to make changes in order to respond to issues as they developed. It sounded like this is essentially Harper Reed's management style (he seems like a cool guy to work for).

ROFISH 14y ago
Kinda odd saying that it has an expiration date. Aren't the same systems required for the next election?
- tomflack 14y ago
  With 3 and a bit years of downtime to sort out the crud and weirdness between uses.
- dsl 14y ago
  Obama For America, Inc. is an actual not for profit corporation[1], and will likely be wound down before the next election. The long term election-to-election databases and such are all ran by the DNC Services Corporation. 1. http://www.corporationwiki.com/Illinois/Chicago/obama-for-america-inc/25770510.aspx http://www.corporationwiki.com/Illinois/Chicago/obama-for-am...
- zaptheimpaler 14y ago
  I was at a talk by Rayid Ghani, the chief scientist for the Obama campaign. He said that they shut down everything after the campaign simply because they can't afford to keep on all the staff full-time for the 4 years, but they don't throw away the code.
  - charlieok 14y ago
    With the billions of dollars pumped into election campaigns, it's odd that a political party wouldn't keep at least a small team on the code to continually test, update, refactor, and generally improve the quality. Fred Brooks, nine women / one month, etc.
    - thatthatis 14y ago
      They keep a small team, just not the full battalion. Source: visit to OFA offices in 2010. May have changed by now because there is no third term, but I doubt it.
    - ZachPruckowski 14y ago
      The problem is that political parties are fueled by donations, and it's hard to get people to donate for something like that. Not to mention you'd be competing with local/state/House/Senate elections with actual candidates for that fundraising money. I've donated politically in the past and even having only given small amounts, I still get emails daily and calls monthly asking me to donate to about half a dozen groups (many of which I've never donated to). Heck, wounded firefighters and cancer-striken kids have a low conversion rate for their fundraising - "money for technical systems to help a candidate to be named later" is even worse.
      - bonertruck 14y ago
        The DNC does the job of folding any successful and reusable parts from the campaign into a continuing technology stack.
      - saraid216 14y ago
        This will seem a bit OT, but it is relevant: http://www.ted.com/talks/dan_pallotta_the_way_we_think_about_charity_is_dead_wrong.html http://www.ted.com/talks/dan_pallotta_the_way_we_think_about... I'm bothered that we as a society have trouble recognizing the point and purpose of overhead.
- harper 14y ago
  The same systems are not used for the next election. Bitrot, differing requirements and a general distrust of technology all account for the political machine to rebuild each cycle. This is obviously not ideal. I hope that OFA 4.0 will release a bunch of it as open source and it will help make the next cycle easier.
scottvdp 14y ago
The subtlety you are referring to in #2 is that we moved from puppet for deployments to puppet to configure template nodes. We baked AMIs from those template nodes and deployed those, drastically shortening our time from boot to serving traffic.
- miles932 14y ago
  hugs

by minimax 14y ago

ROFISH 14y ago

Kinda odd saying that it has an expiration date. Aren't the same systems required for the next election?

tomflack 14y ago
With 3 and a bit years of downtime to sort out the crud and weirdness between uses.
dsl 14y ago
Obama For America, Inc. is an actual not for profit corporation[1], and will likely be wound down before the next election. The long term election-to-election databases and such are all ran by the DNC Services Corporation. 1. http://www.corporationwiki.com/Illinois/Chicago/obama-for-america-inc/25770510.aspx http://www.corporationwiki.com/Illinois/Chicago/obama-for-am...
zaptheimpaler 14y ago
I was at a talk by Rayid Ghani, the chief scientist for the Obama campaign. He said that they shut down everything after the campaign simply because they can't afford to keep on all the staff full-time for the 4 years, but they don't throw away the code.
- charlieok 14y ago
  With the billions of dollars pumped into election campaigns, it's odd that a political party wouldn't keep at least a small team on the code to continually test, update, refactor, and generally improve the quality. Fred Brooks, nine women / one month, etc.
  - thatthatis 14y ago
    They keep a small team, just not the full battalion. Source: visit to OFA offices in 2010. May have changed by now because there is no third term, but I doubt it.
  - ZachPruckowski 14y ago
    The problem is that political parties are fueled by donations, and it's hard to get people to donate for something like that. Not to mention you'd be competing with local/state/House/Senate elections with actual candidates for that fundraising money. I've donated politically in the past and even having only given small amounts, I still get emails daily and calls monthly asking me to donate to about half a dozen groups (many of which I've never donated to). Heck, wounded firefighters and cancer-striken kids have a low conversion rate for their fundraising - "money for technical systems to help a candidate to be named later" is even worse.
    - bonertruck 14y ago
      The DNC does the job of folding any successful and reusable parts from the campaign into a continuing technology stack.
    - saraid216 14y ago
      This will seem a bit OT, but it is relevant: http://www.ted.com/talks/dan_pallotta_the_way_we_think_about_charity_is_dead_wrong.html http://www.ted.com/talks/dan_pallotta_the_way_we_think_about... I'm bothered that we as a society have trouble recognizing the point and purpose of overhead.
harper 14y ago
The same systems are not used for the next election. Bitrot, differing requirements and a general distrust of technology all account for the political machine to rebuild each cycle. This is obviously not ideal. I hope that OFA 4.0 will release a bunch of it as open source and it will help make the next cycle easier.

scottvdp 14y ago

The subtlety you are referring to in #2 is that we moved from puppet for deployments to puppet to configure template nodes. We baked AMIs from those template nodes and deployed those, drastically shortening our time from boot to serving traffic.

miles932 14y ago
hugs