4 ms·

When Twitter was still in managed hosting and basically out of space to put new equipment we started an emergency "build a datacenter" project of which I was in

When Twitter was still in managed hosting and basically out of space to put new equipment we started an emergency "build a datacenter" project of which I was initially one of the only FTE's allocated too it. My plan was simple: I don't have bodies to throw at problems (we didn't even have data center techs and our facility was still under construction) so my plan was to make something that could effectively "self deploy" a functioning data center for us. I locked myself in a really hot closet behind the cafeteria with several pieces of our soon to be purchased hardware and started coding. The whole thing ended up amazing and I am still proud of it to this day. =) Some of my primary goals where simple: No manually edited configs, ever. If you swap a mother board there should be no mac address tracking, no serial number editing, etc. If we can detect it automatically we did and updated the machine database automatically. Hardware techs could simply walk out, swap a drive, and walk away and we would track that. So we wrote a tool called vex that would export all the machines various bits and bobs. Anything we could access, serial numbers, mac addresses, etc. So once the machine was running we knew where it was, what it was doing, etc. The biggest challenge was getting it running in the first place. The plan was to use DHCP option 82 which includes port information when forwarding DHCP requests to an upstream server. Thus our dhcp server (written by an awesome coworker in like two days) was 100% stateless. It knew the IP range for the datacenter, the ip range for the rack, and the port that the request came from and could respond with a static IP every time. Swap the mother board and it wouldn't matter because the machine would still be plugged into the same port on the switch. Each machine would then request a tftp boot image from the server listed in the dhcp reply. This server was smart, it would look in the machine database to see what state the machine was in and where it needed to go "next" so to speak. Step 1) Machine has never booted before. It would go into an identify utility that would allow the installer to verify that each machine was plugged into the right ports. The tech could watch the ipmi identify lights on the machine as it activated each in the rack in order and if any where out of order network cables needed fixed. From this point forward we knew that port 3 was the 3rd machine in the rack on the prod and ipmi networks. Once identified the machine would automatically reboot. Step 2) The machine would run memtest86 to ensure that memory was valid. Sadly I didn't have time to make it automatically update with results and reboot so this step required manual verification when I left. A engineer would check IPMI and mark the test successful. Step 3) The machine would run stressapptest to push the CPU and memory to the extreme. This allowed us to ensure that they would not overheat and that the rack could pull the load needed to get all the machines to 100%. Once done it would mark itself successful and reboot. Step 4) The machine would run fio to ensure that every attached disk was in good state and fully functional. Again, it would then reboot automatically. Step 5) The machine would install our default operating system assigning hostnames and such via parameters from the database. Lastly once the machine booted into its "real os" for the first time it would setup puppet, install all the back packages, look into the machine database for its assignment and then install all the various bits ad bobs required for it to do its selected function (hadoop, web server, dhcp server, puppet server, etc) Everything was driven by the machine database, and you could mark a machine as needing burn in and it would repeat the testing steps. Super useful if you suspect a machine is bad in some way but don't have time to debug. Once running vex would export all the details of the machine as well so we could track failure rates, hardware events, etc. Best part is that the lab ran the same software as the datacenter so we could always test the various bits of this process to see what worked and what didn't. In fact I added several optional steps to do things like get performance profiles and build all kinds of cool graphs from tests on demo hardware. I ended up bootstrapping three datacenters and all it required was an "Ark" which was just a single server with puppet, dhcp, tftp and audubon on it ready to setup all the other servers in the facility. During our initial power up we had a DHCP server doing something like 100-500 requests per second sustained over the course of like a half an hour (until the memory tests started) Sorry, not sure why that memory jumped out at me but it seems you did something alike and it reminded me of how much fun it was to be involved in mine. =)

m3047 4y ago
I had to be walked down the hall several times to the lab before they'd let me do it alone. Forklifts with loads that would crush you and ruin production. It was ritual, on so many levels. Interviewing the Level 0 meat... I think this is about myth and ritual. Resist the impulse to build it to something other than the Building. There wasn't any "cloud" in Colorado in the 1990s. "People lose focus" is an anodyne when they never risked their lives to begin with.
- m3047 4y ago
  I suppose that's not very clear. Somebody built a (very special) product packaging line. The people who built it moved on. There were still engineers doing maintenance (level 1), but changes to the control system (level 2 in the Purdue Model) were almost nonexistent. Once the line was built, the meat moved in to operate it. In fact, the meat had standups: it was the first time I ever experienced a standup (and I wasn't meat, so I wasn't invited to speak unless I submitted an agenda item ahead of time). The meat knew how to operate the line (and operate it safely). Although the culture of safety and quality was everywhere, the operational practicalities had been lost at the control system level: the meat were the experts, and we asked them questions a lot. But nobody asked questions about how to run the line, because nobody was going to change that ever! So those were the kinds of questions I asked the meat and the level 1 engineers in order to construct an end-to-end simulation in the lab. It created a relatively safe shared environment where people could perform experiments and test their understanding of how things worked and appeared to work at the different levels. The level 1 engineers were a pretty easy sell and helped a lot. The meat were generally willing to answer questions, and moved in once they could see it working. I actually authored the SOP by typing up HTML docs in Notepad on a file share. HTML was a new thing for them.

m3047 4y ago

I had to be walked down the hall several times to the lab before they'd let me do it alone. Forklifts with loads that would crush you and ruin production. It was ritual, on so many levels. Interviewing the Level 0 meat... I think this is about myth and ritual. Resist the impulse to build it to something other than the Building. There wasn't any "cloud" in Colorado in the 1990s. "People lose focus" is an anodyne when they never risked their lives to begin with.

m3047 4y ago
I suppose that's not very clear. Somebody built a (very special) product packaging line. The people who built it moved on. There were still engineers doing maintenance (level 1), but changes to the control system (level 2 in the Purdue Model) were almost nonexistent. Once the line was built, the meat moved in to operate it. In fact, the meat had standups: it was the first time I ever experienced a standup (and I wasn't meat, so I wasn't invited to speak unless I submitted an agenda item ahead of time). The meat knew how to operate the line (and operate it safely). Although the culture of safety and quality was everywhere, the operational practicalities had been lost at the control system level: the meat were the experts, and we asked them questions a lot. But nobody asked questions about how to run the line, because nobody was going to change that ever! So those were the kinds of questions I asked the meat and the level 1 engineers in order to construct an end-to-end simulation in the lab. It created a relatively safe shared environment where people could perform experiments and test their understanding of how things worked and appeared to work at the different levels. The level 1 engineers were a pretty easy sell and helped a lot. The meat were generally willing to answer questions, and moved in once they could see it working. I actually authored the SOP by typing up HTML docs in Notepad on a file share. HTML was a new thing for them.