7 ms·
Giving away the company's secret sauce
- walrus01 9y agoISP here: bread racks and minitower PCs are still a common thing in low end hosting. In many cheap facilities the limitation is total amount of electricity and cooling available within a low cost facilities budget, so there would be no way to densely populate 44U cabinets with 1U servers anyhow.
- imeron 9y agoI guess it depends, I don't see too much competition for Backblaze. Yet, they show you everything they got and use it as marketing. Link: https://www.backblaze.com/b2/storage-pod.html https://www.backblaze.com/b2/storage-pod.html
- baud147258 9y agoBecause perhaps their "secret sauce" is not in the hardware?
- meirelles 9y agoI think she's talking about ev1servers.net. I used to be their client. Yes, it was crappy white-boxes, but it worked okay with the proper setup. Look what happened with EV1Servers: EV1servers.net > bought by ThePlanet.com > bought by SoftLayer.com > bought by IBM. The Softlayer was formed by former ThePlanet employees. I used to be a client of the Softlayer when they started too. They elevated to the next level the budget servers market, instead of crappy white boxes they used great Supermicro hardware. The secret was everything automated, very well executed by skilled and experienced people. Today, even Joe's DC is automated and most of them even use branded hardware.
- majewsky 9y ago> he Pretty sure that someone called "Rachel" is female, although I cannot find a full name or imprint on the website.
- meirelles 9y agoMy bad! Fixed. Thanks
- pdpi 9y agoShe's Rachel Kroll, a very very senior production engineer at Facebook.
- dochtman 9y agoI think she might not be at Facebook any longer? http://rachelbythebay.com/w/2018/03/10/free/ http://rachelbythebay.com/w/2018/03/10/free/ Or is your information more recent than this?
- bigiain 9y agoFrom her blog recently, I'm 99% sure that's "ex Facebook very very senior production engineer" now?
- deleted 9y ago[deleted]
- pdpi 9y agoYeah looks like. Seems like I was on holiday when she left, and i then left shortly thereafter, so I missed it
- rachelbythebay 9y agoIsh. I left. Also, "very senior"... only among the women in PE. Otherwise, not particularly. I just barely broke into the upper echelons, and that was back in 2015. I couldn't make it past that point.
- rdl 9y agoI hope you find time to do another book before your next thing.
- martinald 9y agoI remember EV1 servers well. They required root passwords in plaintext on their support forms (iirc) - otherwise they wouldn't get support - and then they got hacked with all the root passwords stolen. Friends boxes got owned with no explanation other than the password had been given out. I also believe they had one of their (or maybe the only?) fibre going into their 'DC' have a split under an interstate in Texas and having horrendous downtime.
- mbrumlow 9y agoEh, that's not true. I am sure some people did put their root passwords there, but no, passwords were only verified over phone and put into this shitty system called pipeline. They did have some fiber go down, but it mostly caused a slowdown relying on the remaining fiber. Brain (in this thread) recalls how fast they scaled to having 10s of providers to avoid having such issues again. You can reply and refute me, but I am not going to bother arguing with you any more than this -- I know how many days and hours I spent at this place, it was like home to me and I can trust that I am right on this. To be honest, this place ran super smooth and was profitable while its sister The Planet was about to fail and miss pay roll when I think it was GI partners who bought them both up and at the same time had The Planet buy the profitable Ev1Servers for tax loopholes. Yeah employed were paid shit, but they had all the opportunity in the world to better them self while having a very very very stable job. Many that I worked with there have gone on to hold 6 figure jobs in Houston elsewhere.
- bvinc 9y agoDon't forget that it was called RackShack before Ev1Servers
- PeterStuer 9y agorelated story including previous HN link: https://web.archive.org/web/20140121020351/http://www.searchenabler.com/blog/build-your-own-data-center/ https://web.archive.org/web/20140121020351/http://www.search...
- chillingeffect 9y agoA great story... it leads me to ponder the architecture and extent of "ignorance," as used in her writing. It encompasses so much... from security through digital keys to unknown quantum structures to unoptimized algorithms and the obscurity especially of human biology and sunconscious desires.
- inopinatus 9y agoI once worked with a hosting provider whose battery backup was that every server was actually a second-hand thinkpad.
- hxtk 9y agoI can't decide if that's horrifying or brilliant. Leaning towards both.
- tonyarkles 9y agoOh wow... Like my sibling commenter, I'm both horrified and... looking around the shop for spare Thinkpads :D
- holtalanm 9y agoso what you're saying is that I could turn the 10 used laptops in my basement into a business? okay....brb...reformatting them to linux.
- Max_aaa 9y agoMy home server is my 2nd gen i7 laptop, the battery can still keep it running for roughly an hour if it is not under load. The modifications done, were replacing the DVD drive with a 2.5" Driver adaptor for a raided system, and putting two new high capacity 2.5" drives in.
- imglorp 9y agoRachael's story aside, I think the idea of commodity PCs for mass storage only became feasible after the software to handle them as a system arose. The popular term was RAIN: Redundant Array of Inexpensive (independent) Nodes - when one box dies, the software spins up another and assigns some load to it. The advancement was that 5 or 10 sketchy PCs could be had for the price of 1 more reliable rackmount box, but you didn't care much about reliability. This all got some press after the Bigtable paper came out, OSDI 2006, after they'd had it in production for a few years and weren't afraid of anyone catching up. 2006: https://research.google.com/archive/bigtable.html https://research.google.com/archive/bigtable.html 2004: https://www.networkworld.com/article/2330258/tech-primers/rain-architecture-scales-storage.html https://www.networkworld.com/article/2330258/tech-primers/ra... 2004: https://patents.google.com/patent/US6704730B2/en https://patents.google.com/patent/US6704730B2/en (search for RAIN)
- hardwaresofton 9y agoBTW for those who are trying to get into this today, Ceph is one of the best (IMO the best, but my opinion isn't worth much in this arena) up and comers. It sits at the intersection of object-based storage (as opposed to "traditional" block storage), distributed systems, and robust forward-looking API support and open source. http://docs.ceph.com/docs/master/architecture/ http://docs.ceph.com/docs/master/architecture/ https://github.com/ceph/ceph https://github.com/ceph/ceph https://ceph.com/wp-content/uploads/2016/08/weil-rados-pdsw07.pdf https://ceph.com/wp-content/uploads/2016/08/weil-rados-pdsw0... (RADOS paper AKA a review of how to store lots of data in clusters) https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.... (CRUSH methodology paper, critical to why Ceph is better than previous approaches)
- chubot 9y agoThis is a tangent, because hosting providers didn't really use distributed computing back in 90's. They don't really have to solve any coordination problems because every customer is separate by definition. As far as I understand, their conception of distributed computing was limited to sys admins maintaining NFS and so forth. But going off on this tangent: I don't think BigTable was a major milestone in terms of fault-tolerant clusters of commodity PCs. For example here is a write-up paper from 2003 that describes Google's search clusters: http://www.eecs.harvard.edu/~dbrooks/cs246-fall2004/google.pdf http://www.eecs.harvard.edu/~dbrooks/cs246-fall2004/google.p... Google’s racks consist of 40 to 80 x86-based servers mounted on either side of a custom made rack (each side of the rack contains twenty 20u or forty 1u servers). Our focus on price/performance favors servers that resemble mid-range desktop PCs in terms of their components This architecture existed for a few years before BigTable did. Also, Google didn't invent this architecture. I did some research on this several years ago -- it appears that Eric Brewer's work at Inktomi in the 90's had an architecture that was largely similar to Google. I also think there was a name for it other than "RAIN". But I can't recall now. This isn't the paper I'm thinking of, but it's from 1997 and has a lot of the ideas from Google (which became a company in 1998): Cluster-Based Scalable Network Services https://people.eecs.berkeley.edu/~brewer/papers/TACC-sosp.pdf https://people.eecs.berkeley.edu/~brewer/papers/TACC-sosp.pd... We observe that clusters of workstations have some fundamental properties that can be exploited to meet these requirements: using commodity PCs as the unit of scaling allows the service to ride the leading edge of the cost/performance curve, the inherent redundancy of clusters can be used to mask transient failures, and "embarrassingly parallel" network service workloads map well onto networks of workstations. ---- Ah OK I found it, the term is "Networks of Workstations". From 1994: A Case for NOW (Networks of Workstations) https://scholar.google.com/scholar?cluster=1089787370567533783&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=10897873705675337... I believe this work influenced first Inktomi and then Google... since the last author is Patterson, that's pretty easy to believe. There are some stories about the Google prototype at Stanford (backrub) using every workstation they could cobble together. In this paper, we argue that because of recent technology advances, networks of workstations (NOWs) are poised to become the primary computing infrastructure for science and engineering, from low end interactive computing to demanding sequential and parallel applications. We identify three opportunities for NOWs that will benefit endusers: dramatically improving virtual memory and file system performance by using the aggregate DRAM of a NOW as a giant cache for disk; achieving cheap, highly available, and scalable file storage by using redundant arrays of workstation disks, using the LAN as the I/O backplane; and finally, multiple CPUs for parallel computing.
- tluyben2 9y agoSorry for the the low res pic but we ran millions of sites on these things [1]. Much cheaper than any hosting we could find. It was a nice time before the real race to the bottom. We hardly had downtime compared to the massive commercial hosters because we lived in the server cages those days. I believe much of my tinitus comes from spending too much time there. [1] https://picturepush.com/+154WU https://picturepush.com/+154WU
- iagovar 9y agoWhat is that? Servers? I may be blind but I don't what those are.
- tonyarkles 9y agoMy guess, having done some things like that at the time, is a motherboard, hard drive, and power supply strapped to a steel tray with no enclosure at all. The sides look like they may be plywood or 2x4s. Noisy and fragile, but easy to work on and dirt cheap to put together.
- tluyben2 9y agoYep! Exactly that!
- AlexB138 9y agoThis story is possibly about Rackspace. Rachel was a Racker at one point, as was I, and the origin story matches up.
- baking 9y agoRobert Marsh, Head Surfer http://web.archive.org/web/20030802132245im_/http://rackshack.net:80/english/images/hs.gif http://web.archive.org/web/20030802132245im_/http://rackshac...
- walrus01 9y agoBread racks and ordinary atx motherboards in mini/midtower cases are a much, much older idea than 2003... like mid-1999 to 2000 or so.
- bvinc 9y agoIt's a story about RackShack, which was renamed to Ev1Servers. Interestingly, I heard a rumour that the real reason for then renaming the business was because of a court case with RackSpace who said the name was confusing similar. But no, RackSpace was big name expensive hosting at the time. When Rachel talks about new competition, she may have been referring to ServerBeach, which was actually started by RackSpace but they sold dirt cheap servers and stole the entire surfer theme concept. I worked at RackShack and Ev1Servers during these times.
- tbyehl 9y agoSome other posts identify her as working at ServerBeach, which was once a part of Rackspace. ServerBeach's "Secret Sauce" was figuring out a hardware hack to cheaply integrate consumer-grade junk hardware into a typical DHCP/PXE-driven on-the-rack inventory and provisioning workflow. ServerBeach wasn't the first company using bread racks and junk hardware, but the others trying to match their provisioning speeds were either throwing bodies at the problem or spending real money on switched PDUs.
- petercooper 9y agoApparently, once this made the rounds, other people realized that customers were willing to pay for something so rough and dirty, and so why not create such a service themselves? This idea of getting "permission" to do things after seeing someone else do it crops up in my thoughts a lot. A lot of new fads in technology aren't necessarily that novel in any technical sense but are simply people realizing it's OK to cut a corner, "misuse" something, or otherwise take an unorthodox approach.
- Regardsyjc 9y agoI think this idea might be mimetic theory, possibly one of Peter Thiel's secrets, that people are mimetic and simply copy one another thus leading to competition. Another related idea might be blue ocean strategy which is basically similar to Peter Thiel's zero to one, on innovation.
- bvinc 9y agoI used to work at RackShack during these times, and when it was renamed to Ev1Servers, which this article is about. I can answer any questions and clear up any misconceptions. The company started out pretty much exclusively with Cobalt Raqs at first. They eventually bought their own real datacenter and physically moved all of the servers there and had a lot more room. I remember when they got these machines they're talking about. They sold them as AMD Athlon "white boxes" I believe. I was a little surprised when I noticed they were literally super cheap desktop machines, similar to "emachines" back then. They even had CDROM drives! But it didn't really phase me. Why the hell not? He obviously got a good deal on them. We just made racks, 5 on each row, 4 rows. They ran Linux just fine and had 100mbit network cards in them. And yeah, no firewalls. That's your job on your server. Most people ran servers on these machines, why would they want a hardware firewall? The networking was simple. Just 5 daisy-chained switches, one mounted on reach rack. 100 servers would share a block of 512 IP addresses. If your subnet ran out of IPs, we would literally move your server to a different subnet. They also saved money because most support techs were 18-20 and made just above minimum wage. But it was a great opportunity and I learned a lot. And we really did have a lot of really smart people working there. Someone else in here said that they would require your root password on support forums. No one was stupid enough to request that and customers were generally not stupid enough to post it, but I'm sure it probably happened once or twice. It was a similar situation in the official IRC channel. Some customers didn't grasp the concept and would accidentally paste their root passwords in there. The story about their fiber connection being down, I remember that I think, but it was before my time. I believe they only had 3 fiber connections, and one of them was severed. They decided that it was because of a dump truck picking up a garbage can. They were pretty open about it and posted pictures on their forums. They kept acquiring fiber connections. I believe they eventually had 10 different 1-gigabit connections at a single DC.
- bvinc 9y agoOh, another thing I just thought of. They didn't just give you access within the hour, like the article suggested. Within the hour may have been the guarantee, but typically, it was instantaneous. New machines would have a default password. When a customer bought a machine, a script would log in, change the password to the customer's, and notify them of their new machine's IP.
- mbrumlow 9y agoMy post was too long so I am breaking it down. Part 1 So... This is where I got my start in tech. I worked there and and supported and racked many of said bread rack servers. I guess I could be wrong and they could be speaking of another company but there was only one company I ever knew to do this at it was rackshack, before it became ev1servers, and then later theplanet and now softlayer. There were a few others in Houston at the time trying to get this working too, namely redhost (I think, or was it alphared or something like that. ) which I had the chance to visit once. This place was much worse they cooled the iles the customers would walk down meanwhile the servers got no AC. They also shimmied servers into parking lot storage rooms. But that is not likely the place the article was about. While what described in the article sounded horrible it was not all that bad. But then again this was also the early 2000s. While each rack did not have a dedicated firewall they did provide some niceties globally to all servers such as basic dos attack prevention and some interesting but easily avoidable hosting panel exploit prevention. It really was the wild wild west of hosting. You had the tech support named webtech who was normally off site and basically was a job for anybody who could spell linux. They did in house training to support the most common linux related issues. When I was there there was a guy named Bill Badoux. This guy ran training and had a fairly comprehensive training class for all incoming techs, but none of that really mattered, most of the real work always filtered down to a handful of knowledgeable people. What was important was somebody answered the phone and was there to listen to the customer -- mostly complaining. On the other end of things we had DC techs. They were mostly about rebooting servers as none of the whitebox servers on bread racks supported any sort of remote reboot. At the time you would phone in your reboot, or submit a ticket in this POS software called "pipeline" which was a total shitstorm in house ticketing system, that would more likely get you fired for it doing something different than you thought it might do than actually help you do anything. But us techs made the best of it and often resorted to IRC to actually get the real work done. The NOC was a separate entity -- I think their paychecks even came from a different org. They mostly sat around and watched graphs -- and again would call somebody who knew what to do when "the graphs looked bad". I know one guy nicked named Brain at the time who was one of the knowledgeable guys. He ended up writing some software to be able to display the graphs in real time. This really helped the techs spot things much faster. I was there about ~4 years. Man there were stories. One time I saw a DC tech drop a handful of hard drives on the way to a server to replace drives that had just failed. He picked the dropped hard drive up off the floor, dusted it off and proceeded to replace a failed dive with the drive he just dropped. By chance the server booted up and entered a FSCK -- we might be okay! -- but atlas the tech just powered the server off and powered it back one and it never saw the light of day again. Back then and probably still interrupting a FSCK was most defiantly doom to the FS. Slightly NFSW but a true story. Rackshack hosted 2 things - Porn, and Jesus sites. Neither were all that nice to work with. The Jesus guys would always find out or suspect their server, hard drive, or IP was used to host porn in a previous life and demand we change one or all of the things they suspected had porn on it. This more than often resulted in your request causing your server to be shut down for a few hours and then turned back on and a note updating the ticket from DC stating the hard drive or whatever that had previously had contact with porn was swapped out. In reality the server just sat powered off for a few hours. Request for IP changes did happen and were actually carried out if they could, but again this is was the early days so that often required the customer to migrate their site to a new server in the data center. But back to the porn. One time I got a call from a guy wanting to know if his site was up. He ask me to navigate to a obvious porn site -- sure enough on our network with the less than 1 ms ping time. He proceeded to give me login information and navigate me to a specific video and asked me to click on it. It played instantly. After a few seconds I reported back to the customer and he asked a few question about the speed of the site. Then he proceeded to tell me the video was of him and his wife.... Now at that point I kept good notes in the POS ticketing system named pipeline. Those notes included the login credentials to the site for future testing -- I was a noob for sure. The next day I walk in to work and it was clear everybody had found the login credentials to the site and that was the first time I caused a policy change of do not record such things as the techs will abuse it for their personal pleasure.
- rdl 9y agoThis kind of "ghetto colo" is prevalent in the cryptocurrency mining space, now (where power consumption is probably 10x higher) -- either using hardware ASIC miners or PCs with many GPUs. Often the hardware has an in-service lifetime of months, vs. ~10 years for some enterprise or networking equipment, and outages cost a lot less (a 75% reliable datacenter would be a joke; a 75% uptime crypto mine that cost $0.01/KWh might be fine -- "turn it off during peak load periods" or otherwise pre-emptable by the grid.
- pbreit 9y agoWhat's the takeaway here? I would say: there's business to be had by questioning all the conventional wisdom. And: customers like low prices.
- mbrumlow 9y ago> I may never know if this picture and/or article ever happened, but it does raise an interesting point. If the only thing keeping you the only player in a certain market is ignorance of how you built your environment, just how long do you think that will last? As a note, it lasted long enough for the guys to make a very very healthy exit and for the data centers to eventually transition into the backbone (along with a bunch of other - mostly non profitable data enters) of Soft Layer now owned by IBM. Edit: Also here are some pictures of the place. http://www.datacenterpics.com/displayimage.php?album=3&pid=614#top_display_media http://www.datacenterpics.com/displayimage.php?album=3&pid=6... Picture in question: http://www.cloudcomputingexpo.com/node/102155/print http://www.cloudcomputingexpo.com/node/102155/print https://www.bizjournals.com/houston/stories/2006/05/08/story4.html https://www.bizjournals.com/houston/stories/2006/05/08/story...
- mbrumlow 9y agoAlso should note that at the time this strategy allowed the company to host about 1% of the internet at the time.
- N8works 9y agoduring these days my company was in a data center in Los Angeles next to "Web hosting for life" a company who's business model was based on a single upfront payment that got you hosting for "life". (Whatever that means). Their secret sauce was cages full of 19" Telco racks with shelves filled with motherboards, power supplies, and hard drives. No cases. Some of the hard drives were zip tied to the shelves. We'd tour potential customers through the data center and say "this is what shared hosting gets you, what if your site was bring served from that "server" with it's hard drive hanging by a broken zip tie?" It was a great sales technique.