12 ms·
The end-to-end refresh of our server hardware fleet
- loser777 10y agoI hope some of the old hardware makes it to ebay, but it looks like many of the form factors are proprietary.
- jacquesm 10y agoWhat do you intend to do with it? I've found that almost any kind of short lived experiment I can do cheaper on AWS than doing it with hardware that I own. If it is longer running then it might become viable to own the hardware.
- archimedespi 10y agoeh, owning hardware is fun and a great learning experience
- jacquesm 10y agoOwning is passive, it's what you do with it that matters.
- hueving 10y agoIt's sad to me this is becoming the status quo. Using other massively centralized companies for compute resources is a sad future. It's bad for privacy, it's bad for diversity to protect against SPOFs, it's bad for general computing hardware (vendors primarily target the giants), it's bad for users via vendor lock-in, and it's bad for open source projects in the infrastructure space. I think hackers justify it to themselves by pretending it's a commodity like electricity, but it's far from that. If my utility goes out, I can turn on generator and get exactly the same electricity. If Amazon goes out I have to build again on another cloud from a (hopefully recent) backup or just sit dead (like the recent s3 outage). Sorry about the rant, but is there anything that would get you to stop giving the keys to the kingdom to Amazon?
- jacquesm 10y agoHardware is a means to an end. I've got plenty of it but at the end of the day what you do with it should be balanced by what it costs. For companies that have instances running long term it can very well be cost effective to own the hardware. My email server, web server and DNS server are on my own hardware with a co-location facility that I trust. But for experimental stuff where you need to spin up a hundred machines for an hour or two you just can't beat the cloud (and that's my only use case for the cloud, though I can see others go much further). I don't like the monoculture any more than you do, but to see this as me having given the 'keys to the kingdown to Amazon' is several steps too far.
- hueving 10y agoI misinterpreted what you meant by a "short lived experiment". I took that to basically mean any project when you're starting out. My apologies. Whenever I'm experimenting I rarely need a burst of 100 instances, it's usually 1 or 2 instances to run things and I prefer to run them on my own hardware.
- dpark 10y ago> I think hackers justify it to themselves by pretending it's a commodity like electricity, but it's far from that. If my utility goes out, I can turn on generator and get exactly the same electricity. If Amazon goes out I have to build again on another cloud from a (hopefully recent) backup or just sit dead (like the recent s3 outage). What's your use case? Are you just futzing around at home? Sure, use a server in your bedroom. Who cares? Are you delivering a service to other people? Then owning hardware is probably a bad idea. If it's in your house, your users are hosed if you lose power or your internet cuts out. Putting it in a DC just means you're handing the same keys to someone else, but your self-managed hardware is definitely going to be less reliable than Amazon's infrastructure. Owning hardware is a bad deal for everyone involved unless you're big enough to build your own HADR infrastructure.
- kuschku 10y agoMy raspberry pi at home has had in the past 3 years less downtime than aws. Local datacenters in the city had even less. I'm not sure where AWS is supposed to get that famous reliability from, but it's not in uptime. (I can't comment on storage reliability, because I only write a few terabytes of data a month — but otherwise, there's RAID 5 or other RAID setups to ensure data staying valid) AWS has its advantages in its immense scalability within of seconds, it has its advantages in convenience. But its uptime isn't much better than most home connections. Home statistics: Power downtime since 2006 is 29 minutes. Internet downtime since 2006 is 6 hours in 2014, 2 times 30 minutes each in 2016. This is on a 100/40 DSL line nowadays (the downtimes were, except for one, when switching ISPs), without any universal power supply, battery or generator. For comparison, this is equivalent to a downtime of 99.99% — the same as AWS advertises, but better than what they delivered in this or the last year.
- ssttoo 10y agos/proprietary/opensource/ http://www.opencompute.org/ http://www.opencompute.org/ :)
- rb2k_ 10y agoA lot of them already are: http://www.ebay.com/sch/i.html?&_nkw=open+compute+server http://www.ebay.com/sch/i.html?&_nkw=open+compute+server
- nananonymous 10y agoIf facebook is just now announcing their upgrade should we expect these prices to go down?
- wmf 10y agoThis is the Nth generation. Facebook has been continuously decommissioning old servers for years already.
- scurvy 10y agoThat's all regular Quanta gear. No idea if it was owned by FB or another OCP adopter. OCP is popular with mineral companies.
- trustfundbaby 10y agoAnyone know what cpus/gpus they use in these?
- jacquesm 10y ago"Built in collaboration with our ODM partner QCT (Quanta Cloud Technology), the current Big Basin system features eight NVIDIA Tesla P100 GPU accelerators. These GPUs are connected using NVIDIA NVLink to form an eight-GPU hybrid cube mesh — similar to the architecture used by NVIDIA's DGX-1 system. This setup, combined with the NVIDIA Deep Learning SDK, utilizes this new architecture and interconnects to improve deep learning training across all GPUs. Compared with Big Sur, Big Basin will bring us much better gain on performance per watt, benefiting from single-precision floating-point arithmetic per GPU increasing from 7 teraflops to 10.6 teraflops. Half-precision will also be introduced with this new architecture to further improve throughput."
- iDemonix 10y agoSo I just run the setup CD, right?
- jacquesm 10y agoAssuming your question is serious: Yes, essentially, but you're going to have to temporarily connect some peripherals if you don't intend to pop in a pre-installed image on a drive. And of course, depending on what OS you intend to run on it you might end up with driver hassles, your best bet is likely a reasonably modern linux distro.
- opsunit 10y agoFacebook PXE boot to Anaconda, use Chef to provision the host and then ultimately use their own scheduler to execute a workload.
- wmf 10y agoFacebook uses Xeon D and Xeon E5 CPUs. https://www.servethehome.com/facebook-at-open-compute-summit-2017-7-5-quadrillion-web-server-instructions-per-second/facebook-ocp-summit-2017-facebook-server-types/ https://www.servethehome.com/facebook-at-open-compute-summit...
- cortesoft 10y agoI really dislike this sort of naming scheme (Bryce Canyon, Honey Badger, Mono Lake, etc) The names tell you nothing. You can't tell which one came before which, or even what they are. You just have to KNOW that information. A good naming scheme tells you information about the thing named.
- wmf 10y agoHistorically, code names were chosen specifically to give up no information and it seems like that tradition continues, perhaps unintentionally.
- hkmurakami 10y agoIs there since information in Intel codenames I'm perhaps unaware of?
- krylon 10y agoI haven't been able to keep those straight for years. Maybe this is just me getting old, but I miss the old days, when you could easily tell that a Pentium is faster than an 80486, and that a Pentium 133 is faster than a Pentium 100. These days, CPU speed matters less than it did back then, but there still are CPU-hungry applications (I'm looking at you, Autodesk Inventor!), and if I had to put together a PC from scratch (which I think I'll actually sometime this year), I would be kind of lost.
- foota 10y agoPart of the issue here, I think, is that cpu's are much more complex than they were then. You have a number of different cpu lines with different models on the market at any time.
- krylon 10y agoThat is true. But that does make the decision what CPU is best for a given use case and budget much more complex, too. (Like I said, the impact of the CPU on overall system performance is less today than twenty years ago for many use cases, so it is not that much of a problem.)
- nodesocket 10y agoHow long until Facebook joins the public cloud business with Amazon, Google, and Microsoft?
- wmf 10y agoNever? Their infrastructure is cool but it's only around half of what a public cloud would need.
- motoboi 10y agoWell, they could double it then and reap the benefits of scale.
- nodesocket 10y agoNever... I'm not convinced. Rollback to when Amazon was pre AWS. Everybody thought they were crazy announcing they were getting into the datacenter and cloud business. I'd say it has worked out well for $AMZN.
- dragonwriter 10y ago> Everybody thought they were crazy announcing they were getting into the datacenter and cloud business. All the comments I heard were positive about how they were diversifyiny by leveraging expertise they had been forced to develop for their own core platform, not that they were crazy. I'm sure there were some.who.said "crazy", but it definitely wasn't everyone.
- addicted 10y agoI agree. Maybe Amazon sold it really well, but as far as I can remember, the response almost universally (including in financial circles) was that this was a great idea since it allowed them to leverage idle resources they needed to build out to handle peak loads (such as during the holiday season).
- 10y ago
- quickben 10y agoIs it that cheaper to custom build, if you can't ebay them off at their half life point to recover some of the cost?
- wmf 10y agoMost of the cost is in processors and RAM so those parts can be sold at end of life. There are server recycling companies that specialize in this.
- saycheese 10y agoHere's an example of one: http://cashforelectronicscrapusa.com http://cashforelectronicscrapusa.com
- PhantomGremlin 10y agoYeah, that kinda reminds me of expertsexchange.com Is it electronic-scrap or is it electronics-crap ?
- saycheese 10y agoThe terms of use page refers to "CJ Environmental" which when combine with the INC 500 reference of the home page returns this profile: http://www.inc.com/profile/cj-environmental http://www.inc.com/profile/cj-environmental From the FAQs, "How does material get processed and refined? All materials are unique and subject to different methods of processing. Newer computers are refurbished and given new homes to maximize ROI for our customers. Scrap product is crushed or shredded before the refining process. All material is processed in accordance with all Federal, State and Local regulations. To learn more about our licensing and compliance measures contact us."
- taf2 10y agoWhen can we start hosting our services with Facebook - similar to aws, gce, etc?
- JohnJamesRambo 10y agoSo much equipment and money devoted for something as pointless as Facebook. I wish it could go for something cooler and more useful. Something that hasn't been shown in studies to make us feel lonelier.
- Swizec 10y agoAs a small business advertiser: Facebook is awesome. Nowhere else can you get $400 of sales for $10 in ads. That's a 40x ROI.
- exodust 10y agoSo every time you spend $10 on ads, you get $400 of sales? And if you spend $0 on ads you get $400 less revenue for that period? Sounds like fake news to me.
- jonknee 10y agoYour ROI is from profit, not revenue. You can easily get $400 in sales for $10 in ads if you don't care about profit.
- nickpsecurity 10y agoYou can get it for $0 in ads on Craigslist. ;)
- deleted 10y ago[deleted]
- yeukhon 10y agoI ask what is useful then? People who are working on Open Compute project are definitely learning new things. Every engineering blog post from FB is the reassurance of quality research done right. FB's work may not be impactful to you, but it is impactful to a lot of others. Ever heard of React and Cassandra? All came out of Facebook. Sorry, I have to be harsh, your comment is useless.
- hyperbovine 10y agoThe last time hey did this it flooded the market with dirt cheap E5-2560s. Are we in for a new updated deal?
- fnbr 10y agoWhere did you find them? eBay?
- quink 10y agoTo the extent that dual socket LGA2011 ATX/LLB boards moved up in price to a point where you could pay three or four times more for a suitable mobo than for sixteen cores of CPU. eBay E5-2670
- matthew-wegner 10y agoThis used to be <$500, but currently $633 for a motherboard, dual E5-2670 CPUs, and 128GB ECC memory: http://www.natex.us/Intel-S2600CP-Motherboard-Package-Deal-p/intel-s2600cp2j-128gb.htm http://www.natex.us/Intel-S2600CP-Motherboard-Package-Deal-p...
- anderspitman 10y ago+1 for natex. That's where I got my motherboard and it's worked great for about 6 months.
- anderspitman 10y agoHighly recommend this case if you go with an Intel S2600CP (or any other SSI EEB mobo): https://www.newegg.com/Product/Product.aspx?Item=N82E16811854003&cm_re=enthoo-_-11-854-003-_-Product https://www.newegg.com/Product/Product.aspx?Item=N82E1681185...
- blakes 10y agoI am using the Fractal Design Define XL R2 (what a name) for my Supermicro dual CPU mobo, I drilled one extra mounting hole and the mobo mounted just fine. Very quite and solid.
- dexterdog 10y ago100 million hours of video played per day. Are people actually watching this video or are they just inflating the number?
- mgkimsal 10y agogiven a lot of auto-play video in my own feed, I'm presuming it's not all actually 'watched'. if they'd give a separate number on videos 'listened' to (where I actually unmute the audio), I'd take that number more seriously.
- Strom 10y agoThis wouldn't be too accurate anymore either, because now even audio autoplays.
- Buge 10y agoIs this a setting or experiment or something? It doesn't happen for me.
- mgkimsal 10y agoNeither for me. Videos play, but the audio is always muted. I have to enable it to play for each video, and have for more than a year.
- Strom 10y agoApparently they've tested it for a year or so already and are now doing a wider rollout. http://www.wxyz.com/money/consumer/dont-waste-your-money/facebook-autoplay-audio-has-users-freaking-out_ http://www.wxyz.com/money/consumer/dont-waste-your-money/fac...
- ploggingdev 10y agoFacebook has over a billion daily active users, so 100 million hours divided over 1 billion users is 0.1 hours/user which is 6 minutes per user. Seems reasonable. Of course, there are lots of people who don't watch any videos and on the flip side there are a lot of people who watch a lot of videos on facebook. Edit : as pointed out below, the autoplaying videos might skew the numbers quite a bit.
- jabl 10y agoI'm disappointed in the "open rack" designs. For a really minor improvement in density they have broken compatibility with standard 19" gear. One could argue that at FB scale it's worth it, but then MS seems to manage just fine with 19".
- oso2k 10y agoIt's interesting. If they wanted to, they could compete with the likes of HP, Dell, Lenovo, and Cisco if they could ramp up production to accommodate customers. I wonder who does their manufacturing on the backend.
- wmf 10y agoFacebook uses Quanta/QCT, Celestica, and Accton for a lot of their manufacturing. You can buy Facebook servers from companies like Hyve, AMAX, and Stack Velocity but they aren't really aiming at the mainstream server market.