13 ms·
This is what 864GB of RAM looks like now
- strictfp 15y ago"...russian-doll architecture of nested caching that I’ll write up in detail soon." A.k.a object orientation? (Maybe an explanation is due here, or do other HNers grok and agree?)
- strictfp 15y agoOk, ok so you don't get it. Fine, I'll write a blog post about it.
- jonknee 15y agoAnd who says Rails doesn't scale?
- callumjones 15y agoI doubt Rails is doing the caching.
- wlll 15y agoFacebook: "…use more than 800 servers supplying over 28 terabytes of memory to our users." (http://www.facebook.com/note.php?note_id=39391378919 http://www.facebook.com/note.php?note_id=39391378919) The scale of a memcached cluster doesn't really say anything about the scaling abilities of their underlying stack.
- jonknee 15y agoComparing BaseCamp and Facebook is laughable. That said I was referring more to their one giant database and self described "russian-doll architecture of nested caching".
- wlll 15y agoI'd bet that Facebook uses a fairly complex cacheing system though, so your point that using extensive cacheing says anything about the ability of the underlying stack to scale still doesn't really work for me.
- callumjones 15y agoTheir caching and database still doesn't really imply much about Rails, which is what wlll was trying to get. Whether it be Rails for BaseCamp or PHP/Facebook stack for Facebook it really says nothing about the database and caching behind it. A web front may be really great at scaling but it's all up to the database.
- badclient 15y agoNote that post is from 2008. They must be at way over 28TB now.
- tezza 15y agoWhat sort of servers are they going into? ~ 13 servers with 4 sticks (64GB) each?
- rwmj 15y agoPfft ... We were testing RHEL 6 on machines with 2 and 4 TB of RAM (in a single machine) the other day. They have lots of cores and are very NUMA.
- Nrsolis 15y agoI would love to hear more about this hardware. Can you elaborate?
- gaius 15y agoYou can buy that sort of kit off-the-shelf now, from HP and others. It's nothing exotic, it's basically your desktop PC on steroids. NUMA is 90's tech from Sequent.
- ilaksh 15y agoCan the CPU(s) address 512GB of RAM? If so, what operating system/OS configuration can handle that?
- cube13 15y agoRHEL6 has a theoretical maximum of 48 bit addressing space(at least with the 1st beta), which is 256 TB. From: http://www.theregister.co.uk/2010/04/21/redhat_rhel6_beta_1/ http://www.theregister.co.uk/2010/04/21/redhat_rhel6_beta_1/
- Hoff 15y agox86-64 architecture presently allows 256 TB (48-bits) physical addressing, and Itanium and at least some its operating systems (HP-UX, OpenVMS) have 50-bit physical addressing available. The big HP NUMA boxes are Superdome and Superdome 2, and those have global memory access, though access to the local memory is faster than the remote memory; the requisite non-uniform memory access speeds. Last I checked, the current Superdome and Superdome 2 series maxed out around two and four terabytes of physical memory. If the current four terabyte maximum memory configurations and possibly the architectural 50-bit addressing aren't enough and you're willing and able to buy that much physical memory, then HP would probably interested in chatting with you. No RHEL on Itanium, though.
- adestefan 15y ago"...russian-doll architecture of nested caching that I’ll write up in detail soon." The details will be interesting, but that description sounds like a headache waiting to happen.
- tomjen3 15y agoOnly if you don't have good enough software engineers. The CPU is organized that way and it works very well and that caching scheme is implemented directly on the metal. Any decent software engineer should be able to design a working implementation with an architecture that is clean.
- jbellis 15y ago"There are only two hard problems in Computer Science: cache invalidation and naming things."
- nicktelford 15y agoAnd off-by-one errors.
- wlll 15y ago"There are only three hard problems in Computer Science: cache invalidation, naming things, off-by-one errors and pedantry."
- deleted 15y ago[deleted]
- deleted 15y ago[deleted]
- nicktelford 15y agoAnd the Spanish Inquisition.
- ck2 15y agoThose must be ECC prices. 16GB DDR3 is "only" $100, that would make non-ECC $5400 ah okay, ECC is $175, so 175 * 3 * 18 = $9450
- bradleyland 15y agoGenerally speaking, you don't want to run a server with 840 GB of RAM and the entire universe cached in RAM without ECC.
- silvestrov 15y agoGenerally speaking, you don't want to run a server without ECC.
- jaxn 15y agoGenerally speaking, you don't want to run a server.
- Nrsolis 15y agoGenerally speaking, you don't want to run.
- dredmorbius 15y agoGenerally speaking, you don't want.
- batista 15y agoA funny exchange? What is this, Reddit? Get serious, all of you.
- drats 15y agoI agree with you telling them to get serious, but I would add that it's not funny (it's not funny on Reddit either).
- hyuen 15y agoI wonder what kind of stuff can be done with 1Tb of SSDs, much cheaper than RAM, but also somewhat slower...
- lallysingh 15y agoWell, it's 256 GB of ram each for 13 servers. I do wonder if the second half of that RAM, with the money used for SSDs, would result in better performance -- as your cache could be larger for the same budget. I think it'd really come down to your cache hit rate / cache size.
- TylerE 15y agoNot for a memached server. If it ain't RAM, memcached can't use it.
- robotresearcher 15y agoDoes memcached not use virtual memory?
- wmf 15y agoIIRC memcached is event-driven and swapping is death for event-driven programs because it blocks the whole event loop.
- TylerE 15y agoIn theory it probably could, but you're losing all benefit at that point. The big selling point of memcahce is that it's VERY fast. Our memcache server (Ours is only 8GB, not nearly as impressive as some) averages under 1ms object fetch times, even accounting for network overhead.
- danudey 15y agoMemcached is designed to be fast. Anything that makes it slower that doesn't need to be there isn't there. For example: authentication (anyone who can access the port can fetch data), indexing (you need to know the key), deallocating memory (you configure memcached with an upper bound; it keeps allocating memory as needed until it gets there), etc. Virtual memory (I assume you actually mean 'does memcached not page data out to disk') would make it much slower. Since memcached is just that - a memory cache - if you're out of memory it just expires the least-recently-used data. In your application, you fetch the key, and if that fails you fetch it from the primary data store (or wherever else you can find it).
- st3fan 15y agoThis is not really special anymore in 2012?
- wmf 15y agoNo, but a lot of "sysadmins" are being raised with no knowledge of hardware and some others like to cache knowledge for years and need a good shock to bring them into the present.
- ajdecon 15y agoWe've been playing with a Kove xpd (http://kove.com/xpress http://kove.com/xpress) with 2 TB RAM lately. 2 TB RAM and four Infiniband cards make for a very fast DB server... :-D
- jebblue 15y agoThat would make a great Minecraft server. For real.
- ch 15y agoHmm. I wonder if those are laid out on a grounded mat or just some random office table, sitting on carpet?
- herge 15y agoWe had a server with half a TB of ram. It was a lot of fun until we had to reboot it, and then it took half an hour to perform a memory check.
- zdw 15y agoI worked on a Mac Pro once that someone had 64GB of RAM installed in... took the better part of a minute to get the startup gong. They ended up splitting the memory between a few machines as it became obvious that the applications being used (Adobe CS stuff) wasn't going to use all that RAM in their use case, and other machines needed it more.
- mashmac2 15y agoAlthough the new Final Cut Pro X eats RAM for breakfast. Our editing Mac Pro at work regularly uses 16GB+ when working in FCPX with multiple projects and events. Frustrating at times, to say the least.
- illumin8 15y agoThis is what 864GB of RAM looks like after you laid it all out on a table without using ESD (electro static discharge) protection. A few of these DIMMs are probably bad now, but you won't know which ones because you're going to stuff them all in a Linux box that you built yourself... this is such a bad idea for a production system.
- wlll 15y agoThe DC took the photo for us, I'd have to check with them what precautions they took but it looks like the build station if my memory serves. The RAM went in Dell boxes.
- Andys 15y agoYou're probably safe. Most data centres have controlled humidity and anti-static flooring
- Aloisius 15y agoIn my decades of building and taking apart computers, I have never had static discharge ruin a component. I can't remember the last time I was shocked by static discharge for that matter. Who are these people who need to be grounded all the time else they build up to component destroying levels of static?
- DanBC 15y agoYou don't need to feel the shock for it to be enough to destroy a component. ICs are more robust when they're built into a circuit, but it's still a good idea to use sensible ESD protection when handing electronics, especially if you're paying lots on production systems.
- jodrellblank 15y agoHow do you know you never had? Are you saying you've never had any component failure, or any inexplicable crashes or unreliability?
- ajack 15y agoIt's great that companies post articles describing their architectures and backend systems (and I'm certainly looking forward to your follow-up post detailing your caching system) but why show a picture of commodity hardware available for anyone with enough money? What's impressive is your skills, now how much money is in your bank account.
- harrylove 15y agoBecause it's fun. Toys are fun. Upgrades are fun. That's all it is. And most of us never get to work with that kind of hardware. It's fun to see it all in one place.
- Cushman 15y agoThis is a picture of $22 million in cash: http://www.common-sense-politics.org/wp-content/uploads/2010/05/postie-media21.jpeg http://www.common-sense-politics.org/wp-content/uploads/2010... This is a picture of more than a ton of dried marijuana: http://calpotnews.com/wp-content/uploads/2010/09/090410-Corning.jpg http://calpotnews.com/wp-content/uploads/2010/09/090410-Corn... A large amount of one thing in the same place makes for a pretty cool picture. Don't take it so serious. Now, with that said: 864GB? Pssh. Let's see some terabytes.
- jrockway 15y agoBefore I worked at Google, 864GB of RAM would have impressed me. Now I think I have that much in my laptop...
- viraptor 15y agoYes, big companies put some things into a different perspective. Just today I heard a sysadmin saying they lost 50GB or so storage per server due to some partman issue. But that's ok... adds up only to ~ 10+ TB and only temporarily (fix via lvm resizing possible).
- brudgers 15y agoThat picture reminded me of the first time I saw a fully populated 8meg board - for about the same price.
- 16s 15y agoIt's common to see servers with 100+ GB of memory these days and I've seen some programs use it all!
- gaius 15y agoIs this supposed to be impressive? Show me what you can do with how little resource. Anyone can brute-force it.
- makmanalp 15y agoDevil's advocate counter-point: Is there any point to not bruteforcing it? While you twiddle bits, I can throw money on hardware and get my product out on the market. Then, when times are more stable, I can worry about optimizations. Also, since I haven't done any twiddling yet, there is likely to be a lot of low hanging fruit. I was going to need that hardware anyway if I was to grow. Now I can have a period of time where my savings allow me to slow down on buying more hardware.
- gaius 15y agoI can throw money on hardware and get my product out on the market. Up to a certain scale what you say is true - but it's not interesting at all. As in, anyone can drive 26 miles but running a Marathon is still impressive. Or anyone can order dinner in a restaurant, but not everyone can cook.
- groby_b 15y agoRunning a marathon is impressive, yes. But if I'm in the business of delivering food (to stretch an already thin analogy ;), there's no point in training for a marathon. The constraints of my business make driving around in a car a better choice. And unless I'm a professional chef, I'd rather not spend time on cooking dinners when I could do something to improve my business. The point of building something (in a commercial environment) is not to impress, but to ship something.
- rbranson 15y agoNo delivery business would post a blog with a picture of their new Ford Taurus delivery car as if it was something impressive. That's the difference.
- deleted 15y ago[deleted]
- wlll 15y agoIt's in a secure datacentre surrounded by millions and millions of dollars of other equipment (some of it ours), it should be OK :)
- russell 15y agoThe obligatory comment from the old guy: My first computer, an IBM 7094, had 32k of 36 bit words, about 200K characters. The memory cost about $1M and was the size of a refrigerator.
- dredmorbius 15y agoI had two rocks. One was down and the other was not available at my security clearance.
- NanoWar 15y agoReminds me of the days when 1 MB of RAM did cost 500 bucks. Used to run Windows 95 :) after installing from floppy.
- prolepunk 15y agoI wonder if they would run into fragmentation problems with this much of memory, and if so how they are going to deal with that?
- st3fan 15y agoI think this site could use some of that memory and caching love. Loading articles or even /newest is sooo slow sometimes.
- iamleppert 15y agoI question the architecture and approach of this much caching. Because this is memory, I assume it's for a read-only cache? That said: most read heavy services only need at most 10% of their working set cached, in memory. The hardest problem in caching is figuring out what that set is and how to control consistency. Not how to store it. So, having this much cache seems to imply that you think you're going to have large amounts of read intensive data to cache. Or else you intend to cache your entire working set? Either way, you'll have a single point of failure (either in a single server, single datacenter, or single geographic area). Any large operation can tell you that the problem becomes 90% network and stuff like a CDN become far more important than how fast and big your cache is. Perhaps a nice technical writeup on your architecture would silence the pundit inside me?
- forgotusername 15y agoI'd be more interested in a writeup of what data you actually store. I've used Basecamp at a customer's before, and it certainly didn't justify the requirement of 1TB RAM, let alone 100kb RAM to serve quickly. This sounds more like Reddit's problem where some architectural simplifications might net a giant win versus piling yet more gunk on top (Reddit is still perceptibly doing random IO for every comment in a thread during page load, or perhaps some insanely slow sorting, I have NFC how they haven't fixed this yet).
- LokiSnake 15y agoI'd love to hear how Reddit is doing it wrong and can be improved. As far as I can tell, they want users to see the freshest content possible (latest comments, up/down vote counts, etc.). Is it really that easy?
- rbranson 15y agoRead on: https://github.com/reddit/reddit https://github.com/reddit/reddit
- dhh 15y agoThis is all for HTML fragment caching using key-based expiration. I'll write up all the details soon.
- lrobb 15y agoOne of the bright spots of my career was building a distributed system that spanned 64 different nodes and regularly did over 100Mb/sec... Built on machines holding (at the time) a top-end config of 2Gb/ram per machine... That's more RAM than our entire system had.
- foobarbazetc 15y agoThat's vastly more interesting and impressive than a blog post about how much RAM some company bought, though. :) You've got $12K? Good for you. Here's a cookie. You've scaled out a system onto 64 nodes? That's something worth discussing on HN.