14 ms·
Numbers everyone should know
- known 16y agohttp://www.beej.us/guide/bgipc/output/html/singlepage/bgipc.html#flocking http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc....
- fleitz 16y agoPeople should also know for writes that there are a variety of ways to mitigate this. The first way is to write sequentially (no seeking). The second way is to install battery backed write cache which will do this for you. If you're ok with losing a little bit of data you can also just have the OS write to memory before flushing to disk. BBWC controllers seem to handle write reorder optimization much better than the OS. So I'd recommend the BBWC over writing to memory. This is a large reason I dislike cloud hosting, most of the hardware is crap and you could get much better performance from it with a little more expense. Also it's mostly pointless to bother making your writes sequential as the performance degrades to random IO because of all the other instances on the machine.
- rythie 16y agoNeeds updating for SSDs, I'm not sure it's really worth the development time and increased time to market to optimise for rotating disks.
- aristus 16y agohttp://www.facebook.com/notes/facebook-engineering/the-full-stack-part-i/461505383919 http://www.facebook.com/notes/facebook-engineering/the-full-...
- idlewords 16y agoYou seem to have accidentally traveled back in time to 2011 from whatever golden future you live in before posting this comment.
- rythie 16y agoWe spent a lot of time over several years tweaking every setting imaginable, trying to get the performance up of a NFS server, none of which really made any difference. Installing SSDs fixed the problem completely. Disk I/O now stays in the single digit percentage range when previously would stay at 100%, sometimes for minutes. SSDs about twice the price of 15k rpm drives, hardly out of reach for most. Also for VPSs, server axis 20mb unmetered SSDs are cheaper than slicehost's rotating disk based systems. http://serveraxis.com/vps-ssd.php http://serveraxis.com/vps-ssd.php
- kragen 16y agoYeah, SSDs are going to totally kill 15krpm disks. But killing disks in general is a ways off. In fact, disks might actually be getting cheaper per bit faster than SSDs are.
- kragen 16y agoSSD still costs US$2-US$4 per gigabyte. Spinning rust costs US$0.10-US$0.20 per gigabyte. SSDs are much faster but disks are still twenty times bigger at the same price. A developer hour costs on the order of US$100. If your service uses more than a few gigabytes per developer hour, it's worth some development time to optimize for rotating disks. Whether it's worth increased time to market is harder to calculate.
- nphase 16y agoThankfully there's a happy fusion of spinning rust and SSD: flashcache [0] and L2ARC/ZIL [1]. Cheaper win. [0]: https://github.com/facebook/flashcache https://github.com/facebook/flashcache [1]: http://blogs.sun.com/brendan/entry/test http://blogs.sun.com/brendan/entry/test
- hammock 16y agoNot trying to be mean, but sometimes I wonder how many programmers actually have friends outside the industry. This happens when I read titles like "Numbers everyone should know" and then find out that it's numbers like the time needed for an L2 cache reference or how to optimize for low write contention. Does anyone realize that this stuff is about as esoteric as it gets?
- cellis 16y agoI thought the same thing until my "put()s" were slow as hell on App Engine. "Oh hey this is just like mysql_query()! I just call entity.put() and everything's fine!" Not! When it comes to parallel operations over multiple machines ( most startups at scale will run into this problem ), these numbers really can come in handy.
- AgentConundrum 16y agoYou're discounting the context. If you read the title as "Numbers Everyone [who reads this site] Should Know" then it makes more sense. If you're complaining about the Hacker News title, then: a) It's the linked article title, so it would be inappropriate to change it, and b) HN shows the domain name of the link in parentheses after the title. The article was pretty much in line with my expectations for a High Scalability article with that title.
- yanowitz 16y agoI can't speak to your specific use cases, but on hacker news (or a blog about scalability) I think it's completely reasonable to have such a title. It's not link bait. Anyone building a system (e.g., for a startup) or maintaining/extending an existing system should be aware of these numbers -- if for no other reason than that they are fully aware of any tradeoffs they are choosing to make ("well, optimizing for seeks on write is premature at te moment, we need some users first, but I can at least add a level of indirection to prepare for future improvements." or even "yes, I know this will require refactoring as soon as we have XXXX users, but f-it, we gotta get to market. Let me notify others of the iceberg I'm building and why.") Knowledge, properly applied, can be power.
- wingo 16y ago
- deleted 16y ago[deleted]
- pfedor 16y agoWell if anyone wants to learn more from Jeff Dean than just twelve numbers, here's a talk he gave at Standford about Google's architecture and system building in general: http://ee380.stanford.edu/cgi-bin/videologger.php?target=101110-ee380-300.asx http://ee380.stanford.edu/cgi-bin/videologger.php?target=101... The slides for the talk: http://www.stanford.edu/class/ee380/Abstracts/101110-slides.pdf http://www.stanford.edu/class/ee380/Abstracts/101110-slides....
- alecco 16y agoBlogspam repost of a 2yo+ presentation. A very good one, but still...
- chuhnk 16y agoI'm sorry you feel that way. I was re-reading information on systems architecture and came across the article. I thought it was relevant to HN and thought hey this is something people might like to read. Not everyone here has been here since 09 and not everyone here will have seen the article. Just trying to help educate fellow members of this community.
- Getahobby 16y agoI know noSQL is all the rage with the crazy kids these days and I know it has its place but why is half of the literature regarding noSQL about solving problems that are trivial in an RDBMS?
- Johngibb 16y agoBecause RDBMS can only scale as far as your most powerful machine. Unless of course you start sharding to several machines, and then you hit the same issues mentioned in this article.
- boyter 16y agoStack overflow runs on a single DB server. I think people really overestimate their data-store needs, and underestimate how much a big iron DB server can deliver. Its probably worth thinking about scaling out on your data end, but until you start hitting some sort of limit on a 8 core 96 gig machine with server level SSD's its probably worth investing your time in other issues.
- Johngibb 16y agoThat's a very valid point. You can get reallllly far with just one really powerful database. But, if you've built a site on the precondition that one database is enough, with a lot of joins so you can't split tables, and your hardware can't keep up.. you're in a really bad place. I say this out of experience ;)
- boyter 16y agoJust out of curiosity what sort of site would that be? I would be really interested to hear of the use cases that cause a single powerful db server to be insufficient. Of course over a certain scale this is normal, so im not interested in a site with 50 millions users, but cases where a single DB is not the best solution.
- 16y ago
- velshin 16y agoExcellent tidbits!
- grammr 16y agoMy databases professor at my university once brought up Jim Gray's data latency analogy to illustrate the costs of using different types of storage. Basically it goes something like this: Using a register is like interacting with information in your own brain, using an L1 cache is like walking to a bookshelf in the room you're in, using an L2 cache is like walking across campus, using RAM is like going to Sacramento (we were in Berkeley), hitting the disk is like going to Pluto, and using tape drives is like going to Andromeda. I never fully registered the consequences of hitting disk until then.
- kingkilr 16y agoMy company does this, but in reverse, whenever someone is getting something to eat there's a metaphor for it. Cache means you've already got it, hitting memory means you need to go to the kitchen, disk means you need a trip to the store, regisers of course means you're actively chewing.
- iwwr 16y agoWell, sometimes cache becomes stale. Core dump and flush need no explanation :)
- Johngibb 16y agoI really like this analogy. When the article said "Numbers everyone should know", it's not really the numbers that matter, but the general relative scale of performance costs for common operations. IOTW, memorizing the nanoseconds of a cache hit isn't useful, but knowing how many times slower disk is than ram is.
- darwinGod 16y agoAn article with 'numbers everyone should know' without linking to Peter Norvig?! Here goes: http://norvig.com/21-days.html http://norvig.com/21-days.html
- mleonhard 16y agoAnother way to make unique ordered keys is to use a timestamp and append a random value, such as a random UUID. key = timestamp + UUID4
- j_random_hacker 16y agoI didn't like this idea at first, since it struck me that if for some reason you had to create many records in quick succession (e.g. some kind of bulk upload) they would essentially be placed in random order. But thinking about it, the only scenarios where you would need to do this and want to preserve the order exactly would be if all the records were coming from the same source -- in which case you could just choose a single UUID for the batch and append increasing values of a local counter to this. TL;DR: This would actually work really well I think!
- bmm6o 16y agoIsn't this exactly what the article suggests doing? Time stamp + user id + comment count.
- j_random_hacker 16y agoI love the concept of sharding counters to trade write performance for read performance! As so often happens, my understanding advances by looking at something I thought I knew the only possible answer to ("Of course a counter goes in a single variable!") and seeing that other alternatives are possible.