5 ms·
What database structure powers News.YCombinator efficiency?
As database web-dev newbie (or at least elementary level), I'm trying to get up to speed on optimization issues for my growing rdbms.
I learn a lot from studying examples, and I'm really impressed with the speed of HN's performance / data freshness. Has there been a HN post that describes the HN-news database structure behind its burgeoning service? [ I tried some search within site and via searchyc, but general terms like "database structure" yield way too much, and I've not known what terms would helpfully narrow search. ] Thanks!
- davidw 18y agoThis has some information about it: http://en.wikipedia.org/wiki/DRAM http://en.wikipedia.org/wiki/DRAM
- CalmQuiet 18y agoWell, that's a nice intro to DRAM, but I'm unclear as to its relevance to database/file structure [whether on hard disk or RAM].
- davidw 18y agoYou could always get the source code... But the speed mostly comes from being in memory (or at least that's my guess). That pretty much trumps any structure: http://news.ycombinator.com/item?id=468611 http://news.ycombinator.com/item?id=468611 Also, it's fairly slow for some operations: http://news.ycombinator.com/submitted?id=davidw http://news.ycombinator.com/submitted?id=davidw Seems to be quite slow, for instance.
- CalmQuiet 18y agoHmmm... I hadn't realized submissions retrieval would be so slow, since my 17 submissions show almost instantaneously; your hundreds do take a while. Something about the indexing for submissions doesn't work that fast, even though all in memory?
- davidw 18y agoMaybe, since the submissions stretch back over a longer time (I submit less than I comment), they're hitting the disk, and consequently slow?
- CalmQuiet 18y agoThat seems likely. But I thought I saw a thread where pg seemed to say it was all in RAM ? [Though in a fairly recent post, I seem to recall he was musing about upgrading from a 32 to a 64 system - and working with more than 4GB RAM. Can all HN comments & indexes be happening in 4GB RAM? ]
- allenbrunson 18y agopg prefers to avoid databases where possible. viaweb stored all its information in flat files, and so does hacker news.
- eatenbyagrue 18y agoHmm, that seems a little... anachronistic. From my perspective, I figure the many thousands of programming hours that have gone into making your average RDBMS fast and reliable might be worth taking advantage of...
- lacker 18y agoThere have also been many thousands of programming hours invested in making your average filesystem fast and reliable. It all depends on what you want to do. If you want relational queries a RDBMS provides a huge advantage. If you only want key-value lookup an RDBMS might be overkill.
- CalmQuiet 18y agoWow! That's amazing. I guess I need to be learning more about pros/cons of such strategies for data storage. Thanks for tip on viaweb. Wikipedia article on that led me to pg's interesting article ( http://paulgraham.com/vwplan.html http://paulgraham.com/vwplan.html ) on original 1995 viaweb business plan. I'll head off and do searches on 'viaweb'/'webgen' and data structure.
- CalmQuiet 18y agoSo it turns out that when viaweb was sold to Yahoo pg learned that Yahoo was also using flat files. But "that was 1995 and there were no free db solutions," as someone pointed out. Couple years ago http://news.ycombinator.com/item?id=14608 http://news.ycombinator.com/item?id=14608 noted that Lisp works very well with flat files:: the plot thickens. Alas, what's a baby boomer accustomed to simple PHP/MySQL solutions to do... try to wrap his head around a completely new file and language approach?
- 18y ago