4 ms·

>> We didn't use one. We just stored everything in files Meaning what? What kind of file formats? What did you query the data with, etc? >> It turned out the

>> We didn't use one. We just stored everything in files Meaning what? What kind of file formats? What did you query the data with, etc? >> It turned out the Yahoo accounting department used Oracle. I remember reading some Oracle advertisement (PC Magazine circa 2001) where a company like Amazon switched to them 'in one day'. It was probably the marketing department or something.

sillysaurusx 6y ago
If it's anything like Hacker News, the way it works is as follows: 1. on startup, load all items into memory from the files. 2. whenever an item is changed, save it to disk. In modern times, you can store each item as a separate .json file, for example. With this technique, there is no risk of data corruption. There is a risk of inconsistency; e.g. if I remember correctly, when you upvote an item, the vote is saved to disk, then the author's karma is incremented, and finally the author's profile is saved to disk. If the webserver dies between saving the vote and saving the karma count, the karma will no longer be the proper value. Stuff like that. Such things tend not to matter if you design it carefully, though. EDIT: I was curious what the order of operations was, so I pulled up HN's old source code: (unless (or (author user i) (and (is ip i!ip) (~editor user)) (is i!type 'pollopt)) (++ (karma i!by) (case dir up 1 down -1)) (save-prof i!by)) (wipe (comment-cache* i!id))) (push vote i!votes) (save-item i) (push (list (seconds) i!id i!by (sitename i!url) dir) (uvar user votes)) (= ((votes* user) i!id) vote) (save-votes user) (zap [firstn votewindow* _] (uvar user votes)) (save-prof user) (push (cons i!id vote) recent-votes*)))) The user's karma is incremented / decremented, then the user's profile is saved; the vote is added to an item's votes, then the item is saved; the vote is stored in the global votes table, then the votes table is saved; the vote is added to the user's "votes" list, then the user's profile is saved.
- spenczar5 6y ago
  Do all servers share a file system, or is there some sort of copying done in the background?
  - ubercow13 6y ago
    As far as I remember HN runs on a single core of a single server. I wonder whether this has changed more recently, though.
- tinus_hn 6y ago
  So you’re using the filesystem as a poor database. Is this even faster than using a real database?
tlb 6y ago
Each file held an item (a product or section page), and it was stored as an s-expression. Storing Lisp data as s-expressions is even more natural than storing Javascript data as JSON, which takes some effort if you want it to handle objects correctly.

sillysaurusx 6y ago

If it's anything like Hacker News, the way it works is as follows: 1. on startup, load all items into memory from the files. 2. whenever an item is changed, save it to disk. In modern times, you can store each item as a separate .json file, for example. With this technique, there is no risk of data corruption. There is a risk of inconsistency; e.g. if I remember correctly, when you upvote an item, the vote is saved to disk, then the author's karma is incremented, and finally the author's profile is saved to disk. If the webserver dies between saving the vote and saving the karma count, the karma will no longer be the proper value. Stuff like that. Such things tend not to matter if you design it carefully, though. EDIT: I was curious what the order of operations was, so I pulled up HN's old source code: (unless (or (author user i) (and (is ip i!ip) (~editor user)) (is i!type 'pollopt)) (++ (karma i!by) (case dir up 1 down -1)) (save-prof i!by)) (wipe (comment-cache* i!id))) (push vote i!votes) (save-item i) (push (list (seconds) i!id i!by (sitename i!url) dir) (uvar user votes)) (= ((votes* user) i!id) vote) (save-votes user) (zap [firstn votewindow* _] (uvar user votes)) (save-prof user) (push (cons i!id vote) recent-votes*)))) The user's karma is incremented / decremented, then the user's profile is saved; the vote is added to an item's votes, then the item is saved; the vote is stored in the global votes table, then the votes table is saved; the vote is added to the user's "votes" list, then the user's profile is saved.

spenczar5 6y ago
Do all servers share a file system, or is there some sort of copying done in the background?
- ubercow13 6y ago
  As far as I remember HN runs on a single core of a single server. I wonder whether this has changed more recently, though.
tinus_hn 6y ago
So you’re using the filesystem as a poor database. Is this even faster than using a real database?

tlb 6y ago

Each file held an item (a product or section page), and it was stored as an s-expression. Storing Lisp data as s-expressions is even more natural than storing Javascript data as JSON, which takes some effort if you want it to handle objects correctly.