4 ms·

This post is a follow on to: http://news.ycombinator.com/comments?id=10001 http://news.ycombinator.com/comments?id=10001 Paul Buchheit @ startup school told us

by npk 20y ago

This post is a follow on to: http://news.ycombinator.com/comments?id=10001 http://news.ycombinator.com/comments?id=10001 Paul Buchheit @ startup school told us to put our heavily accessed, small data, in hash-tables kept in RAM. Paul Graham mentions viaweb used the FreeBSD fs as its database. It seems to me as if these two applications are very different, and cover a broad spectrum of requirements. Is there somewhere I can read more about this subject? I mean, there is a whole industry developed around databases, they must add value... The typical response to such a query is: what are you trying to do? Well, given the two applications listed above, what would you have recommended? Me? I'd have recommended a database server. Why would you not have? How can I find out more? How do I achieve your level of enlightenment? n

ustrip 20y ago
If I would have heavily accessed 'small' data I'd keep them in SQLite database.
- npk 20y ago
  Suppose the application was gmail? Obviously, one big sqlite DB is not going to cut it. Someone, somewhere on here, mentioned having a sqlite DB for each user. I can't imagine this is what gmail does. It may have worked in the case of viaweb, but I'm not sure.
  - randallsquared 20y ago
    If I were building a gmail, I'd be using a maildir-derivative.
  - mattjaynes 20y ago
    Here's the discussion I think you were thinking of: http://news.ycombinator.com/comments?id=10001 http://news.ycombinator.com/comments?id=10001 Based on this article: http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/ http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/
chris_l 20y ago
"I mean, there is a whole industry developed around databases, they must add value..." The traditional RDBMS adds random access by computed criteria, transaction safety, multi-threadedness, a standard query language, ... If you don't need these, you're likely to suffer a performance hit / programming overhead compared to a system that does not offer them. A filesystem only offers random access by one key (pathname). Also, the more specialised your needs are, the further down the stack you should start your own code. For a simple web application with a bit of user data, I would probably go with a DB. For your own search engine, you probably want something a little more customized, building on top of a standard fs.
- neilc 20y ago
  Another reason to not use a DBMS for a search engine is that typical implementations of transaction-oriented SQL databases are a terrible fit for the performance requirements of a search engine. For example, search engines don't need concurrent writes or ACID transactions, or SQL-like query language; search engines want to optimize for large-scale updates, not small, random writes; typical DBMS index structures (btree) don't work well for search engine indices. Eric Brewer has an interesting paper that lays out an architecture for a search engine that is consistent with DBMS design principles, but differs significantly in the implementation details: http://www.cs.berkeley.edu/~brewer/papers/SearchDB.pdf http://www.cs.berkeley.edu/~brewer/papers/SearchDB.pdf

by npk 20y ago

ustrip 20y ago

If I would have heavily accessed 'small' data I'd keep them in SQLite database.

npk 20y ago
Suppose the application was gmail? Obviously, one big sqlite DB is not going to cut it. Someone, somewhere on here, mentioned having a sqlite DB for each user. I can't imagine this is what gmail does. It may have worked in the case of viaweb, but I'm not sure.
- randallsquared 20y ago
  If I were building a gmail, I'd be using a maildir-derivative.
- mattjaynes 20y ago
  Here's the discussion I think you were thinking of: http://news.ycombinator.com/comments?id=10001 http://news.ycombinator.com/comments?id=10001 Based on this article: http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/ http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/

chris_l 20y ago

"I mean, there is a whole industry developed around databases, they must add value..." The traditional RDBMS adds random access by computed criteria, transaction safety, multi-threadedness, a standard query language, ... If you don't need these, you're likely to suffer a performance hit / programming overhead compared to a system that does not offer them. A filesystem only offers random access by one key (pathname). Also, the more specialised your needs are, the further down the stack you should start your own code. For a simple web application with a bit of user data, I would probably go with a DB. For your own search engine, you probably want something a little more customized, building on top of a standard fs.

neilc 20y ago
Another reason to not use a DBMS for a search engine is that typical implementations of transaction-oriented SQL databases are a terrible fit for the performance requirements of a search engine. For example, search engines don't need concurrent writes or ACID transactions, or SQL-like query language; search engines want to optimize for large-scale updates, not small, random writes; typical DBMS index structures (btree) don't work well for search engine indices. Eric Brewer has an interesting paper that lays out an architecture for a search engine that is consistent with DBMS design principles, but differs significantly in the implementation details: http://www.cs.berkeley.edu/~brewer/papers/SearchDB.pdf http://www.cs.berkeley.edu/~brewer/papers/SearchDB.pdf