4 ms·

I've been using SQLite since it has the advantage of flat file storage and also the power to run SQL queries against your data. It's very light and powerful. Bu

I've been using SQLite since it has the advantage of flat file storage and also the power to run SQL queries against your data. It's very light and powerful. But I was wondering about the specific setups other folks have used? From: http://paulgraham.com/vwfaq.html http://paulgraham.com/vwfaq.html "What database did you use? We didn't use one. We just stored everything in files. The Unix file system is pretty good at not losing your data, especially if you put the files on a Netapp. It is a common mistake to think of Web-based apps as interfaces to databases. Desktop apps aren't just interfaces to databases; why should Web-based apps be any different? The hard part is not where you store the data, but what the software does. While we were doing Viaweb, we took a good deal of heat from pseudo-technical people like VCs and industry analysts for not using a database-- and for using cheap Intel boxes running FreeBSD as servers. But when we were getting bought by Yahoo, we found that they also just stored everything in files-- and all their servers were also cheap Intel boxes running FreeBSD. (During the Bubble, Oracle used to run ads saying that Yahoo ran on Oracle software. I found this hard to believe, so I asked around. It turned out the Yahoo accounting department used Oracle.)"

brlewis 20y ago
With Viaweb, the stores were islands, each dealing exclusively (or almost exclusively) with its own dataset. They would not run into locking/contention issues as they scaled the way some other apps would if they used flat files and conventional locking. The database landscape was different in 1995. There were no good free relational databases, and no cheap Oracle licenses. Lisp makes it very easy to write out any data structure. (Most Scheme implementations can only write non-circular structures easily, but Viaweb used CL). So if your app is amenable to using flat files, coding storage/retrieval is trivial. So basically, if you're writing an app with lots of separate datasets, flat files are a viable option. If you're using Lisp, they're an easy option. If it's 1995, they're a cheap option.
- mattjaynes 20y ago
  Great points, thanks ;)
- ntoshev 20y ago
  It is interesting how flat files can manage the evolution of the data structures you use? Any advice?
  - brlewis 20y ago
    If I understand your question you're wondering how to deal with older flat files when your app expects a newer data structure. If you're using s-exprs make your structure a list where the first element is a version identifier. If you aren't using s-exprs then use some other extensible format like XML.
  - vikram 20y ago
    Use hashtable for everything. So Person who has a name address age and occupation is represented as a hashtable with those fields. When you want to add a new field like emailaddress it's just another key in the hashtable. Ideally you want to code the person accessors so that they still specific externally person-name person-age the rest of the program uses these so it becomes easy to move person to a different structure if that's what you want . You want to write some function to read a hash-table and write on to disc. In lisp this is trivial, but most languages have some sort of serializer. Then a function to load all the hash tables up when you start up.

brlewis 20y ago

With Viaweb, the stores were islands, each dealing exclusively (or almost exclusively) with its own dataset. They would not run into locking/contention issues as they scaled the way some other apps would if they used flat files and conventional locking. The database landscape was different in 1995. There were no good free relational databases, and no cheap Oracle licenses. Lisp makes it very easy to write out any data structure. (Most Scheme implementations can only write non-circular structures easily, but Viaweb used CL). So if your app is amenable to using flat files, coding storage/retrieval is trivial. So basically, if you're writing an app with lots of separate datasets, flat files are a viable option. If you're using Lisp, they're an easy option. If it's 1995, they're a cheap option.

mattjaynes 20y ago
Great points, thanks ;)
ntoshev 20y ago
It is interesting how flat files can manage the evolution of the data structures you use? Any advice?
- brlewis 20y ago
  If I understand your question you're wondering how to deal with older flat files when your app expects a newer data structure. If you're using s-exprs make your structure a list where the first element is a version identifier. If you aren't using s-exprs then use some other extensible format like XML.
- vikram 20y ago
  Use hashtable for everything. So Person who has a name address age and occupation is represented as a hashtable with those fields. When you want to add a new field like emailaddress it's just another key in the hashtable. Ideally you want to code the person accessors so that they still specific externally person-name person-age the rest of the program uses these so it becomes easy to move person to a different structure if that's what you want . You want to write some function to read a hash-table and write on to disc. In lisp this is trivial, but most languages have some sort of serializer. Then a function to load all the hash tables up when you start up.