3 ms·
You definitely want to store raw text in flat files and store metadata in a database. By metadata I'm not referring to only the values found in the html page's
by yogo 13y ago
You definitely want to store raw text in flat files and store metadata in a database. By metadata I'm not referring to only the values found in the html page's meta tags but other things like page hash, word count, link count, etc. It all depends on what you are doing. If it's in a database you will have a very very big file for that archives table (mongodb and mysql+innodb come to mind).
- riffraff 13y agoin a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk. At some point querying the db to check what URLs you have can become quite heavy
- deleted 13y ago[deleted]
- rb2k_ 13y agoWhat would a filesystem do better than a DB in that case? After all a filesystem is just a database who's primary keys are mostly filenames. Most of them use the same data-structures (e.g. b+trees) and especially the newer ones turn into copy-on-write systems like e.g. CouchDB I guess if you want to have backups of that table, you probably would like to keep it at a manageable size, but why not just save the html blobs in a separate table?