4 ms·
My current favorite cache for whole pages is a single sqlite file with the page source stored with brotli compression. Additional columns for any metadata you m
by muxator 5y ago
My current favorite cache for whole pages is a single sqlite file with the page source stored with brotli compression. Additional columns for any metadata you might need (URL, scraping sessionid, age). The resulting file is big (but brotli for this is even better than zstd), and having a single file is very convenient.
- jmnicolas 5y agoIt is not advised to use SQLite with several threads, are you running only a single thread?
- muxator 5y agoYou can't use SQLite from multiple threads if you are in single thread mode [0]. If you are in multi-thread mode you can use multiple connections to the DB from different threads, or if you want to share a connection then you have to serialize over it. Or you can use SERIALIZED mode. This is if you do not care writing any logic. If you do, there are other possibilities, for example using a queue and a a dedicated thread for handling database access, but I personally do not think there are many advantages to this more complicated approach. [0] https://www.sqlite.org/threadsafe.html https://www.sqlite.org/threadsafe.html
- jmnicolas 5y agoThanks, I wasn't aware of this. I'm not sure if it was old info or I "imagined" it but in any case it's good to know. I hope it wasn't true in 2019, since I painstakingly wrote a multi-threaded app that was accessing SQLite on only 1 thread!