4 ms·
If large datasets are impossible to fetch from a SQL database, even more impossible is to load them to memory. From my point of view, the SQL database can stor
by pleasecalllater 9y ago
If large datasets are impossible to fetch from a SQL database, even more impossible is to load them to memory.
From my point of view, the SQL database can store the huge dataset, with its changes, and I can iterate through the results, and make lots of nice queries getting only the data I need.
- jampekka 9y agoThis depends largely on the use case. For example PostgreSQL insists of transferring all data using ASCII encoding, so eg. high sampling rate floating point sensor readings are extremely slow to fetch from the database. And not all operations can be done incrementally by iterating through the results.
- anarazel 9y ago> This depends largely on the use case. For example PostgreSQL insists of transferring all data using ASCII encoding, so eg. high sampling rate floating point sensor readings are extremely slow to fetch from the database. It doesn't. There's a binary version of the protocol. The output conversion for that is near trivial (transformation to big endian).
- jampekka 9y agoI stand corrected. The problem was due to Python drivers at the time not supporting it. Now there appears to be asyncpg.
- anarazel 9y agoYea, the driver situation around the binary protocol isn't ideal :(. A bunch of the newer drivers have it, but a lot of the old stuff doesn't. Especially things like transferring some columns in binary, others not, isn't supported enough - even though it's pretty crucial.