3 ms·
I'm not talking about streaming live data, I'm talking about streaming query results if you re-read my comment. If you're reading many MB's or GB's of data fro
by crazygringo 6y ago
I'm not talking about streaming live data, I'm talking about streaming query results if you re-read my comment.
If you're reading many MB's or GB's of data from a database, it's a lot more performant to stream it from the database directly into your local data structure, rather than rely on default processing of query results as a single chunk which will be a lot worse for memory and speed.
- tomnipotent 6y agoStreaming aggregation relies on knowing the problem beforehand, so it's pointless in the context of the day-to-day work of analysts where most of the project is spent figuring out the right questions to ask. If I need to stream data because the size of data is a constraint, I'm either working on the 0.1% of projects that require it or on an analytical product rather than an analytical project.