3 ms·
Could share some more on the data pipelines you mention? What kind of data store do you use for this purpose? We are reading a lot of the same data from disk to
by Cwizard 4y ago
Could share some more on the data pipelines you mention? What kind of data store do you use for this purpose? We are reading a lot of the same data from disk to do transformation (joins) with new data, I’ve been looking for something that might speed this process up but would like to keep the setup small and simple.
- dvt 4y agoI've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things. [1] https://arrow.apache.org/ https://arrow.apache.org/