4 ms·
Do you think you'll add local file support in the future? Also, do you have any plans on making the reading of a source parallel? For example, connectorx uses a
by kipukun 3y ago
Do you think you'll add local file support in the future? Also, do you have any plans on making the reading of a source parallel? For example, connectorx uses an optional partition column to read chunks of a table concurrently. Cool how it's abstracted.
- karakanb 3y agoI am working on file support right now as a destination to begin with. I believe I should get local files as well as S3-compatible sources going by tonight. Reading the sources in parallel is an interesting idea, I'll definitely take a look at it. ingestr supports incremental loads by a partitioned column, but there's no parallelized partition reading at the moment. Thanks a lot for your comment!
- adawg4 3y agoI second this!
- karakanb 3y agoI have just released v0.1.2 which supports CSV destinations with the URI format `csv://path/to/file.csv`, hope that's helpful!