4 ms·
Arrow Flight is a very neat project! How do you not end up with race conditions? Or data overwriting each other? Now that you have multiple writers. I think typ
by memhole 2y ago
Arrow Flight is a very neat project! How do you not end up with race conditions? Or data overwriting each other? Now that you have multiple writers. I think typically this is solved with WAL files. So maybe it's something internal to the Flight server?
- koromak 2y agoYeah I'm confused as to what its actually doing
- deleted 2y ago[deleted]
- neomantra 2y agoI do like this article a lot for showing how to do this pattern of slurping data and inserting it into a DB, in the context of Arrow Flight. The concurrency rules of DuckDB is here [1]. Reads/writes need to happen in the same process, but multiple threads can do so. This is putting a server in front of a DuckDB instance, so all read/writes are funneled there in that one process. DuckDB takes care of the concurrency within the process via MVCC. You could do the same thing with an HTTP server or other system, but this shows it with the Flight RPC framework. NOTE: I had an incorrect comment for about 2 minutes that I deleted and restructured here. Sorry if you saw that noise. [1] https://duckdb.org/docs/connect/concurrency https://duckdb.org/docs/connect/concurrency
- memhole 2y agoThanks! That makes sense now.
- mritchie712 2y agosorry, should have covered this in the post (will likely add). neomantra nailed it: https://news.ycombinator.com/item?id=42865405 https://news.ycombinator.com/item?id=42865405