4 ms·
Though deprecated probably in favor of more of a database/DBMS like DuckDB, the arrow plasma store holds handles to objects as a separate process: $ plasma_s
by westurner 3y ago
Though deprecated probably in favor of more of a database/DBMS like DuckDB, the arrow plasma store holds handles to objects as a separate process:
$ plasma_store -m 1000000000 -s /tmp/plasma
Arrow arrays are like NumPy arrays but they're made for zero copy e.g. IPC Interprocess Communication. There's a dtype_backend kwarg to the Pandas DataFrame constructor and read_ methods:
df = pandas.Dataframe(dtype_backend="arrow")
The Plasma In-Memory Object Store > Using Arrow and Pandas with Plasma > Storing Arrow Objects in Plasma
https://arrow.apache.org/docs/dev/python/plasma.html#storing-arrow-objects-in-plasma https://arrow.apache.org/docs/dev/python/plasma.html#storing...
Streaming, Serialization, and IPC >
https://arrow.apache.org/docs/python/ipc.html https://arrow.apache.org/docs/python/ipc.html
"DuckDB quacks Arrow: A zero-copy data integration between Apache Arrow and DuckDB" (2021)
https://duckdb.org/2021/12/03/duck-arrow.html https://duckdb.org/2021/12/03/duck-arrow.html
- Simon_O_Rourke 3y agoWhat's the latency delta of reading from a local database instance versus reading from memory - is there much additional overhead to slow things down?
- westurner 3y agoWith most databases, a `SELECT * FROM tblname` has to serialize a copy of the query result and let the ODBC or similar database driver unserialize, so the query result unavoidably gets copied at least once at query time. Plasma and e.g. DuckDB do zero copy so that there is no: unmarshal of the complete database file into RAM, table scan in order to unindexed query, and then marshal/unmarshal (serialize/deserialize) of the query result for the database driver (which could allow access to a DB/DBMS over a local pipe(), a TCP socket, HTTP(S), protobufs over HTTPS) If the memory is held in RAM by the plasma store, I'm not sure which queries are possible to service by handing an object reference to where the full non-virtual table is allocated in RAM (on only one node)? Presumably if there's no filtering or transformation in the query, and the query does not access "virtual" or "materialized" tables