3 ms·
In Python, we use Arrow a lot for going between diff frameworks: file -> ( pandas <> cudf <> cugraph ). When we work with DB vendors, we now push them to provi
by lmeyerov 6y ago
In Python, we use Arrow a lot for going between diff frameworks: file -> ( pandas <> cudf <> cugraph ). When we work with DB vendors, we now push them to provide Arrow instead of whatever proprietary in-house thing they've been pushing for the same, e.g., slow JSON / ODBC, untyped CSV, or some ad-hoc wire protocol. Also, we use for going between our services within a server + across.
On the nodejs services + frontend JS side, there was no real equiv tool for interop -- traditional soln is slowly round-tripping through SQL/ORM or manually doing protobuf through some sort of pubsub -- so Arrow is part of how we have been taming that mess too. It'll be a longer journey for Arrow or something like it to get adopted in JS land, but I can see folks doing TypeScript and serverless wanting it for a cleaner solution to typed data, faster serialization & streaming, etc. (There is no true JS equiv of pandas nor a typed variant.) We were a bit early here b/c we wanted streaming through WebGL + OpenCL/CUDA, and while we are fans of typed data, found protobuf tooling to be too unintegrated and manual in practice.
- infinite8s 6y agoHow successful have you been in pushing database vendors to use arrow on the wire? We've started using turbodbc (to connect to SQL Server), but as I understand it the data on the wire is still in odbc's row-oriented wire format, and turbodbc is responsible for packing into arrow column oriented buffers. As you mention in your second paragraph, arrow is perfect for getting typed query results from a database server directly to a web frontend with minimal overhead. With the Transferable interface, it's even possible to zero-copy transfer the arrow data from the network buffer to a webworker.