4 ms·
Congrats! And a big thank you to Wes for being such a devoted community leader! We've been on quite the journey here: https://www.graphistry.com/blog/graphistr
by lmeyerov 6y ago
Congrats! And a big thank you to Wes for being such a devoted community leader!
We've been on quite the journey here: https://www.graphistry.com/blog/graphistry-2-29-5-upload-100x-more-rapids-0-13-learnrapids-com-and-more https://www.graphistry.com/blog/graphistry-2-29-5-upload-100... . Think json -> protobuf -> arrow, and paralleled in our work on parallel js -> opencl+js -> python rapids/cuda for accelerated native compute over it. The blogpost demos the ~100X bigger & faster dataset result of supporting Arrow for our uploader & RAPIDS for our parser when you can't and want us to convert for you.
Something folks miss with Arrow, IMO, is it is like google protobufs for everyone else. Arrow is not just having a nice binary format, but is also ready out-of-the-box for streaming, rich datatypes, and for larger/longer-term projects, standardized & auto self-describing schemas. If you've had to manually decipher, maintain, and update generated protobuf schemas (because you aren't google with all the internal ~protobuf tooling / integrations / infra / etc.), that should sound pretty good ;-) A lot more to do, but already way ahead of most other things, esp. in aggregate.