This has been a great and informative write-up. I learnt of Arrow when Wes and Hadley announced Feather. I think Arrow was still 0.4.0 At the end of 2016, we h

This has been a great and informative write-up. I learnt of Arrow when Wes and Hadley announced Feather. I think Arrow was still 0.4.0 At the end of 2016, we had a project at work where we were trying to migrate a credit modeling architecture away from SAS to something different. There were enterprise "SAP HANA" hammer+nail in the room (I'm biased because I think HANA is rubbish vaporware), and directors who thought "open source is immature and unstable". They gave me the lead on the project, and I started out with a naive Pandas port, where I hit the data size issue. I got more resources and we made an R attempt, similar perf. We made a PySpark impl, serde and high latency were the problem. Wrote another version in Scala; better but similar. Given the nature of the business, we canned the project. I learnt a lot though. Data serialisation is a pain. When we started using Feather, we were able to speed up some of our implementations by bigly magnitudes. At the time, Parquet interop was sketchier, but much better than CSV. I tried taking advantage of Spark and R by building a bridging service where I could run the fast parts of each service, dump data to Feather files, and have the next service pick up processing. R hasn't had a good prod/serve API story, so I didn't get good results. I used gRPC for Python and Scala, and tried some REST thing as a proxy for R. What I could have benefited from was an IPC. Throughout Arrow releases, and the various integrations over the past year and a bit, a lot of what we struggled with was solved. The recent improvements (as I'm keenly interested in Arrow for computing) make me excited. I'm starting a project where I'll be building web-based viz and reporting. I'm planning on using Arrow, undecided on whether to use JVM or Rust on the back end given that JDBC to Arrow will be a thing in the next release, or Diesel.rs to Arrow from DataFusion. Nonetheless, exciting future for data engineers on the horizon!