4 ms·
Super cool, this is a major need. At the public protocol level, doesn't feel like much has happened here in practice since posix/http/odbc, s3, & hdfs/parquet/a
by lmeyerov 5y ago
Super cool, this is a major need. At the public protocol level, doesn't feel like much has happened here in practice since posix/http/odbc, s3, & hdfs/parquet/arrow.
I'm curious how this will play out as I don't think there's an open source impl + UI for it, and most contributors are from one company? As some relevant data points:
- the pydata oss community has been centralizing on uniform data IO for the same aws/azure/etc. systems via pyarrow, fsspec, etc., including high performance parquet support. Databricks uses some of that, so that's promising!
- ... but no pydata standardization on the control plane, esp. wrt security. fsspec normalizes on "posix acls", if I remember right: it's a real need the protocol+project can fill. Adding modern ABAC/RLS/..., sync, ... .
- ... and I can see why databricks wants to take control of the datalake protocol from aws/az/gcp (s3, ...), so this is super clever if they can keep control, and second place, ensure no one else does!
From an OSS adoption perspective, donating the protocol + self-hostable UI tiers to Apache + major non-databricks contributors would make me excited to use. Without that level of belief & health, as an architect, it feels like switching one lockin for another, so still a proprietary offer in practice
So I'm pretty excited as a user, interested in lightweight integration as a partner, and wondering what's in store for community-minded OSS governance for the confidence to go as deep as we did for Arrow like contributing core code and betting our own infra on it..