5 ms·

Welcome to Apache OpenDAL

86 points by Kinrany 3y ago

NuSkooler 3y ago
This looks very promising. I love that Apache went straight to Rust vs the usual Java or similar.
sbt567 3y ago
Very interesting. Is this a kind of "programmable rclone"?
- tison 3y ago
  Sounds likely. The core part of OpenDAL is a Rust crate that provides fs-like APIs over different storage backends, but we also investigate providing other interfaces like a CLI. We have an experimental binary named `oli`[1]. You're welcome to start a discussion[2] to share how you use rclone and we may find it fit in OpenDAL's scope :D [1] https://github.com/apache/incubator-opendal/tree/main/bin/oli https://github.com/apache/incubator-opendal/tree/main/bin/ol... [2] https://github.com/apache/incubator-opendal/discussions https://github.com/apache/incubator-opendal/discussions
  - calvinmorrison 3y ago
    Sounds like a job for 9p
andrewstuart 3y ago
A programmer friend and I had a running joke that it was impossible to work out what any given Apache project actually does.
- crancher 3y ago
  "OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way." Generally agree, but this one seems clear enough.
  - TeMPOraL 3y ago
    The quote by itself is so generic it's useless. That's my peeve with marketing advice this day. Describing what the product will do for user in emotional/vague terms carries zero information relevant to evaluate the product and make a use/purchase decision. It's either treating the customer as a generic unsophisticated idiot, that can't understand what the product actually does, and just needs to be told it'll make them happy, or it's a pure manipulative play. Either way, this is not the style OSS projects should pick up.
    - bryanrasmussen 3y ago
      >The quote by itself is so generic it's useless. to me it means - you will be able to write code that allows you to save and retrieve data from various online services that offer data storage without having to know the particulars of each one of those services apis. Is this what it does? And if so I wouldn't think that was useless information, at least it is useful enough for me to determine should I Read on?
      - Kinrany 3y ago
        It is accurate but not concise. The explanation should start with the purpose of the project described in about one sentence, then elaborate by adding context that may or may not be known to the reader.
        xuanwo 3y ago
        Could you offer some guidance? This slogan is the clearest explanation I've come across. I'm open to better suggestions! Thanks in advance.
        Kinrany 3y ago
        I linked the overview page because the image on it explains the project better than any description I could find. One of the issues I see is mixing "what" and "why". The "what": an abstraction for accessing object storage services, implemented as client libraries in multiple languages plus utilities built on top. The "why", the problem to be solved: there are many services that provide key-value storage with values being possibly very large; it is reasonable to want to write code that works with any of these services but there is no common interface to write this code against. Caveat, I may very well be misunderstanding the project. Questions I have: - how does it compare to object_store - what is the philosophy of the project; in what way is it opinionated about the design of object storage services
- speedgoose 3y ago
  I agree it took me some time to understand what it does. It seems to be a Rust library/crate allowing filesystem like operations on many storage systems, as long as you can do basic key/values things. From your filesystem, to redis, S3, sqlite, etcd… They also provide Python and NodeJS bindings. The operations are apparently (not supported by all storage systems): stat read write create_dir delete copy rename list scan presign blocking
- tison 3y ago
  It's hard to me ever on OpenDAL also. But now I have an image that you can use it either as: 1. A drop-in replacement to S3 SDK (AWS's Rust SDK is ...); 2. A quick way to support your users to configure different cloud storage they have (release you from supporting multiple cloud OSS backends with different SDKs). A few increasing DB projects use OpenDAL in the second way, like Databend, GreptimeDB, QuestDB, RisingWave, etc.
  - wey-gu 3y ago
    So does project like Mozilla sccache. - https://github.com/mozilla/sccache https://github.com/mozilla/sccache
- deleted 3y ago
  [deleted]
janejeon 3y ago
This is really interesting. For a project I was building I figured out that having a sort of "universal access layer" for all types of storage was a requirement, and this looks like it's right up my alley.
- xuanwo 3y ago
  I'm an OpenDAL committer. Thank you for your interest! I'm happy to assist with integrating OpenDAL into your project.
elsadek 3y ago
This reminds me of Mule ESB when it was in its beginning.
- latchkey 3y ago
  How it ended: In 2018, Mulesoft was acquired by Salesforce for $6.5 billion in a cash-and-stock deal.
prabir 3y ago
Was surprised that rust didn’t have vfs libraries. Created my own async-vfs crate but now using opendal for a Nextcloud alternative that I have been working in rust.
- xuanwo 3y ago
  Thanks for using OpenDAL!
j-a-a-p 3y ago
Slightly related: https://xkcd.com/927/ https://xkcd.com/927/
contravariant 3y ago
Just because the author of this piece seems to be looking at this post, some minor nitpicks regarding spelling/phrasing: > GCS has native JSON API which more powerful Missing a verb. > OpenDAL needs to implement features in zero cost way which means: "in a zero cost way" is probably better. > What OpenDAL does? What does OpenDAL do? > Free to zero cost This is a tricky one, right now it sounds a bit weird because you're turning 'to zero cost' into a verb, suggesting users would have to 'zero cost' things themselves. The problem is that 'free' and 'at zero cost' mean the same thing, so it's hard to use both in the same phrase. Free of cost could work.
- xuanwo 3y ago
  Thanks a lot! I will fix those spelling issues ASAP.
  - xuanwo 3y ago
    Addressed in https://github.com/apache/incubator-opendal/pull/3805 https://github.com/apache/incubator-opendal/pull/3805
- verdverm 3y ago
  Did you notice the "edit this page" link at the bottom? I always put those buttons on my sites and get a lot of these minor edits like this on GitHub. I'm curious in the user research sense
  - contravariant 3y ago
    I hadn't noticed it actually, though it's also a bit inconvenient for me to write a git commit at the moment. Hope this helps.
    - verdverm 3y ago
      Yea, very normal response to dropping someone into an edit page on GitHub. That option works really well for typos I put three links on my site pages 1. edit page 2. Open issue for page (autofill some details like page name in the subject) 3. Open issue for project Sounds like you put my 2 here
AdrianoKF 3y ago
Does this intend to fill a similar spot in the Rust ecosystem as fsspec (https://filesystem-spec.readthedocs.io/en/latest/ https://filesystem-spec.readthedocs.io/en/latest/) does for Python, or am I getting the wrong idea?
- cjalmeida 3y ago
  I was thinking about this as well. Arrow/PyArrow integration is very useful.
fishpen0 3y ago
It’s a shame they have committed to never supporting Bigquery and other cloud provider services. We really need something like this at my org but there’s no compelling reason to move all of analytics off of BQ nor is there a good reason to add yet another copy layer between the apps and the analytics tools We literally have an entire set of teams building a DAL internally and its basically a forever project while existing app engineers just keep doubling down on table federation and other stuff that breaks lineage and ownership models
- erickj 3y ago
  OOC what's the scale of data storage that you're using with bq?
- sanderjd 3y ago
  I don't really grok how the API of this - which appears to be very file-oriented - would be adapted to querying a sql interface like bigquery. How would that look?
asah 3y ago
[flagged]
- jpitz 3y ago
  Looking at the kind of APIs they are supporting, this appears to be aiming at filesystem-like behavior. If I was building such a thing, I would leave SQL out of scope as well.
gavinray 3y ago
This is essentially Hasura without a networked API or the ability to make queries that do cross-datasource joins
lijok 3y ago
Looks great. Very impressed with the vision (although language could be improved), and the "wont implement" decisions made. I would provide a bit of challenge on tenet 2 however. Supporting "storage_class" for S3 is a compromise that clearly has to be made, and yet, it appears you're not realizing you'll have to make other compromises like it in the future. I would suggest a storage-specific configuration class for each storage backend, and then you wont need to make these arbitrary concessions. The power of OpenDAL will be its standardized data API, not its simplified configuration. I'm also not convinced that the project should implement OpenDAL Gateway. I cannot see how it will provide anyone any value other than making things more confusing.

86 points by Kinrany 3y ago

NuSkooler 3y ago

This looks very promising. I love that Apache went straight to Rust vs the usual Java or similar.

sbt567 3y ago

Very interesting. Is this a kind of "programmable rclone"?

tison 3y ago
Sounds likely. The core part of OpenDAL is a Rust crate that provides fs-like APIs over different storage backends, but we also investigate providing other interfaces like a CLI. We have an experimental binary named `oli`[1]. You're welcome to start a discussion[2] to share how you use rclone and we may find it fit in OpenDAL's scope :D [1] https://github.com/apache/incubator-opendal/tree/main/bin/oli https://github.com/apache/incubator-opendal/tree/main/bin/ol... [2] https://github.com/apache/incubator-opendal/discussions https://github.com/apache/incubator-opendal/discussions
- calvinmorrison 3y ago
  Sounds like a job for 9p

andrewstuart 3y ago

A programmer friend and I had a running joke that it was impossible to work out what any given Apache project actually does.

crancher 3y ago
"OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way." Generally agree, but this one seems clear enough.
- TeMPOraL 3y ago
  The quote by itself is so generic it's useless. That's my peeve with marketing advice this day. Describing what the product will do for user in emotional/vague terms carries zero information relevant to evaluate the product and make a use/purchase decision. It's either treating the customer as a generic unsophisticated idiot, that can't understand what the product actually does, and just needs to be told it'll make them happy, or it's a pure manipulative play. Either way, this is not the style OSS projects should pick up.
  - bryanrasmussen 3y ago
    >The quote by itself is so generic it's useless. to me it means - you will be able to write code that allows you to save and retrieve data from various online services that offer data storage without having to know the particulars of each one of those services apis. Is this what it does? And if so I wouldn't think that was useless information, at least it is useful enough for me to determine should I Read on?
    - Kinrany 3y ago
      It is accurate but not concise. The explanation should start with the purpose of the project described in about one sentence, then elaborate by adding context that may or may not be known to the reader.
      - xuanwo 3y ago
        Could you offer some guidance? This slogan is the clearest explanation I've come across. I'm open to better suggestions! Thanks in advance.
        Kinrany 3y ago
        I linked the overview page because the image on it explains the project better than any description I could find. One of the issues I see is mixing "what" and "why". The "what": an abstraction for accessing object storage services, implemented as client libraries in multiple languages plus utilities built on top. The "why", the problem to be solved: there are many services that provide key-value storage with values being possibly very large; it is reasonable to want to write code that works with any of these services but there is no common interface to write this code against. Caveat, I may very well be misunderstanding the project. Questions I have: - how does it compare to object_store - what is the philosophy of the project; in what way is it opinionated about the design of object storage services
speedgoose 3y ago
I agree it took me some time to understand what it does. It seems to be a Rust library/crate allowing filesystem like operations on many storage systems, as long as you can do basic key/values things. From your filesystem, to redis, S3, sqlite, etcd… They also provide Python and NodeJS bindings. The operations are apparently (not supported by all storage systems): stat read write create_dir delete copy rename list scan presign blocking
tison 3y ago
It's hard to me ever on OpenDAL also. But now I have an image that you can use it either as: 1. A drop-in replacement to S3 SDK (AWS's Rust SDK is ...); 2. A quick way to support your users to configure different cloud storage they have (release you from supporting multiple cloud OSS backends with different SDKs). A few increasing DB projects use OpenDAL in the second way, like Databend, GreptimeDB, QuestDB, RisingWave, etc.
- wey-gu 3y ago
  So does project like Mozilla sccache. - https://github.com/mozilla/sccache https://github.com/mozilla/sccache
deleted 3y ago
[deleted]

janejeon 3y ago

This is really interesting. For a project I was building I figured out that having a sort of "universal access layer" for all types of storage was a requirement, and this looks like it's right up my alley.

xuanwo 3y ago
I'm an OpenDAL committer. Thank you for your interest! I'm happy to assist with integrating OpenDAL into your project.

elsadek 3y ago

This reminds me of Mule ESB when it was in its beginning.

latchkey 3y ago
How it ended: In 2018, Mulesoft was acquired by Salesforce for $6.5 billion in a cash-and-stock deal.

prabir 3y ago

Was surprised that rust didn’t have vfs libraries. Created my own async-vfs crate but now using opendal for a Nextcloud alternative that I have been working in rust.

xuanwo 3y ago
Thanks for using OpenDAL!

j-a-a-p 3y ago

Slightly related: https://xkcd.com/927/ https://xkcd.com/927/

contravariant 3y ago

Just because the author of this piece seems to be looking at this post, some minor nitpicks regarding spelling/phrasing: > GCS has native JSON API which more powerful Missing a verb. > OpenDAL needs to implement features in zero cost way which means: "in a zero cost way" is probably better. > What OpenDAL does? What does OpenDAL do? > Free to zero cost This is a tricky one, right now it sounds a bit weird because you're turning 'to zero cost' into a verb, suggesting users would have to 'zero cost' things themselves. The problem is that 'free' and 'at zero cost' mean the same thing, so it's hard to use both in the same phrase. Free of cost could work.

xuanwo 3y ago
Thanks a lot! I will fix those spelling issues ASAP.
- xuanwo 3y ago
  Addressed in https://github.com/apache/incubator-opendal/pull/3805 https://github.com/apache/incubator-opendal/pull/3805
verdverm 3y ago
Did you notice the "edit this page" link at the bottom? I always put those buttons on my sites and get a lot of these minor edits like this on GitHub. I'm curious in the user research sense
- contravariant 3y ago
  I hadn't noticed it actually, though it's also a bit inconvenient for me to write a git commit at the moment. Hope this helps.
  - verdverm 3y ago
    Yea, very normal response to dropping someone into an edit page on GitHub. That option works really well for typos I put three links on my site pages 1. edit page 2. Open issue for page (autofill some details like page name in the subject) 3. Open issue for project Sounds like you put my 2 here

AdrianoKF 3y ago

Does this intend to fill a similar spot in the Rust ecosystem as fsspec (https://filesystem-spec.readthedocs.io/en/latest/ https://filesystem-spec.readthedocs.io/en/latest/) does for Python, or am I getting the wrong idea?

cjalmeida 3y ago
I was thinking about this as well. Arrow/PyArrow integration is very useful.

fishpen0 3y ago

It’s a shame they have committed to never supporting Bigquery and other cloud provider services. We really need something like this at my org but there’s no compelling reason to move all of analytics off of BQ nor is there a good reason to add yet another copy layer between the apps and the analytics tools We literally have an entire set of teams building a DAL internally and its basically a forever project while existing app engineers just keep doubling down on table federation and other stuff that breaks lineage and ownership models

erickj 3y ago
OOC what's the scale of data storage that you're using with bq?
sanderjd 3y ago
I don't really grok how the API of this - which appears to be very file-oriented - would be adapted to querying a sql interface like bigquery. How would that look?

asah 3y ago

[flagged]

jpitz 3y ago
Looking at the kind of APIs they are supporting, this appears to be aiming at filesystem-like behavior. If I was building such a thing, I would leave SQL out of scope as well.

gavinray 3y ago

This is essentially Hasura without a networked API or the ability to make queries that do cross-datasource joins

lijok 3y ago

Looks great. Very impressed with the vision (although language could be improved), and the "wont implement" decisions made. I would provide a bit of challenge on tenet 2 however. Supporting "storage_class" for S3 is a compromise that clearly has to be made, and yet, it appears you're not realizing you'll have to make other compromises like it in the future. I would suggest a storage-specific configuration class for each storage backend, and then you wont need to make these arbitrary concessions. The power of OpenDAL will be its standardized data API, not its simplified configuration. I'm also not convinced that the project should implement OpenDAL Gateway. I cannot see how it will provide anyone any value other than making things more confusing.