5 ms·

Curious if a potential solution would be having open, read only databases, that you could query directly, vs everyone copying the same data over and over. Kind

by sbr464 8y ago

Curious if a potential solution would be having open, read only databases, that you could query directly, vs everyone copying the same data over and over. Kind of how you don’t download Wikipedia but access what you need. I realize there are a lot of things to consider. But not even a rest api/etc, an actual database. Realize it wouldn’t scale, would cost money etc, but could be interesting

sbr464 8y ago
Thanks for the resource though! Looks interesting
hedora 8y ago
That’s what the current academic publication infrastructure was built to be. Then capitalism happened, and we have abusive monopolies instead. Historically, as empires collapse, they burn their libraries, but the usually don’t get to the copies that are kept on the outskirts.
- sbr464 8y ago
  For sure, I was thinking more simply how most of us developers etc, immediately create an abstraction (rest/graphql/sdk) vs allowing direct queries to a db. Would be interesting if libraries/knowledge/datasets could be a query away, actual free info, not a search engine/promoted/abstraction in any way.
qixxiq 8y ago
Google BigQuery does this though. They host huge public data files and then only charge for the queries.
- sbr464 8y ago
  Your right, I've tried it before and forgot about it.
- gwern 8y ago
  Also 'requester pays' on Amazon S3 buckets.
sixdimensional 8y ago
I've been thinking about the same for quite a while now. In fact, look at the overlap between GraphQL and SQL conceptually. I absolutely think there is something to this. In the past, I have used certain wide open read only genomics databases (not going to name it so it doesn't get hammered by HN). Other posters are right about services such as BigQuery but I think there's a place for an open source project here that interfaces SQL to databases through a layer that adds caching, throttling and more services on top of that. That's how you make it scale. The Dremio project (open source by the backers of Apache Arrow) has a SQL REST API that converts a standard SQL dialect/datatypes to the underlying systems. I think that's a good start and Dremio has a ton of other awesome functionality like Apache Arrow caching. Simple model is expose an expression language (even could be not SQL, like jsoniq, or other expression languages), mapper from that to SQL, web service API on top with a pluggable connector model. I say that I'm going to start an open source project around this all the time but haven't gotten the inertia to do it. Argh!
dwiel 8y ago
This is the kind of application that dat, swarm, ipfs/filecoin are aiming to support.
- dwiel 8y ago
  More specifically orbitdb [1] which is built on top of ipfs is a potential starting point. It implements a decentralized and distributed append only log data structure as well as object store and key/value lookup db. Not quite a SQL database though. [1] https://github.com/orbitdb/orbit-db/blob/master/README.md https://github.com/orbitdb/orbit-db/blob/master/README.md
Too 8y ago
How about using a database engine with built in horizontal sharding to distribute a big database to multiple users in smaller pieces, just like bittorrent but with query-capabilities. Thanks to the recent big-data nosql craze this should be easy to find off the shelf. As example i know MongoDB has all this built in, not sure it is designed with untrusted nodes in mind but otherwise fits all requirements.
- sbr464 8y ago
  I think cockroachDB has the automatic node registration/balancing? I guess you’d have to deal with the shard or data you would choose to host There’s one feature of Faunadb that stood out, the ability to create a seemingly unlimited number of nested databases, where the db assigned to you appeared to be the root/full database, with full query/write privileges. Imagine a public dataset that you could query but also update with your own findings and metadata, and choose to have it shared upstream or keep private. The engine could diff your work to avoid duplicating a copy of the data. You could sync it locally for speed if needed

by sbr464 8y ago

sbr464 8y ago

Thanks for the resource though! Looks interesting

hedora 8y ago

That’s what the current academic publication infrastructure was built to be. Then capitalism happened, and we have abusive monopolies instead. Historically, as empires collapse, they burn their libraries, but the usually don’t get to the copies that are kept on the outskirts.

sbr464 8y ago
For sure, I was thinking more simply how most of us developers etc, immediately create an abstraction (rest/graphql/sdk) vs allowing direct queries to a db. Would be interesting if libraries/knowledge/datasets could be a query away, actual free info, not a search engine/promoted/abstraction in any way.

qixxiq 8y ago

Google BigQuery does this though. They host huge public data files and then only charge for the queries.

sbr464 8y ago
Your right, I've tried it before and forgot about it.
gwern 8y ago
Also 'requester pays' on Amazon S3 buckets.

sixdimensional 8y ago

I've been thinking about the same for quite a while now. In fact, look at the overlap between GraphQL and SQL conceptually. I absolutely think there is something to this. In the past, I have used certain wide open read only genomics databases (not going to name it so it doesn't get hammered by HN). Other posters are right about services such as BigQuery but I think there's a place for an open source project here that interfaces SQL to databases through a layer that adds caching, throttling and more services on top of that. That's how you make it scale. The Dremio project (open source by the backers of Apache Arrow) has a SQL REST API that converts a standard SQL dialect/datatypes to the underlying systems. I think that's a good start and Dremio has a ton of other awesome functionality like Apache Arrow caching. Simple model is expose an expression language (even could be not SQL, like jsoniq, or other expression languages), mapper from that to SQL, web service API on top with a pluggable connector model. I say that I'm going to start an open source project around this all the time but haven't gotten the inertia to do it. Argh!

dwiel 8y ago

This is the kind of application that dat, swarm, ipfs/filecoin are aiming to support.

dwiel 8y ago
More specifically orbitdb [1] which is built on top of ipfs is a potential starting point. It implements a decentralized and distributed append only log data structure as well as object store and key/value lookup db. Not quite a SQL database though. [1] https://github.com/orbitdb/orbit-db/blob/master/README.md https://github.com/orbitdb/orbit-db/blob/master/README.md

Too 8y ago

How about using a database engine with built in horizontal sharding to distribute a big database to multiple users in smaller pieces, just like bittorrent but with query-capabilities. Thanks to the recent big-data nosql craze this should be easy to find off the shelf. As example i know MongoDB has all this built in, not sure it is designed with untrusted nodes in mind but otherwise fits all requirements.

sbr464 8y ago
I think cockroachDB has the automatic node registration/balancing? I guess you’d have to deal with the shard or data you would choose to host There’s one feature of Faunadb that stood out, the ability to create a seemingly unlimited number of nested databases, where the db assigned to you appeared to be the root/full database, with full query/write privileges. Imagine a public dataset that you could query but also update with your own findings and metadata, and choose to have it shared upstream or keep private. The engine could diff your work to avoid duplicating a copy of the data. You could sync it locally for speed if needed