kevinemoore

6 ms·

1.
by kevinemoore 7y ago
The search is powered by Elasticsearch. We want to support the full power of elastic’s query language, but so far haven’t found a great way to prevent overly intense (or malicious) queries and still allow arbitrary queries—love to hear from
2.
by kevinemoore 7y ago
We'll look into that for sure!
3.
by kevinemoore 7y ago
+1 for everything Aneesh said, but I also wanted to add that the public cloud offers opportunities in data sharing that academia hasn't yet provided, specifically the ability for collaborators to bring their code to the data. I posted
4.
by kevinemoore 7y ago
There's no doubt that local processing is a lot cheaper than the cloud for a lot of workloads. That's a very interesting pattern--publishing "fake" (perhaps safe or anonymized) data online along with code to spur researc
5.
by kevinemoore 7y ago
The list of datasets is pretty dynamic. Would an API call work as a URL? (assuming it could return a CSV/TSV/etc.)?
6.
by kevinemoore 7y ago
Thanks! That's really interesting feedback. We hadn't thought of that.
7.
by kevinemoore 7y ago
Maybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators
8.
by kevinemoore 7y ago
Hmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?
9.
by kevinemoore 7y ago
In general, a data publisher simply creates an S3 bucket in their account and sets the access control to allow public read. In the specific case of the AWS Registry of Open Data, data providers create a clean AWS account to hold the data (t
10.
by kevinemoore 7y ago
Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc . Looking forward to checking
11.
by kevinemoore 7y ago
I just want to give a plug for sharing data in the public cloud and S3 in particular. Jed Sundwall (AWS Global Open Data Lead) sums it up really well: "The cloud completely changes the dynamic for sharing data. When data is shared in t
12.
by kevinemoore 7y ago
Sirix sounds like a very interesting system! Is it similar in its internal structure to noms ( https://github.com/attic-labs/noms ) or Dolt ( https://github.com/liquidata-inc/dolt )? I think S3 is a g
13.
by kevinemoore 7y ago
Aneesh's co-founder here. I just want to add a word of thanks to Jed Sundwall and the AWS Registry of Open Data. The support of AWS makes publishing data at this scale possible. I also want to thank Jackson Brown and everyone else who
14.
by kevinemoore 8y ago
Quilt co-founder here, no, Quilt doesn’t use tar or zip. Each package version has a manifest that specifies the set of items it contains. Each item in the collection is stored and transported separately. Items are identified by their hash
15.
by kevinemoore 9y ago
We definitely imagine saving edits as creating new versions of the same package. The most common pattern we've heard is adding more dataframes or files to a package as new results are generated. But, we can certainly imagine other tran
16.
by kevinemoore 9y ago
That would be awesome! We're happy to help, but we only know a little bit of R. We've been looking at Sparklyr in case it might help read Parquet into R.
17.
by kevinemoore 9y ago
No, no relation.
18.
by kevinemoore 9y ago
try: example.sales() it's shorthand for: example.sales._data()
19.
by kevinemoore 9y ago
Once the interfaces are mostly settled, we plan to open-source the server so that other organizations can run Quilt registries. If you know of non-profit data-indexes that Quilt should work with or organizations who might be interested in r
20.
by kevinemoore 9y ago
Thanks for describing the problem. That's really interesting. We can certainly aggregate counts of downloads and installs across versions in Quilt. I'll definitely look into providing DOIs within Quilt and see if it's somethi
21.
by kevinemoore 9y ago
No, there's no charge for bandwidth. The most common uses so far are users installing datasets locally, which caches the data at the destination or running batch jobs in ECS/EC2, which doesn't accrue charges on AWS.
22.
by kevinemoore 9y ago
The package metadata is stored in JSON so that should be pretty easy to access in either Ruby or Go. Tabular data is stored in Parquet by default. Do you know of any good libraries for reading and writing Parquet in Ruby and Go? Do either o
23.
by kevinemoore 9y ago
Yes and yes. Do you use frictionless data packages? If so, what do you like and not like? We've looked at their specs and have thought about ways we could integrate. We'd love to hear your suggestions.
24.
by kevinemoore 9y ago
Currently, we support 2 "targets" a Pandas DataFrame and a file. Files can be any format. The Quilt build logic uses Pandas to read files into DataFrames so any format Pandas can read should work in Quilt to create a DataFrame nod
25.
by kevinemoore 9y ago
We'd love to include DOIs for each version of datasets. I think it's feasible for us to do that, but we haven't scoped out how hard or expensive that will be. In the meantime, if you have a way of creating DOIs from URLs, cre
26.
by kevinemoore 9y ago
Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.
27.
by kevinemoore 9y ago
Good suggestion! From what we've heard from academic users, they'll want a DOI for a specific version, e.g., data from a particular paper or journal article. Any thing else we should watch out for?
28.
by kevinemoore 9y ago
We're definitely planning to make column names and types more easily accessible and searchable. It'd be great to learn more about what information and metadata would be most useful to you. We also have an on-prem version we'r
29.
by kevinemoore 9y ago
Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.
30.
by kevinemoore 9y ago
We've definitely been looking at that! Are you using frictionless data packages now?

More ›