9 ms·

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable

• pmoriarty 9y ago
  Any way your web page could be made viewable without requiring Javascript? I disable it in my web browsers because of security and privacy concerns, and it would be great if I could just read about your project without it.
  • akarve 9y ago
    Ah, that would be useful. In the meantime our docs are browsable without JS https://docs.quiltdata.com/ https://docs.quiltdata.com/. The articles linked at the bottom of the docs landing page are also legible without JS. Unfortunately the docs menu is not available without JS. I will look into fixing these issues. Email is feedback at quiltdata dot io if you have questions :)
    • jwilk 9y ago
      The menu on https://docs.quiltdata.com/ https://docs.quiltdata.com/ isn't visible if you have JS disabled. I had to disable CSS to be able to navigate.
• deleted 9y ago
  [deleted]
• zitterbewegung 9y ago
  This is a really impressive project. Do you think you would add data cleaning commands? How do you think you would handle datasets that are only available to academics or other restrictions? On your front page you should have the owner/dataset name instead of just the dataset name so I wouldn't have to click through to find the owner name.
  • kevinemoore 9y ago
    We've been thinking of making a Python build file to make it easier add richer transformations than the simple file->DataFrame that runs by default. We'd love suggestions and input on the types of transformations that would be most useful.
  • akarve 9y ago
    Data cleaning is so necessary. `build.yml` already supports a limited set of feature (through pandas). In addition to custom data transformations, any "out of the box" cleaning functions you'd like? In the spirit of dplyr? We've looked at e.g. scikit feature for normalization, 1-of-n encoding, etc.
    • zitterbewegung 9y ago
      How about deduplication. Also table / dictionary lookup . And string replacement and regex replacement .
      • akarve 9y ago
        Got it. If you'd like me to ping you once we have custom build hooks: aneesh at quiltdata dot io. Have you tried Luigi or Bonobo for data cleaning?
        • zitterbewegung 9y ago
          No I haven't I think I just did it myself I'll look into that .
• RangerScience 9y ago
  This is pretty neat! Is there an API or anything, so we can write support for other languages?
  • dimaryaz 9y ago
    There's a RESTful API that's used by the Python client, though it's not documented yet. If you're willing to go through the code, you can find /api/... calls in https://github.com/quiltdata/quilt/blob/master/quilt/tools/command.py https://github.com/quiltdata/quilt/blob/master/quilt/tools/c... . We're planning to write proper documentation soon.
  • akarve 9y ago
    Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel where we can support you if you want to tackle adding language bindings to Quilt. feedback at quiltdata dot io.
    • RangerScience 9y ago
      Ruby and Go, personally.
      • kevinemoore 9y ago
        The package metadata is stored in JSON so that should be pretty easy to access in either Ruby or Go. Tabular data is stored in Parquet by default. Do you know of any good libraries for reading and writing Parquet in Ruby and Go? Do either of those languages have a DataFrame-like class/struct?
• koolba 9y ago
  This looks pretty cool. Am I right to understand that this is providing a virtualized filesystem interface that dynamically loads the slices of data actually being accessed (but through a direct API rather than something generic like FUSE)? Unrelated I found a typo on your blog. Search for "seriailize" on your "Manage data like source code" post.
  • kevinemoore 9y ago
    Yes, essentially the quilt library abstracts the serialization (building packages) and deserialization (importing packages) so that user/client code doesn't need to include the mechanics of loading data (e.g., from a file system path or URL in S3).
    • amygdyl 9y ago
      I hope this is not too tangential, but I have been thinking about the best ways to make use of direct access to non volatile memories, ignoring the block and driver level and ordering the layout in your code. I suspect that your project is one which could take that direction very usefully. I would very much like to hear your reaction to this suggestion, and I would like to add if you have looked at the work at weka.io for their take on the convergence of storage. I increasingly like the idea of having the ability to use the hardware and have the management directed at the applications level where the developer is able to use intelligent measures and policies to tune their systems in discreet manner from the OS. Close cooperation will enable the collection of data to provide a valuable resource for administrators and directly benefit the pace of production development, by providing a comprehensive universal instrumentation context. I know this is almost arguing about the reason why databases should use their own on disk format and there is long history of the tradeoff involved with that. But the trouble I have with the current storage space is that the equity in the file systems is not flexible enough for the kind of smaller mixed deployment I come into contact with in the lower small business market, as example of this, CEPH or any FS which is a monolithic investment in which you are going to find the most restricted resource is management time. Additional FSs are difficult propositions to small shops. But the idea I'm looking for is the application layer should be responsible for storage management and performance tuning and follow best practice set by the software publisher learned from collected instrument data. I think the epiphany of the general operating system is nigh or even last century. Nobody is able to use large software programs in a turnkey way, making assumptions about the OS environment. I hand wave plenty saying that, but certainly I can follow up flippantly to add that a friend's experience providing contracted management to small businesses is not atypical by my experience, he joked that he loves Linux because it meant he got a clean install and nobody likely to be able to know how to mess it up. The context where I see very little leveraging of OS capabilities, particularly in the Windows Server user world, it looks like a lot of wasted effort and license expense. I beg forgiveness in advance of this facetious illustration, but in conversation with a small business Web developer recently, I cited the example of Plenty of Fish, and rhetorically asked if he knew it was a one man gig, on Windows and IIS? He was unaware of this, so I teased him that he would be forever in his first money rounds and hiring, if he had similarly accepted a bet on building such a dating site, if he kept on reading HN so much... My joke is off color sorry, but I wanted then as now to make the point where it has become all too accepted to automatically get started with a complete development stack and seek advantages in terms of the customization and deep power that is leveraging highly experienced professionals, and I worry about whether we all just do too much of this, and it's time to review the situation more broadly than my rotten humour alludes to, because the problem, if it is a problem, is much wider.
• purvis 9y ago
  Do you plan to curate the free plans? If not, how will you prevent abuse? (thinking in terms of both the service and its users)
  • deleted 9y ago
    [deleted]
  • akarve 9y ago
    Yes :) Happy to discuss in detail if you have specific types of abuse in mind. Our first line of defense is to get help from the community through downvoting useless content.
    • purvis 9y ago
      The main thing is how you are able to offer unlimited disk, when anyone can start uploading data? Some users might upload very large amounts of innocent data, others might simply abuse it as a sort of data backup system. More nefarious users might even try to distribute malware.
• Zaheer 9y ago
  Would be great if you also had the schema / file format information as well on the site.
  • akarve 9y ago
    Agreed. We're considering doing more along these lines, such as generating the Hive DDL. So you would like to browse the schema, e.g. under Contents?
    • Zaheer 9y ago
      Awesome! Yup would like to browse the schema.
  • kevinemoore 9y ago
    What type of representation would be most useful for schema information? Avro?
    • Zaheer 9y ago
      That or just a basic visual editor is fine too. Goal is to just know what I'm getting before getting it in more detail.
• RangerScience 9y ago
  Also, if we want to produce our own data set for community consumption, what are your requirements, and what kind of payout could we expect?
  • akarve 9y ago
    To start with, we can add stars (pay with prestige). Getting more into science fiction--but very possible science fiction--we can put data on the blockchain and let people transact. The data owner would get the lion's share of the transaction.
    • RangerScience 9y ago
      Hmm. I can see some ways for this to work. Awhile ago I started a project converting government voting records (both elections and congressional) into a database. Would that be interesting to you? Here's an idea: You could also host a data bounty program, and/or start a grant program for the production of these data sets. Still missing an answer to "what are you requirements"? How do you verify data quality, etc? What format(s)?
      • kevinemoore 9y ago
        Currently, we support 2 "targets" a Pandas DataFrame and a file. Files can be any format. The Quilt build logic uses Pandas to read files into DataFrames so any format Pandas can read should work in Quilt to create a DataFrame node.
      • akarve 9y ago
        We support arbitrary data formats in that Quilt falls back to a raw copy if it can't parse the file. On the columnar side (things we convert to Parquet) we support XLS, CSV, TSV, and actually anything that `pandas.read_csv` can parse. We use pandas and pyarrow for column type inference. We want to add a "data linter" that checks data against user-provided rules, and welcome such feature requests on GitHub or in our Slack Channel.
• masklinn 9y ago
  The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software) https://en.wikipedia.org/wiki/Quilt_(software)
  • akarve 9y ago
    Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?
    • xyzzy_plugh 9y ago
      I think it conflicts a bit: $ brew install quilt >/dev/null 2>&1 && which quilt /usr/local/bin/quilt I have used it extensively in automation in the past. I don't know if it's a good idea to call it "quilt" on the $PATH. Maybe "dataquilt" or "data-quilt"? "quilted"?
    • raimue 9y ago
      When choosing a name, you should check the package manager index sites of large distros for potential conflicts. For example in Debian: https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=contents&keywords=quilt https://packages.debian.org/search?suite=default&section=all...
    • masklinn 9y ago
      Quilt is a patchset manager and used a lot in linux distro spaces (it was created by one of Linux's maintainers for his own use). Any global install of your tool will conflict with its existence since the utility names are the same, which you'll get will depend on your PATH setup.
  • kevinemoore 9y ago
    Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.
• adriancooney 9y ago
  Is there planned support for any other languages than Python?
  • kevinemoore 9y ago
    Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.
    • mbeigel 9y ago
      I'd definitely be interested in looking into this but I'm on an R team. Didn't see anything on how to contribute in another language or an API to hit. Will be there be information soon?
      • akarve 9y ago
        Hi. R is really important to us and we want to add support (probably through SparklyR). If you email me I can add you to our Slack channel and we can talk through extending Quilt (aneesh at quiltdata dot io). There is a sliver in our docs but it's not complete: https://docs.quiltdata.com/basics.html https://docs.quiltdata.com/basics.html The hooks for extending build targets are here: https://github.com/quiltdata/quilt/blob/4aa6897f9e33349b777808dbdcc7b0edfe0c55e6/quilt/tools/const.py https://github.com/quiltdata/quilt/blob/4aa6897f9e33349b7778... Again it's a bit raw but we're here to help make R support easy.
• thenipper 9y ago
  What's your privacy policy for private data?
  • akarve 9y ago
    Only the owner and designated collaborators can view it. We offer private data in the cloud (S3) and are rolling out on-premise in case you want to run Quilt on your own infrastructure, in which case you control blob storage. Does that answer your question?
• dhbradshaw 9y ago
  This could be a big deal. Fantastic concept and great implementation.
  • akarve 9y ago
    Thanks. Where do you see it going in 2-5 years? Sometimes outsiders see things about the future that we haven't thought about :)
• ivotron 9y ago
  are you aware of datapackages [1]? do you plan on opensourcing the backend components? [1]: https://github.com/frictionlessdata https://github.com/frictionlessdata
  • kevinemoore 9y ago
    Yes and yes. Do you use frictionless data packages? If so, what do you like and not like? We've looked at their specs and have thought about ways we could integrate. We'd love to hear your suggestions.
  • akarve 9y ago
    Frictonless is interesting but 1) doesn't handle serialization (which is essential for performance); 2) requires users to hand annotate schemas (we think schemas should be auto-generated whenever possible).
• mbreese 9y ago
  Could you tell us how this compares with Synapse (https://www.synapse.org/ https://www.synapse.org/)?. They've been doing this for a long time and have a large presence in computational biology.
  • akarve 9y ago
    I know only a bit about Synapse. It seems like they have some valuable data. I think the biggest differences are in the culture and user experience. Quilt gets its DNA from the open/community-driven cultures of GitHub and npm. We want to make it extremely light-weight for anyone to push and share data packages. Synapse frankly seems to have a lot more features. We want to compete on simplicity. Does Synapse offer server-side filtering? That's something we're thinking about.
    • dtech81 9y ago
      > We want to compete on simplicity. Bravo, this is an excellent strategy. I've been using NPM packages to share small datasets for a little while due to the simplicity of distribution.
• mgalka 9y ago
  Love the concept! Clear value proposition for anyone who knows both worlds (code + data). Look forward to giving it a try.
• philipov 9y ago
  Is this only cloud hosted? I have been looking for this exact solution, and I'm very excited to try yours, but banks and other institutions will not tolerate storing their data in my butt. It must be totally local without ever needing an outside network. Can you provide that?
  • akarve 9y ago
    We are rolling out on-prem where the customer runs the package registry on their own infrastructure. The set up process takes a bit of customization depending on your environment. If you have Docker containers it's easier. If you email me I can get you rolling: aneesh at quiltdata dot io.
• Galanwe 9y ago
  > We built Quilt to bring package management to data. No, you built Quilt to own the data. Package managers don't make users pay if they want their own package repositories. > so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. You should spend more time with data analysts then. You would understand that cleaning, filtering and preparing the data is actually part of data analysis. I would refuse to do any work on data that was pre processed without having the exact list Of what was done to the raw data. Honestly all I see technically speaking in this project is a python program to download data frames... Woo-hoo. And for the business side, the now usual attempt to own a community of users and their data. Boring.
  • akarve 9y ago
    Namespaces are free. Public repos are free for unlimited data. Should we run a charity and also make private date free? :) We understand that cleaning/filtering/preparation are key to the analysis, and we plan to support those operations as part of package construction. The question is whether or not that work should be done repeatedly, or once for the benefit of your collaborators. You can store any kind of data in Quilt. Not just data frames. What's on Quilt today is just the beginning of what is possible. Do you also dislike GitHub? We've opened up the client source and the community for as much free data as people can publish. It seems naive to expect that we wouldn't charge for anything ever. The users always and forever control their own data, by the way.
    • Galanwe 9y ago
      I'm not criticizing the price here. It is obviously totally OK to pay for data hosting. > "Contact us to start Business or On-premise service." I DO reject the whole concept of having a "hosted-only" software. To me it makes the whole project useless. It means I cannot have my own private or confidential data. I cannot use it in my company, etc. More importantly, the data is not mine anymore. It's yours. I cannot find any place on your website that explains what happens to the data once it is uploaded! As far as I'm concerned, it means I am giving you my data, for free, without any restriction on your side. You can resell it, modify it, rebrand it, prevent me from accessing it. Since there is no mention of how the data is stored, I also cannot know if the data is encrypted on your side, or have the capability to read everything that is uploaded.