5 ms·

Strawman marketing alert: "The most common way to share data today is to post CSV files on a website". Maybe there are a bunch of people that still do that some

by tlb 10y ago

Strawman marketing alert: "The most common way to share data today is to post CSV files on a website". Maybe there are a bunch of people that still do that somewhere, but if so, they ain't early adopters of decentralized database technology and so not your target customers. It's always better to talk about what your most likely customers are doing now.

spraak 10y ago
Yeah, that totally made me laugh. Good point about the target audience.
archgoon 10y ago
This is actually extremely common. For example, if you browse the UC Irvine ML datasets https://archive.ics.uci.edu/ml/index.html https://archive.ics.uci.edu/ml/index.html You'll find that many are in csv format. If you do a search on data.gov http://catalog.data.gov/dataset#sec-res_format http://catalog.data.gov/dataset#sec-res_format You'll see that it's about as popular as JSON. Also, the World Health Organization http://www.who.int/tb/country/data/download/en/ http://www.who.int/tb/country/data/download/en/ Also, many of the datasets at kaggle are in csv format. https://www.kaggle.com/datasets https://www.kaggle.com/datasets And this isn't that surprising, it's human readable, and gets the job done, and zipping will give decent compression. I'm not sure who you think the target market for this would be, but I'm sure that if it's an efficient local format, you could probably get the ML crowd on board.
- aboodman 10y ago
  Right. A shocking amount of public data is distributed this way. Also, we routinely talk to developers who complain about the difficulty of consuming data snapshots from partners, parsing it, trying to understand how it has changed since last time, etc. With high value datasets, people frequently build an API to combat these problems. But it's hard to design a good API, and even if you succeed, it has to be secured, documented, scaled, and maintained indefinitely.
  - spotman 10y ago
    So if nom takes off would you see 'download a nom dataset by clicking here' or would it be 'use this hostname to sync the nom to your own computer' Or would it be a dsn sort of and you just instantiate a client and your on your way? Or, some combo?
    - aboodman 10y ago
      In some glorious future world you might see things like: ``` <a href="http://www.who.int/tb/country/data/download/en/::case-data/by-country">Download http://www.who.int/tb/country/data/download/en/::case-data/b... the Data</a> ```
      - kragen 10y ago
        Hmm, if you want people to be able to link to Noms datasets on the web, maybe you should switch to using URLs to name the datasets, instead of a two-part identifier with an URL separated from a dataset name by a "::"? Darcs and Git seem to get by more or less with URLs and relative URLs; do you think that cold work for Noms too? The super REST harmonious way to do this would be to define a new media-type for Noms databases with a smallish document that links to the component parts. Like torrent files, but using URLs (maybe relative URLs) instead of SHA1 hashes for the components, maybe?
        aboodman 10y ago
        This is a good point. We never thought of these strings as URLs, but there are places where it would be nice to use them that only want URLs (the href attribute, for example). The way we have it now is nice in that any valid URL can be used to locate a database. I am loathe to restrict that. Interesting point though - thank you!
        kragen 10y ago
        Sure, I hope the ideas are useful! As some other commenters have said, if you just use # instead of ::, I think the problem goes away?
spotman 10y ago
Yea and then a paragraph about git under it, made me double take to think maybe they meant CVS, but then realized they really did mean CSV. If this is supposed to replace CSV cool, but there is a lot of ways to cross that bridge. Curious however, what the target use case is. Is it a format, or a database , or both?
- spotman 10y ago
  After more reading it kind of sounds likes better couch db? And cool if so, but the CSV analogy is really confusing. When I think of CSV I think of a ghetto data exchange format that I can send or receive to a less technical person. As I understand nom, it does not sound like its for non technical people.
napoleond 10y ago
Is it possible that their target market is not current users of decentralized DBs? At first glance, it strikes me as a solution for people storing something like scientific data sets rather than application data. In which case, posting CSV files on a website is pretty much best-case scenario. EDIT: Although, in the "scientific data set" scenario, I'm not sure how much value there would be in storing version history.
- jameshart 10y ago
  Right, this looks like it has far more immediate value for storing data which will be collaboratively mutated, so: a company directory, a knowledgebase, a CRM datastore... For large scale datasets, I'd be looking at GIS data maybe.
deleted 10y ago
[deleted]
penetrarthur 10y ago
And Dropbox is just an FTP with SVN.

by tlb 10y ago

spraak 10y ago

Yeah, that totally made me laugh. Good point about the target audience.

archgoon 10y ago

This is actually extremely common. For example, if you browse the UC Irvine ML datasets https://archive.ics.uci.edu/ml/index.html https://archive.ics.uci.edu/ml/index.html You'll find that many are in csv format. If you do a search on data.gov http://catalog.data.gov/dataset#sec-res_format http://catalog.data.gov/dataset#sec-res_format You'll see that it's about as popular as JSON. Also, the World Health Organization http://www.who.int/tb/country/data/download/en/ http://www.who.int/tb/country/data/download/en/ Also, many of the datasets at kaggle are in csv format. https://www.kaggle.com/datasets https://www.kaggle.com/datasets And this isn't that surprising, it's human readable, and gets the job done, and zipping will give decent compression. I'm not sure who you think the target market for this would be, but I'm sure that if it's an efficient local format, you could probably get the ML crowd on board.

aboodman 10y ago
Right. A shocking amount of public data is distributed this way. Also, we routinely talk to developers who complain about the difficulty of consuming data snapshots from partners, parsing it, trying to understand how it has changed since last time, etc. With high value datasets, people frequently build an API to combat these problems. But it's hard to design a good API, and even if you succeed, it has to be secured, documented, scaled, and maintained indefinitely.
- spotman 10y ago
  So if nom takes off would you see 'download a nom dataset by clicking here' or would it be 'use this hostname to sync the nom to your own computer' Or would it be a dsn sort of and you just instantiate a client and your on your way? Or, some combo?
  - aboodman 10y ago
    In some glorious future world you might see things like: ``` <a href="http://www.who.int/tb/country/data/download/en/::case-data/by-country">Download http://www.who.int/tb/country/data/download/en/::case-data/b... the Data</a> ```
    - kragen 10y ago
      Hmm, if you want people to be able to link to Noms datasets on the web, maybe you should switch to using URLs to name the datasets, instead of a two-part identifier with an URL separated from a dataset name by a "::"? Darcs and Git seem to get by more or less with URLs and relative URLs; do you think that cold work for Noms too? The super REST harmonious way to do this would be to define a new media-type for Noms databases with a smallish document that links to the component parts. Like torrent files, but using URLs (maybe relative URLs) instead of SHA1 hashes for the components, maybe?
      - aboodman 10y ago
        This is a good point. We never thought of these strings as URLs, but there are places where it would be nice to use them that only want URLs (the href attribute, for example). The way we have it now is nice in that any valid URL can be used to locate a database. I am loathe to restrict that. Interesting point though - thank you!
        kragen 10y ago
        Sure, I hope the ideas are useful! As some other commenters have said, if you just use # instead of ::, I think the problem goes away?

spotman 10y ago

Yea and then a paragraph about git under it, made me double take to think maybe they meant CVS, but then realized they really did mean CSV. If this is supposed to replace CSV cool, but there is a lot of ways to cross that bridge. Curious however, what the target use case is. Is it a format, or a database , or both?

spotman 10y ago
After more reading it kind of sounds likes better couch db? And cool if so, but the CSV analogy is really confusing. When I think of CSV I think of a ghetto data exchange format that I can send or receive to a less technical person. As I understand nom, it does not sound like its for non technical people.

napoleond 10y ago

Is it possible that their target market is not current users of decentralized DBs? At first glance, it strikes me as a solution for people storing something like scientific data sets rather than application data. In which case, posting CSV files on a website is pretty much best-case scenario. EDIT: Although, in the "scientific data set" scenario, I'm not sure how much value there would be in storing version history.

jameshart 10y ago
Right, this looks like it has far more immediate value for storing data which will be collaboratively mutated, so: a company directory, a knowledgebase, a CRM datastore... For large scale datasets, I'd be looking at GIS data maybe.

deleted 10y ago

[deleted]

penetrarthur 10y ago

And Dropbox is just an FTP with SVN.