6 ms·
Hi all. I'm one of the creators of Noms. Happy to answer any questions!
by aboodman 10y ago
Hi all. I'm one of the creators of Noms. Happy to answer any questions!
- sigzero 10y agoThere's a question about what you consider "large data" above.
- Kinnard 10y agoYou probably want to prepend a "Show HN: "
- the_duke 10y agoAre you aware of https://github.com/bup/bup https://github.com/bup/bup? It's a git file format based backup incremental backup tool. Do you also use rolling checksums (like bup) to prevent re-storing data when only a few bytes have changed? --> What are the major use cases you imagine for noms? I've read the section on Github, but can you give some specific examples that you see as good use cases?
- aboodman 10y agoYes, we credited bup in various places, such as the design overview: https://github.com/attic-labs/noms/blob/master/doc/intro.md https://github.com/attic-labs/noms/blob/master/doc/intro.md We were also heavily influenced by camlistore (which I hacked on for awhile), irmin, ipfs, and others who have done a lot of interesting work in this space. We do use rolling checksums, but I think we have done some novel work here: https://github.com/attic-labs/noms/blob/master/doc/intro.md#prolly-trees-probabilistic-b-trees https://github.com/attic-labs/noms/blob/master/doc/intro.md#...
- pinko 10y agoFYI, the "HTTP protocol" link[1] in the intro page above is broken. PS: I love the name "Prolly Tree". [1] <https://github.com/attic-labs/noms/blob/master/datas/database_server.go> https://github.com/attic-labs/noms/blob/master/datas/databas...
- aboodman 10y ago:) Thanks, we like it too.
- ripa 10y agobup is focused on backups. Would you say that noms is useful as a backup utility? Do you have any mechanisms for securing integrity, specifically repairing the store in case of inconsistencies? Is there any plans to support any data retention policy/functionality?
- aboodman 10y agoNoms should be useful as a backup utility, but I'd say it's especially useful for backing up data which is not files. Think about backing up data which you only have access to via API. You can take the JSON output of an API and drop it into Noms, then do the same thing tomorrow, and Noms will automatically deduplicate the data as well as give you a nice structured API to read and interact with it. We have an example of this here: https://github.com/attic-labs/noms/tree/master/samples/js/flickr https://github.com/attic-labs/noms/tree/master/samples/js/fl... but it's not working atm due to a bug introduced right before launch. You can look at the code though.
- pinko 10y agoAnother question to help me better understand the tool: who do you see as your competitors, technically? What do you see as viable alternatives to Noms, but worse? (Or better!) I do see that you were inspired by git, but clearly your use-cases are different.
- aboodman 10y agoGit is a competitor. It is fairly common to check data (e.g., csv or json files) into Git today. However, this falls down pretty rapidly. In order to get reasonable diffs, the data has to be sorted, and line-oriented. Also Git just doesn't scale well to larger repos or individual objects. Otherwise, we see the competitors as the way that people distribute data today - custom APIs, zip files full of CSV, etc.
- jrsnyder 10y agoHow would you say Noms compares to Datomic[1]? Both projects are working on the same idea of representing a database as tree of commits over time. From my quick inspection, it looks like Noms shows some focus towards working in multiple branches, whereas Datomic, at least in its marketing materials, just talks about preserving a single timeline. [1]: http://www.datomic.com/benefits.html http://www.datomic.com/benefits.html
- aboodman 10y agohttps://news.ycombinator.com/item?id=12213886 https://news.ycombinator.com/item?id=12213886
- carterehsmith 10y ago> Also Git just doesn't scale well to larger repos or individual objects. I guess the above means that Noms does scale to larger repos... Do you guys have any numbers, comparisons, benchmarks against git? If so, it would be useful to include in the readme.md as it would be kind of a big thing, and quite attractive to many people.
- bra-ket 10y agodoes noms understand sql and can it do joins?
- pinko 10y agoThe linked page says, as its last line, "many important features are not yet implemented including a query system."
- daveloyall 10y agoIt doesn't support queries. It's a datastore, not a database. This is from their FAQ. https://github.com/attic-labs/noms/blob/master/doc/faq.md https://github.com/attic-labs/noms/blob/master/doc/faq.md
- fizzbatter 10y agoI mean, by their definition it is a database, but i can understand your usage. Then again, they both say it is a database, and in your link, they say it "isn't quite there yet", so /shrug heh.
- ah- 10y agoIt'd be fantastic to combine something like noms with https://arrow.apache.org/ https://arrow.apache.org/ and then use Spark/Drill/Impala to query it.
- amelius 10y agoHow does Noms compare to ipfs? https://ipfs.io/ https://ipfs.io/
- aboodman 10y agoFirst, IPFS is awesome. IPFS is essentially providing a globally decentralized filesystem. Noms is providing (or hopes to provide) a database. By database, I mean: - small individual records - efficient queries, updates, and range scans - ability to support complex queries - ability to enforce structural data validity These are all things that IPFS could eventually grow to support, but in order to do it, I think it would have to grow into or layer something like noms on top.
- alexatkeplar 10y agoWhy would you exclusively support schema inference, rather than also allowing users to manually specify their schemas? Schema inference is very difficult to do correctly and safely, especially with small initial samples of instances (source: work on https://github.com/snowplow/schema-guru https://github.com/snowplow/schema-guru).
- aboodman 10y agoThere's a little bit of terminology overloading going on here. In Noms every value has a type. It's an immutable system, so this type just is. The type of `42` is `Number`. The type of `"foobar"` is `String`. The type of `[42,44]` is `List<Number>`. And if you add "foo" to that list, the type becomes `List<Number|String>`. We don't try to infer a general database schema from a few instances of data. We just apply this aggregation up the tree and report the result. That all said, we do want to eventually add schema _validation_, by which I mean the ability to associate a type with a dataset and have the database enforce that any value committed to the dataset is compatible with that type (following subtyping rules).
- kragen 10y agoIt sounds like Noms is dynamically typed, rather like SQLite — types are associated with values, not (just) with datasets. The difference is that SQLite (like Python or JS) only types leaf/atomic data, while you're also typing aggregate data. Is that right? Are you planning on writing complete reference documentation at some point, like https://www.sqlite.org/limits.html https://www.sqlite.org/limits.html, https://www.sqlite.org/howtocorrupt.html https://www.sqlite.org/howtocorrupt.html, https://www.sqlite.org/lang.html https://www.sqlite.org/lang.html, https://docs.python.org/2/reference/index.html https://docs.python.org/2/reference/index.html, and https://golang.org/ref/spec https://golang.org/ref/spec? Or is using Noms going to be more of a UTSL kind of thing? The documentation I've found so far seems to be purely tutorial and introductory in nature. (I'm really glad you're writing Noms, by the way. There's an enormous need for it.)
- alexatkeplar 10y agoRight - the challenge is that with dynamic typing and without schema validation, it's incredibly easy to break any strongly typed client/consuming application. You think you are dealing with a `List<Number>`, you have Go/Java/Haskell/whatever apps which are consuming that in a strongly typed fashion using their idiomatic record types, and then suddenly a user accidentally sends in a single value which turns the tree of values into a `List<Number|String>`, and all your consuming apps break. Given that schema validation ("does this instance match this type?") is simpler to implement than schema inference ("what is the type of this instance?"), it's surprising to me to deliver inference first...
- zimbatm 10y agoHi, is there any place where you talk about how merging of two different branches works?
- ioquatix 10y agoHow would NOMS work for massive amounts of geo-temporal data? with many inserts and queries but few (zero) updates? Efficient queries on geo-temporal keys are useful.