4 ms·

Hi Hacker News. I'm one of the founders of the Noms project and Attic Labs, the company behind it. Happy to answer any questions. In the meantime, as long as I

Hi Hacker News. I'm one of the founders of the Noms project and Attic Labs, the company behind it. Happy to answer any questions. In the meantime, as long as I've got your attention, here's a few new stuffs we've been working on since last time Noms was discussed here in August: - A prototype query language, and a demo of how to create indexes in Noms: https://www.youtube.com/watch?v=fv6_T5yaWns https://www.youtube.com/watch?v=fv6_T5yaWns - Support for merging concurrent (and potentially conflicting) changes: https://www.youtube.com/watch?v=--7dgoJBdjU https://www.youtube.com/watch?v=--7dgoJBdjU

mostafah 10y ago
Does/will Noms have proper support for multiuser sync? That’s an issue with CouchDB. I mean: instead of syncing whole the database, only syncing the parts that a user has access to, and being able to define those accesses. The standard use case for consumer apps.
- aboodman 10y ago
  Noms syncs at the level of a "value" not the entire database. Since a Noms database is a tree of values, you can sync the entire db, but you don't have to. You can also sync a single list. So yes, in principle, we can definitely do this. In practice we are missing some conveniences that would make it a really easy drop-in feature.
esfandia 10y ago
Looks really interesting! This may sound like a stupid question, but do you have any publication I can cite by any chance?
- deleted 10y ago
  [deleted]
freekh 10y ago
Wow, congrats - this looks really interesting!
ComodoHacker 10y ago
What about ACID? How far do you plan to divert from it?
- Normal_gaussian 10y ago
  This is the most important question in my mind. If it isn't ACID it needs to make a very strong case for itself to even be played with by most DBA's, including myself.
  - Ericson2314 10y ago
    Depending on details, content addressing could make it very hard to not be ACID. Here's to hoping.
    - aboodman 10y ago
      Yes, it kind of falls out of content-addressing. See: https://news.ycombinator.com/item?id=12722023 https://news.ycombinator.com/item?id=12722023
- senand 10y ago
  More specifically, what isolation guarantee do you make? Are writes linearizable? What consistency guarantees do you claim? How well do you think you will do when reviewed by aphyr (https://aphyr.com https://aphyr.com)?
  - aboodman 10y ago
    You should understand that Noms is doing something quite a lot simpler than the systems that are usually reviewed on Aphyr. Noms doesn't manage its own storage - it relies on an underlying key/value store that must provide strongly consistent reads for at least one key. In other words, we delegate most of the hard part to somebody else. With that all said... Currently our intent is that: - Transactions that read and write from a single dataset have strong serializability - Transactions that read from multiple datasets and write to a single dataset have snapshot isolation - Transactions that write to multiple datasets aren't possible In the future, we will probably allow additional configuration, such that, e.g., one could choose snapshot isolation within a dataset for additional concurrency, or strong serializability for transactions that span datasets.
rwmj 10y ago
There's quite a dramatic claim on the website, "merge [...] changes efficiently and correctly days, weeks, or years later." How does that work? For example if you have two records saying userid 3's name is "ann" and userid 3's name is "jane", I don't see how you could merge those without extra information or human input.
- deleted 10y ago
  [deleted]
- aardvark179 10y ago
  It's a very hard problem to solve. The best you can really manage is to automatically merge non-conflicting changes, and defining what is a conflicting change will require some extra knowledge if you have multiple tables with relations between them. For extra fun try doing this with geographic data and try merging geometry changes correctly.
- striking 10y ago
  Well, hopefully you're not just using an incrementing user id field. Twitter's Snowflake format is much closer to "definitely unique". So then you have a very simple set, which you could very simply merge with other sets. So that might help you get started with merging those pieces of data. But I agree, there are certain kinds of data that you can't so efficiently merge. Your best bet is to try to adapt it into some sort of known, proven CRDT. (Dunno how you do that with this database, haven't really read up on it.)
- aboodman 10y ago
  The claim on the website is not meant to suggest that any two changes can be automatically merged. I will try to clarify that. The world contains logical conflicts because physical constraints mean that processes can operate disconnected from each other. No database can wave that away. Noms will automatically, efficiently, and correctly merge changes that don't logically conflict. Which is a pretty cool and unique property in a database. If any conflicts are found, there is a callback to user software to perform a resolution. More info in the documentation: https://godoc.org/github.com/attic-labs/noms/go/merge https://godoc.org/github.com/attic-labs/noms/go/merge
amelius 10y ago
How do you deal with access control? I can imagine that it would be necessary in many applications to have parts of the database only accessible to some people, and that access rules could be complicated and also depend on values in the database itself.
- aboodman 10y ago
  We haven't implemented access control yet. See https://github.com/attic-labs/noms/issues/1183 https://github.com/attic-labs/noms/issues/1183 for one idea.
chuhnk 10y ago
What was the process like for raising funding for an OSS project? Especially one that was pre-launch.
dman 10y ago
Any pointers to design docs or papers that inspired Noms?
seego 10y ago
Do you have any plans to provide C/C++ bindings?
- aboodman 10y ago
  We want to. It would be nice if we could just use cgo but it's not complete. So we need to build some kind of simplified API to Noms that can be exported via cgo. I created a bug for this just now: https://github.com/attic-labs/noms/issues/2718 https://github.com/attic-labs/noms/issues/2718 Please feel free to get involved there.

mostafah 10y ago

Does/will Noms have proper support for multiuser sync? That’s an issue with CouchDB. I mean: instead of syncing whole the database, only syncing the parts that a user has access to, and being able to define those accesses. The standard use case for consumer apps.

aboodman 10y ago
Noms syncs at the level of a "value" not the entire database. Since a Noms database is a tree of values, you can sync the entire db, but you don't have to. You can also sync a single list. So yes, in principle, we can definitely do this. In practice we are missing some conveniences that would make it a really easy drop-in feature.

esfandia 10y ago

Looks really interesting! This may sound like a stupid question, but do you have any publication I can cite by any chance?

deleted 10y ago
[deleted]

freekh 10y ago

Wow, congrats - this looks really interesting!

ComodoHacker 10y ago

What about ACID? How far do you plan to divert from it?

Normal_gaussian 10y ago
This is the most important question in my mind. If it isn't ACID it needs to make a very strong case for itself to even be played with by most DBA's, including myself.
- Ericson2314 10y ago
  Depending on details, content addressing could make it very hard to not be ACID. Here's to hoping.
  - aboodman 10y ago
    Yes, it kind of falls out of content-addressing. See: https://news.ycombinator.com/item?id=12722023 https://news.ycombinator.com/item?id=12722023
senand 10y ago
More specifically, what isolation guarantee do you make? Are writes linearizable? What consistency guarantees do you claim? How well do you think you will do when reviewed by aphyr (https://aphyr.com https://aphyr.com)?
- aboodman 10y ago
  You should understand that Noms is doing something quite a lot simpler than the systems that are usually reviewed on Aphyr. Noms doesn't manage its own storage - it relies on an underlying key/value store that must provide strongly consistent reads for at least one key. In other words, we delegate most of the hard part to somebody else. With that all said... Currently our intent is that: - Transactions that read and write from a single dataset have strong serializability - Transactions that read from multiple datasets and write to a single dataset have snapshot isolation - Transactions that write to multiple datasets aren't possible In the future, we will probably allow additional configuration, such that, e.g., one could choose snapshot isolation within a dataset for additional concurrency, or strong serializability for transactions that span datasets.

rwmj 10y ago

There's quite a dramatic claim on the website, "merge [...] changes efficiently and correctly days, weeks, or years later." How does that work? For example if you have two records saying userid 3's name is "ann" and userid 3's name is "jane", I don't see how you could merge those without extra information or human input.

deleted 10y ago
[deleted]
aardvark179 10y ago
It's a very hard problem to solve. The best you can really manage is to automatically merge non-conflicting changes, and defining what is a conflicting change will require some extra knowledge if you have multiple tables with relations between them. For extra fun try doing this with geographic data and try merging geometry changes correctly.
striking 10y ago
Well, hopefully you're not just using an incrementing user id field. Twitter's Snowflake format is much closer to "definitely unique". So then you have a very simple set, which you could very simply merge with other sets. So that might help you get started with merging those pieces of data. But I agree, there are certain kinds of data that you can't so efficiently merge. Your best bet is to try to adapt it into some sort of known, proven CRDT. (Dunno how you do that with this database, haven't really read up on it.)
aboodman 10y ago
The claim on the website is not meant to suggest that any two changes can be automatically merged. I will try to clarify that. The world contains logical conflicts because physical constraints mean that processes can operate disconnected from each other. No database can wave that away. Noms will automatically, efficiently, and correctly merge changes that don't logically conflict. Which is a pretty cool and unique property in a database. If any conflicts are found, there is a callback to user software to perform a resolution. More info in the documentation: https://godoc.org/github.com/attic-labs/noms/go/merge https://godoc.org/github.com/attic-labs/noms/go/merge

amelius 10y ago

How do you deal with access control? I can imagine that it would be necessary in many applications to have parts of the database only accessible to some people, and that access rules could be complicated and also depend on values in the database itself.

aboodman 10y ago
We haven't implemented access control yet. See https://github.com/attic-labs/noms/issues/1183 https://github.com/attic-labs/noms/issues/1183 for one idea.

chuhnk 10y ago

What was the process like for raising funding for an OSS project? Especially one that was pre-launch.

dman 10y ago

Any pointers to design docs or papers that inspired Noms?

seego 10y ago

Do you have any plans to provide C/C++ bindings?

aboodman 10y ago
We want to. It would be nice if we could just use cgo but it's not complete. So we need to build some kind of simplified API to Noms that can be exported via cgo. I created a bug for this just now: https://github.com/attic-labs/noms/issues/2718 https://github.com/attic-labs/noms/issues/2718 Please feel free to get involved there.