10 ms·

Noms – A versioned, forkable, syncable database

399 points by jaytaylor 10y ago

rpedela 10y ago
Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
- aeharding 10y ago
  If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
  - chubot 10y ago
    Which ones are you thinking of? The nice thing about git is that it doesn't really require much of the server. I run my own "git hosting" with linode, apt-get, and ssh. Most of the solutions for immutable big data don't have that level of convenience, as far as I know. Anything which requires a lot of sysadmin and ops work will eventually become a commercial cloud service... git on the other hand is useful by itself, and good companies can obviously be built on top of it as well.
    - TylerE 10y ago
      Most binary formats don't diff well.
  - rpedela 10y ago
    No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
    - emmelaich 10y ago
      They are so different it's hard to know what you mean by "basically the same thing"
      - rpedela 10y ago
        Yes there are significant differences in terms of feature, implementation, etc but they have the same goal: manage source code versions.
        emmelaich 10y ago
        That's how Git is used but I think that by design, it is foremost a userlevel append-only content-addressable file system. The difference between this and a version control system is a major part of why the UI is so awful.
  - rspeer 10y ago
    Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of: * Convert the versioned data to tab-separated values * COPY it into Postgres every time * Hope Postgres can act immutable enough even though it wasn't designed to be The closest I've come to improving this situation was Kyoto Cabinet (unusable license) and rolling my own damn hashtable (it worked okay but adding new kinds of indexes was just unmaintainable, there's a reason databases should be made by experts).
    - ykler 10y ago
      I was curious about Camlistore and Tahoe-LAFS at one point. I didn't investigate them very thoroughly, however. How do they compare?
    - deleted 10y ago
      [deleted]
    - sjezewski 10y ago
      (Disclaimer - I work at pachyderm) http://pachyderm.io http://pachyderm.io Pachyderm is git for data. We work hard to make sure we can store data of different types (binary, text, json) efficiently. We also work hard to give you good mechanisms to read the data in a distributed way. I'd be curious how this suits your purposes.
      - rspeer 10y ago
        Just started looking at Pachyderm. While I can see how a git-based filesystem can help with some use cases, does it do any kind of indexing at all? I see that the FAQ recommends exporting the data from Pachyderm into PostgreSQL, which leaves me where I am now.
- AvenueIngres 10y ago
  So.. CRDTs?
  - aboodman 10y ago
    https://github.com/attic-labs/noms/blob/master/doc/faq.md#why-dont-you-just-use-crdts https://github.com/attic-labs/noms/blob/master/doc/faq.md#wh...
    - ergl 10y ago
      Those reasons are a bit outdated, you can certainly have add-and-remove crdts with or-sets, and you can do so with almost no garbage. One of the Tomtom engineers explained one approach at Strangeloop last month: https://www.youtube.com/watch?v=veeWamWy8dk https://www.youtube.com/watch?v=veeWamWy8dk
- aboodman 10y ago
  Noms performs diff and merge in time proportional to size of the diff. The size of the source data is not really relevant. One way to think of Noms is that is an index optimized for computing diffs. Here's a screencast that shows off Noms diffing things fast: https://www.youtube.com/watch?v=Zeg9CY3BMes https://www.youtube.com/watch?v=Zeg9CY3BMes
- dwhitena 10y ago
  I definitely agree here. As a data scientist, sometimes it seems like we are in the wild west as far as reproducibility and versioning of our analyses. This seems like an interesting project that tackles some of the data versioning stuff. However, I believe that, at least in data science, we need data versioning closely tied to the analyses themselves for complete reproducibility. That is, we need the versioning tied to the inputs/outputs of data pipeline stages, such that we can reproduce pipeline runs at any time and incrementally improve and run pipelines based on diffs in data. As mentioned elsewhere in the comments, Pachyderm (http://pachyderm.io/ http://pachyderm.io/) does exactly this. Working both as git for data, but also enabling data pipelining and analyses with the data versioning.
niftich 10y ago
Open-source tech like this is nice. This could be used to build a distributed document editing application, for example. Or any application where you want to spin off multiple instances and reconcile the data later. EDIT: At least one team is investigating layering Noms on top of IPFS [1]. I guess the idea would be to construct something similar to GitTorrent [2]; layering various version-controlled datastores on various p2p protocols could result in several viable architectures. [1] https://github.com/attic-labs/noms/issues/2123#issuecomment-245162820 https://github.com/attic-labs/noms/issues/2123#issuecomment-... [2] https://github.com/cjb/GitTorrent https://github.com/cjb/GitTorrent
sohilv 10y ago
A couchbase killer.
tominous 10y ago
Adam Leventhal (DTrace, OpenZFS) took a look at building a FUSE filesystem on Noms using Go. http://dtrace.org/blogs/ahl/2016/08/09/nomsfs/ http://dtrace.org/blogs/ahl/2016/08/09/nomsfs/ https://news.ycombinator.com/item?id=12255450 https://news.ycombinator.com/item?id=12255450
- orblivion 10y ago
  Sounds like what you'd want for a self-hosted Dropbox clone. I wonder what Syncthing, for instance, uses for reconciling differences on different clients.
  - davidron 10y ago
    Syncthing doesn't reconcile differences. Instead a copy of the file is created. Syncthing does recognize conflicts. When a file has been modified on two devices simultaneously, one of the files will be renamed to <filename>.sync- conflict-<date>-<time>.<ext>. The device which has the larger value of the first 63 bits for his device ID will have his file marked as the conflicting file. Note that we only create sync-conflict files when the actual content differs. https://docs.syncthing.net/users/faq.html https://docs.syncthing.net/users/faq.html
  - deleted 10y ago
    [deleted]
marknadal 10y ago
Noms is a great example of the power of decentralized database technology, the interesting research that goes into such systems, and wonderful documentation to browse. I do want to note some tradeoffs with Content-Addressed and Append-Only systems, as my work on a similar project ( an Open Source Firebase, https://github.com/amark/gun https://github.com/amark/gun ) made me move away from those ideas (even though they are great ideas). - Content-Addressed stores are going to revolutionize data integrity and efficiency. But they do have a trade off, it makes it a lot harder to read the data if you do not already know the data you are trying to read! The bottom of the repo metions for instance that a query system has not yet been built. From my experience the reason why is because it is difficult to build query systems on Content-Addressed stores, which is a tradeoff from all the gains you can get from it. - Append-Only gives you rich features like offline-first support and (if implemented) lovely things like rewind/fastforward data time travel. All very cool. However, do not forget that this then also makes it difficult for you to retrieve the latest whole snapshot of your data. So you are not going to get the read performance that you could. But the only possible way for us as a community, and people playing around with databases, can figure out what the best system is - is for people to build and experiment. Which is part of the reason why Nom is so cool. It is an invitation to others to actually join, play, and experiment with database technology in an open and encouraging environment. That is incredibly valuable and needed!
- aboodman 10y ago
  1. We have prototyped basic query functioanlity already (https://www.youtube.com/watch?v=fv6_T5yaWns https://www.youtube.com/watch?v=fv6_T5yaWns) and Noms was designed from the beginning to support efficient indexes and range scans. So I'm not sure why it would be harder for us to support a query language than any other database. 2. It's true that content addressing can exacerbate data locality which can hurt read performance. However, there are thing you can do to get a lot of that back.
duck 10y ago
Previous discussion from back in August: https://news.ycombinator.com/item?id=12211754 https://news.ycombinator.com/item?id=12211754
100ideas 10y ago
Looks farther along than http://dat-data.com/ http://dat-data.com/, another commendable distributed VCS for data. One distinction is that dat provides additional utilities for querying and compositing the data structures represented in any csv, json, and yaml files that stores.
- 100ideas 10y ago
  One of the other design goals of Dat is to support continually-divergent forking, which they perceive as being useful for communities of analysts processing common datasets but to ultimately different ends. Of course, you never have to merge forks in git, but in it's current form they (dat devs) say that it's not really ideal.
haalcion3 10y ago
I've been wanting to use something like this. But... * It's a big jump from relational or noSQL DB's, so there aren't (m)any adapters that I can see for it for JPA, ActiveRecord, etc. * I'd really like to see a benchmark for each noms implementation compared to postgres, mysql, oracle, and mssql server, if there is a way to do apples-to-apples. * "noms" is unfortunately is really bad for SEO because noms is a common word in French. If it could be nomsdb or nomnomnoms or something less exactly French, that'd be better. It's going to be tough to find support online easily otherwise. * SQL compatibility. * Fault tolerance (how easily does it corrupt), HA, mirroring, full/partial replication, sharding, archival, partial history truncation, etc. It seems a little like a dolphin jumping into a pool of hungry sharks. It might be more evolved and more capable in some ways, but it's going to get its ass handed to it on speed and lack of features. Still- I can't wait to try it.
- Ericson2314 10y ago
  With content addressing, you may loose data but never get corrupted data.
- actuallyalys 10y ago
  >It seems a little like a dolphin jumping into a pool of hungry sharks. It might be more evolved and more capable in some ways, but it's going to get its ass handed to it on speed and lack of features. I'm inclined to agree for large, centralized databases, but I wonder if this would be a good fit for places where sqlite is used? This seems like it could be a good foundation for situations where you want to sync information without a central server, like between devices. An Access/Filemaker clone built on top of this would be cool, too.
WayneBro 10y ago
How is it that you have 2 reference implementations, written in 2 different cross platform environments, yet there is no support for Windows? Why would I use this if I can't use it everywhere?
- alsetmusic 10y ago
  Not everyone uses Windows. I will use it because it supports macOS and Linux, the two platforms that I use every day. Not everyone has the same needs.
- colemickens 10y ago
  ... well, from the link... "Noms is supported on Mac OS X and Linux. You can compile a Windows build from source, and it usually works, but isn't officially supported." Also, supporting Windows is often a pain in the ass compared to Linux/Mac. I don't fault them for not supporting it officially, especially in the project's life.
  - perlgeek 10y ago
    To be fair, Mac support is also a huge pain, unless you happen to have a Mac. Just getting a machine to test on is expensive. If you look for Mac OS VMs in the cloud, you find they start at $1 an hour (https://www.macincloud.com/ https://www.macincloud.com/) or around $80 / month (http://xcloud.me/pricing-signup/ http://xcloud.me/pricing-signup/). Compare that to around $5 / month for a Linux VM. And then you have to go the whole dance of getting xcode and homebrew just to have a development environment. It's been a few years since I used Mac OS X to develop stuff, but it wasn't intuitive at all back then.
    - huxley 10y ago
      Mac hosting doesn't tend to be VMs largely because of Apple's licensing rules for macOS (which is a big hassle, no question) so they'll naturally be more expensive than a $5 virtualized Linux VM. But Macincloud has ~$20/month plans (with 8GB of RAM which isn't a $5 option for most Linux or Windows VMs) and there are several others going for between $30-50, so you hardly have to go with $80/month.
deleted 10y ago
[deleted]
ReAzem 10y ago
Their logo is a squirrel giving an invisible blowjob
- a3n 10y ago
  It's an otter, floating on its back.
- thght 10y ago
  ROFL Seriously, down voted for laughing?
- aboodman 10y ago
  You say that like it's a bad thing.
- hossbeast 10y ago
  HN needs a +10 button
- Xorlev 10y ago
  Cannot unsee.
mahyarm 10y ago
I hope there is a prune option to delete very old commits.
- aboodman 10y ago
  There isn't yet, but there could be (ala shallow clone in Git).
aboodman 10y ago
Hi Hacker News. I'm one of the founders of the Noms project and Attic Labs, the company behind it. Happy to answer any questions. In the meantime, as long as I've got your attention, here's a few new stuffs we've been working on since last time Noms was discussed here in August: - A prototype query language, and a demo of how to create indexes in Noms: https://www.youtube.com/watch?v=fv6_T5yaWns https://www.youtube.com/watch?v=fv6_T5yaWns - Support for merging concurrent (and potentially conflicting) changes: https://www.youtube.com/watch?v=--7dgoJBdjU https://www.youtube.com/watch?v=--7dgoJBdjU
- mostafah 10y ago
  Does/will Noms have proper support for multiuser sync? That’s an issue with CouchDB. I mean: instead of syncing whole the database, only syncing the parts that a user has access to, and being able to define those accesses. The standard use case for consumer apps.
  - aboodman 10y ago
    Noms syncs at the level of a "value" not the entire database. Since a Noms database is a tree of values, you can sync the entire db, but you don't have to. You can also sync a single list. So yes, in principle, we can definitely do this. In practice we are missing some conveniences that would make it a really easy drop-in feature.
- esfandia 10y ago
  Looks really interesting! This may sound like a stupid question, but do you have any publication I can cite by any chance?
  - deleted 10y ago
    [deleted]
- freekh 10y ago
  Wow, congrats - this looks really interesting!
- 10y ago
kccqzy 10y ago
I think there is a very similar library in Haskell called project m36. Here's its github page on transactions: https://github.com/agentm/project-m36/blob/master/docs/transaction_graph_operators.markdown https://github.com/agentm/project-m36/blob/master/docs/trans...
benjismith 10y ago
Nice. I'm super excited about this! I've hand-rolled something a lot like this already for the Shaxpir backend, but it would be really nice to have a well-engineered database that already supports this kind of model, out of the box.
jbverschoor 10y ago
Weird choice for the uri: http://localhost:8000::dbname http://localhost:8000::dbname Why not http://localhost:8000/dbname http://localhost:8000/dbname ?
- lost_my_pwd 10y ago
  Wow, I missed that. That violates the URI spec, assuming the author(s) were intending to use a URI. https://tools.ietf.org/html/rfc3986 https://tools.ietf.org/html/rfc3986 (see "3.3. Path").
  - drostie 10y ago
    Right, it intentionally violates the URI spec by appending something to the end of it. The data structure they're storing has a natural pair-of-structures at its top level: data Database = InMemory | LevelDB Path | ViaHTTP URL newtype DataSet = DS Text newtype Hash = Hash Text data Accessor = AccessDS DataSet | AccessValue (Either DataSet Hash) Path type DBAccessor = (Database, Accessor) They elected to basically encode a DBAccessor above as a string which you can split on "::", with the URL above being stored on the left in the case of the ViaHTTP databases.
    - aboodman 10y ago
      The string wasn't originally intended to be a URI, but I've been subsequently convinced that it would be useful for it to be one. We'll change it eventually.
- Kiro 10y ago
  https://news.ycombinator.com/item?id=12212967 https://news.ycombinator.com/item?id=12212967
kfk 10y ago
Have you had a look to the finance world? Git for data seems to be something we in finance really need, especially the possibility of seeing all the changes and of reconciliating things.
- barrkel 10y ago
  diff for financial data, with attendant workflow for breaks, is an already existing whole market segment. Duco, the startup I work for, is tackling it as a service.
jimktrains2 10y ago
I've been working on some syncing addressbook, calendar, password manager, and notes applications. My idea was to use mdns to announce presence and git to sync, but this might be (more) useful
- aboodman 10y ago
  Noms should definitely be more useful in that scenario. We have some customers who were using Git the way you describe and replaced it with Noms and have been very happy with the results.
angel-manuel 10y ago
I really like this, I've always thought that git needed to support diff modes different from textline-based because even if this is fit for most programming languages what you really what is to see differences between ASTs (take into account those absurd change counts when just changing the indentation or imagine a normal diff of LISP source). Maybe there's some way of replacing git with noms to get there(even if it may be killing flies with cannonballs)
- aboodman 10y ago
  If you take a language parser and pipe the resulting AST into Noms, you basically get something like codeq (http://blog.datomic.com/2012/10/codeq.html http://blog.datomic.com/2012/10/codeq.html).
- mbrock 10y ago
  Git lets you define custom "merge drivers."
- WorldMaker 10y ago
  For what it is worth, in my experiments most ASTs (the rare exception being something like Roslyn's C#/VB ASTs) don't do well in "degenerate states" such as a partially finished files. (A good source control system should let you commit unfinished work.) I did have great success using a syntax highlighting tokenizers. I was able to create really nice-looking character-based diffs that were relatively semantic, quite quickly. I've not tried to use that as the basis diffs for something like git, though I've suggested trying it before. Python code, if interested: https://github.com/WorldMaker/tokdiff https://github.com/WorldMaker/tokdiff
sroussey 10y ago
How does this compare to gun.js.org?

399 points by jaytaylor 10y ago

Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?

aeharding 10y ago
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
- chubot 10y ago
  Which ones are you thinking of? The nice thing about git is that it doesn't really require much of the server. I run my own "git hosting" with linode, apt-get, and ssh. Most of the solutions for immutable big data don't have that level of convenience, as far as I know. Anything which requires a lot of sysadmin and ops work will eventually become a commercial cloud service... git on the other hand is useful by itself, and good companies can obviously be built on top of it as well.
  - TylerE 10y ago
    Most binary formats don't diff well.
- rpedela 10y ago
  No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
  - emmelaich 10y ago
    They are so different it's hard to know what you mean by "basically the same thing"
    - rpedela 10y ago
      Yes there are significant differences in terms of feature, implementation, etc but they have the same goal: manage source code versions.
      - emmelaich 10y ago
        That's how Git is used but I think that by design, it is foremost a userlevel append-only content-addressable file system. The difference between this and a version control system is a major part of why the UI is so awful.
- rspeer 10y ago
  Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of: * Convert the versioned data to tab-separated values * COPY it into Postgres every time * Hope Postgres can act immutable enough even though it wasn't designed to be The closest I've come to improving this situation was Kyoto Cabinet (unusable license) and rolling my own damn hashtable (it worked okay but adding new kinds of indexes was just unmaintainable, there's a reason databases should be made by experts).
  - ykler 10y ago
    I was curious about Camlistore and Tahoe-LAFS at one point. I didn't investigate them very thoroughly, however. How do they compare?
  - deleted 10y ago
    [deleted]
  - sjezewski 10y ago
    (Disclaimer - I work at pachyderm) http://pachyderm.io http://pachyderm.io Pachyderm is git for data. We work hard to make sure we can store data of different types (binary, text, json) efficiently. We also work hard to give you good mechanisms to read the data in a distributed way. I'd be curious how this suits your purposes.
    - rspeer 10y ago
      Just started looking at Pachyderm. While I can see how a git-based filesystem can help with some use cases, does it do any kind of indexing at all? I see that the FAQ recommends exporting the data from Pachyderm into PostgreSQL, which leaves me where I am now.
AvenueIngres 10y ago
So.. CRDTs?
- aboodman 10y ago
  https://github.com/attic-labs/noms/blob/master/doc/faq.md#why-dont-you-just-use-crdts https://github.com/attic-labs/noms/blob/master/doc/faq.md#wh...
  - ergl 10y ago
    Those reasons are a bit outdated, you can certainly have add-and-remove crdts with or-sets, and you can do so with almost no garbage. One of the Tomtom engineers explained one approach at Strangeloop last month: https://www.youtube.com/watch?v=veeWamWy8dk https://www.youtube.com/watch?v=veeWamWy8dk
aboodman 10y ago
Noms performs diff and merge in time proportional to size of the diff. The size of the source data is not really relevant. One way to think of Noms is that is an index optimized for computing diffs. Here's a screencast that shows off Noms diffing things fast: https://www.youtube.com/watch?v=Zeg9CY3BMes https://www.youtube.com/watch?v=Zeg9CY3BMes
dwhitena 10y ago
I definitely agree here. As a data scientist, sometimes it seems like we are in the wild west as far as reproducibility and versioning of our analyses. This seems like an interesting project that tackles some of the data versioning stuff. However, I believe that, at least in data science, we need data versioning closely tied to the analyses themselves for complete reproducibility. That is, we need the versioning tied to the inputs/outputs of data pipeline stages, such that we can reproduce pipeline runs at any time and incrementally improve and run pipelines based on diffs in data. As mentioned elsewhere in the comments, Pachyderm (http://pachyderm.io/ http://pachyderm.io/) does exactly this. Working both as git for data, but also enabling data pipelining and analyses with the data versioning.

niftich 10y ago

Open-source tech like this is nice. This could be used to build a distributed document editing application, for example. Or any application where you want to spin off multiple instances and reconcile the data later. EDIT: At least one team is investigating layering Noms on top of IPFS [1]. I guess the idea would be to construct something similar to GitTorrent [2]; layering various version-controlled datastores on various p2p protocols could result in several viable architectures. [1] https://github.com/attic-labs/noms/issues/2123#issuecomment-245162820 https://github.com/attic-labs/noms/issues/2123#issuecomment-... [2] https://github.com/cjb/GitTorrent https://github.com/cjb/GitTorrent

sohilv 10y ago

A couchbase killer.

tominous 10y ago

Adam Leventhal (DTrace, OpenZFS) took a look at building a FUSE filesystem on Noms using Go. http://dtrace.org/blogs/ahl/2016/08/09/nomsfs/ http://dtrace.org/blogs/ahl/2016/08/09/nomsfs/ https://news.ycombinator.com/item?id=12255450 https://news.ycombinator.com/item?id=12255450

orblivion 10y ago
Sounds like what you'd want for a self-hosted Dropbox clone. I wonder what Syncthing, for instance, uses for reconciling differences on different clients.
- davidron 10y ago
  Syncthing doesn't reconcile differences. Instead a copy of the file is created. Syncthing does recognize conflicts. When a file has been modified on two devices simultaneously, one of the files will be renamed to <filename>.sync- conflict-<date>-<time>.<ext>. The device which has the larger value of the first 63 bits for his device ID will have his file marked as the conflicting file. Note that we only create sync-conflict files when the actual content differs. https://docs.syncthing.net/users/faq.html https://docs.syncthing.net/users/faq.html
- deleted 10y ago
  [deleted]

marknadal 10y ago

Noms is a great example of the power of decentralized database technology, the interesting research that goes into such systems, and wonderful documentation to browse. I do want to note some tradeoffs with Content-Addressed and Append-Only systems, as my work on a similar project ( an Open Source Firebase, https://github.com/amark/gun https://github.com/amark/gun ) made me move away from those ideas (even though they are great ideas). - Content-Addressed stores are going to revolutionize data integrity and efficiency. But they do have a trade off, it makes it a lot harder to read the data if you do not already know the data you are trying to read! The bottom of the repo metions for instance that a query system has not yet been built. From my experience the reason why is because it is difficult to build query systems on Content-Addressed stores, which is a tradeoff from all the gains you can get from it. - Append-Only gives you rich features like offline-first support and (if implemented) lovely things like rewind/fastforward data time travel. All very cool. However, do not forget that this then also makes it difficult for you to retrieve the latest whole snapshot of your data. So you are not going to get the read performance that you could. But the only possible way for us as a community, and people playing around with databases, can figure out what the best system is - is for people to build and experiment. Which is part of the reason why Nom is so cool. It is an invitation to others to actually join, play, and experiment with database technology in an open and encouraging environment. That is incredibly valuable and needed!

aboodman 10y ago
1. We have prototyped basic query functioanlity already (https://www.youtube.com/watch?v=fv6_T5yaWns https://www.youtube.com/watch?v=fv6_T5yaWns) and Noms was designed from the beginning to support efficient indexes and range scans. So I'm not sure why it would be harder for us to support a query language than any other database. 2. It's true that content addressing can exacerbate data locality which can hurt read performance. However, there are thing you can do to get a lot of that back.

duck 10y ago

Previous discussion from back in August: https://news.ycombinator.com/item?id=12211754 https://news.ycombinator.com/item?id=12211754

100ideas 10y ago

Looks farther along than http://dat-data.com/ http://dat-data.com/, another commendable distributed VCS for data. One distinction is that dat provides additional utilities for querying and compositing the data structures represented in any csv, json, and yaml files that stores.

100ideas 10y ago
One of the other design goals of Dat is to support continually-divergent forking, which they perceive as being useful for communities of analysts processing common datasets but to ultimately different ends. Of course, you never have to merge forks in git, but in it's current form they (dat devs) say that it's not really ideal.

haalcion3 10y ago

I've been wanting to use something like this. But... * It's a big jump from relational or noSQL DB's, so there aren't (m)any adapters that I can see for it for JPA, ActiveRecord, etc. * I'd really like to see a benchmark for each noms implementation compared to postgres, mysql, oracle, and mssql server, if there is a way to do apples-to-apples. * "noms" is unfortunately is really bad for SEO because noms is a common word in French. If it could be nomsdb or nomnomnoms or something less exactly French, that'd be better. It's going to be tough to find support online easily otherwise. * SQL compatibility. * Fault tolerance (how easily does it corrupt), HA, mirroring, full/partial replication, sharding, archival, partial history truncation, etc. It seems a little like a dolphin jumping into a pool of hungry sharks. It might be more evolved and more capable in some ways, but it's going to get its ass handed to it on speed and lack of features. Still- I can't wait to try it.

Ericson2314 10y ago
With content addressing, you may loose data but never get corrupted data.
actuallyalys 10y ago
>It seems a little like a dolphin jumping into a pool of hungry sharks. It might be more evolved and more capable in some ways, but it's going to get its ass handed to it on speed and lack of features. I'm inclined to agree for large, centralized databases, but I wonder if this would be a good fit for places where sqlite is used? This seems like it could be a good foundation for situations where you want to sync information without a central server, like between devices. An Access/Filemaker clone built on top of this would be cool, too.

WayneBro 10y ago

How is it that you have 2 reference implementations, written in 2 different cross platform environments, yet there is no support for Windows? Why would I use this if I can't use it everywhere?

alsetmusic 10y ago
Not everyone uses Windows. I will use it because it supports macOS and Linux, the two platforms that I use every day. Not everyone has the same needs.
colemickens 10y ago
... well, from the link... "Noms is supported on Mac OS X and Linux. You can compile a Windows build from source, and it usually works, but isn't officially supported." Also, supporting Windows is often a pain in the ass compared to Linux/Mac. I don't fault them for not supporting it officially, especially in the project's life.
- perlgeek 10y ago
  To be fair, Mac support is also a huge pain, unless you happen to have a Mac. Just getting a machine to test on is expensive. If you look for Mac OS VMs in the cloud, you find they start at $1 an hour (https://www.macincloud.com/ https://www.macincloud.com/) or around $80 / month (http://xcloud.me/pricing-signup/ http://xcloud.me/pricing-signup/). Compare that to around $5 / month for a Linux VM. And then you have to go the whole dance of getting xcode and homebrew just to have a development environment. It's been a few years since I used Mac OS X to develop stuff, but it wasn't intuitive at all back then.
  - huxley 10y ago
    Mac hosting doesn't tend to be VMs largely because of Apple's licensing rules for macOS (which is a big hassle, no question) so they'll naturally be more expensive than a $5 virtualized Linux VM. But Macincloud has ~$20/month plans (with 8GB of RAM which isn't a $5 option for most Linux or Windows VMs) and there are several others going for between $30-50, so you hardly have to go with $80/month.

deleted 10y ago

[deleted]

ReAzem 10y ago

Their logo is a squirrel giving an invisible blowjob

a3n 10y ago
It's an otter, floating on its back.
thght 10y ago
ROFL Seriously, down voted for laughing?
aboodman 10y ago
You say that like it's a bad thing.
hossbeast 10y ago
HN needs a +10 button
Xorlev 10y ago
Cannot unsee.

mahyarm 10y ago

I hope there is a prune option to delete very old commits.

aboodman 10y ago
There isn't yet, but there could be (ala shallow clone in Git).

aboodman 10y ago

Hi Hacker News. I'm one of the founders of the Noms project and Attic Labs, the company behind it. Happy to answer any questions. In the meantime, as long as I've got your attention, here's a few new stuffs we've been working on since last time Noms was discussed here in August: - A prototype query language, and a demo of how to create indexes in Noms: https://www.youtube.com/watch?v=fv6_T5yaWns https://www.youtube.com/watch?v=fv6_T5yaWns - Support for merging concurrent (and potentially conflicting) changes: https://www.youtube.com/watch?v=--7dgoJBdjU https://www.youtube.com/watch?v=--7dgoJBdjU

mostafah 10y ago
Does/will Noms have proper support for multiuser sync? That’s an issue with CouchDB. I mean: instead of syncing whole the database, only syncing the parts that a user has access to, and being able to define those accesses. The standard use case for consumer apps.
- aboodman 10y ago
  Noms syncs at the level of a "value" not the entire database. Since a Noms database is a tree of values, you can sync the entire db, but you don't have to. You can also sync a single list. So yes, in principle, we can definitely do this. In practice we are missing some conveniences that would make it a really easy drop-in feature.
esfandia 10y ago
Looks really interesting! This may sound like a stupid question, but do you have any publication I can cite by any chance?
- deleted 10y ago
  [deleted]
freekh 10y ago
Wow, congrats - this looks really interesting!
10y ago

kccqzy 10y ago

I think there is a very similar library in Haskell called project m36. Here's its github page on transactions: https://github.com/agentm/project-m36/blob/master/docs/transaction_graph_operators.markdown https://github.com/agentm/project-m36/blob/master/docs/trans...

benjismith 10y ago

Nice. I'm super excited about this! I've hand-rolled something a lot like this already for the Shaxpir backend, but it would be really nice to have a well-engineered database that already supports this kind of model, out of the box.

jbverschoor 10y ago

Weird choice for the uri: http://localhost:8000::dbname http://localhost:8000::dbname Why not http://localhost:8000/dbname http://localhost:8000/dbname ?

lost_my_pwd 10y ago
Wow, I missed that. That violates the URI spec, assuming the author(s) were intending to use a URI. https://tools.ietf.org/html/rfc3986 https://tools.ietf.org/html/rfc3986 (see "3.3. Path").
- drostie 10y ago
  Right, it intentionally violates the URI spec by appending something to the end of it. The data structure they're storing has a natural pair-of-structures at its top level: data Database = InMemory | LevelDB Path | ViaHTTP URL newtype DataSet = DS Text newtype Hash = Hash Text data Accessor = AccessDS DataSet | AccessValue (Either DataSet Hash) Path type DBAccessor = (Database, Accessor) They elected to basically encode a DBAccessor above as a string which you can split on "::", with the URL above being stored on the left in the case of the ViaHTTP databases.
  - aboodman 10y ago
    The string wasn't originally intended to be a URI, but I've been subsequently convinced that it would be useful for it to be one. We'll change it eventually.
Kiro 10y ago
https://news.ycombinator.com/item?id=12212967 https://news.ycombinator.com/item?id=12212967

kfk 10y ago

Have you had a look to the finance world? Git for data seems to be something we in finance really need, especially the possibility of seeing all the changes and of reconciliating things.

barrkel 10y ago
diff for financial data, with attendant workflow for breaks, is an already existing whole market segment. Duco, the startup I work for, is tackling it as a service.

jimktrains2 10y ago

I've been working on some syncing addressbook, calendar, password manager, and notes applications. My idea was to use mdns to announce presence and git to sync, but this might be (more) useful

aboodman 10y ago
Noms should definitely be more useful in that scenario. We have some customers who were using Git the way you describe and replaced it with Noms and have been very happy with the results.

angel-manuel 10y ago

I really like this, I've always thought that git needed to support diff modes different from textline-based because even if this is fit for most programming languages what you really what is to see differences between ASTs (take into account those absurd change counts when just changing the indentation or imagine a normal diff of LISP source). Maybe there's some way of replacing git with noms to get there(even if it may be killing flies with cannonballs)

aboodman 10y ago
If you take a language parser and pipe the resulting AST into Noms, you basically get something like codeq (http://blog.datomic.com/2012/10/codeq.html http://blog.datomic.com/2012/10/codeq.html).
mbrock 10y ago
Git lets you define custom "merge drivers."
WorldMaker 10y ago
For what it is worth, in my experiments most ASTs (the rare exception being something like Roslyn's C#/VB ASTs) don't do well in "degenerate states" such as a partially finished files. (A good source control system should let you commit unfinished work.) I did have great success using a syntax highlighting tokenizers. I was able to create really nice-looking character-based diffs that were relatively semantic, quite quickly. I've not tried to use that as the basis diffs for something like git, though I've suggested trying it before. Python code, if interested: https://github.com/WorldMaker/tokdiff https://github.com/WorldMaker/tokdiff

sroussey 10y ago

How does this compare to gun.js.org?