6 ms·

I've enjoyed using Irmin so far -- the APIs are well-designed and it's really handy to be able to use 'git log' for debugging. In fact a 'git log --pretty=onlin

by djs55 12y ago

I've enjoyed using Irmin so far -- the APIs are well-designed and it's really handy to be able to use 'git log' for debugging. In fact a 'git log --pretty=online' has started to replace a lot of the ad-hoc debug logging which permeates my code. Given how cheap storage is now, I think there are a lot of applications for which it makes sense to record all previous states by default (e.g. for debugging) and make mutating history (by squashing/compacting it away) the special case (a bit like running 'logrotate' every week)

a0 12y ago
Just out of curiosity – where are you using Irmin? I'm curious to look at some real world code beyond the official examples and the merge-queues project.
- djs55 12y ago
  I'm refactoring "Xenstore" (part of the control-plane for a Xen host) to use it. I want to make the server stateless so I can restart it if it crashes.. this mainly involves remembering lots of bits of state: connection information, watch registration, pending events etc. I also want to make it easier to debug while I'm at it. I've got a draft of a blog post here: https://github.com/djs55/mirage-www/blob/a1ab2fec78fa7d96c648c300c37105a0f799c30a/tmpl/blog/introducing-irmin-in-xenstore.md https://github.com/djs55/mirage-www/blob/a1ab2fec78fa7d96c64... edited to add: This work is a natural evolution of the "OXenstored" work presented at ICFP 2009 http://web.cecs.pdx.edu/~apt/icfp09_accepted_papers/83.html http://web.cecs.pdx.edu/~apt/icfp09_accepted_papers/83.html
  - sitkack 12y ago
    I know it won't help, but I was thinking that one could keep hot backups by doing a fork() suspend() and then diffing the heap in realtime to migrate state between processes. Trying to keep state on the heap instead of the stack is a huge start. But this is stuff you already know.
    - avsm 12y ago
      Using process checkpointing is indeed one good way to implement fault tolerance. The Irmin style is more explicit -- the heap itself is tree structured, and the application uses it to checkpoint itself to disk/memory as a matter of course. This ensures that only the minimal state required is stored (as opposed to the entire process heap), and also that state can be reconstructed intelligently to preserve sharing and special resources. For instance, file descriptors (if running in Unix mode) could be reified to a filename/offset and reopened, and memory mapped areas (such as the shared ring structures that Xen uses) could be re-granted from the hypervisor.
      - sitkack 12y ago
        It would be interesting to have an operating system that used a queue of transactions for IO, esp against an immutable FS somewhat like Datomic.
- avsm 12y ago
  We've got an experimental IMAP server using Irmin as a maildir-replacement backend, to give us e-mail provenance and undo for when mass moves go horribly wrong. It's not quite ready for open-sourcing yet, but should be over the summer. I'm switching my personal e-mail over to it soon gulp
  - walterbell 12y ago
    How does lrmin compare to camlistore?
    - avsm 12y ago
      They are different design philosophies: Irmin is closer to "Sqlite-with-Git-instead-of-SQL", since it's just an OCaml library that lets you build graph-based datastructures that are persisted into memory or Git. Higher-level datastructures are built in the usual OCaml style on top of it. You could build a Camlistore-like service on top of Irmin, but this design/eval hasn't happened yet to my knowledge. It's on the TODO list for Irmin applications -- while I really like Camlistore, I do also want finer control over routing of blobs in a content-addressed network to manage the physical placement of my personal data. Another interesting aspect of Irmin is that the backend blob stores are also just another functor. In addition to the existing memory/HTTP-REST/Git backends, we've sketched out the design for a convergent-encrypted backend store as well. Other backends such as existing k/v stores like LevelDB or Arakoon shouldn't be difficult, and patches are welcome.
wordplay 12y ago
I frequently get thrown into projects with aging codebases, technical debt, and few people around who wrote the original code. In this scenario using this kind of verbose state logging starts to sound like a huge win, especially if tools exist to visualize how state mutation is different on various legs and use that information to infer dependency guesses between execution branches.
pooper 12y ago
For anyone new following at home, 'git log --pretty=online' has a typo. It is 'git log --pretty=oneline'

by djs55 12y ago

a0 12y ago

Just out of curiosity – where are you using Irmin? I'm curious to look at some real world code beyond the official examples and the merge-queues project.

djs55 12y ago
I'm refactoring "Xenstore" (part of the control-plane for a Xen host) to use it. I want to make the server stateless so I can restart it if it crashes.. this mainly involves remembering lots of bits of state: connection information, watch registration, pending events etc. I also want to make it easier to debug while I'm at it. I've got a draft of a blog post here: https://github.com/djs55/mirage-www/blob/a1ab2fec78fa7d96c648c300c37105a0f799c30a/tmpl/blog/introducing-irmin-in-xenstore.md https://github.com/djs55/mirage-www/blob/a1ab2fec78fa7d96c64... edited to add: This work is a natural evolution of the "OXenstored" work presented at ICFP 2009 http://web.cecs.pdx.edu/~apt/icfp09_accepted_papers/83.html http://web.cecs.pdx.edu/~apt/icfp09_accepted_papers/83.html
- sitkack 12y ago
  I know it won't help, but I was thinking that one could keep hot backups by doing a fork() suspend() and then diffing the heap in realtime to migrate state between processes. Trying to keep state on the heap instead of the stack is a huge start. But this is stuff you already know.
  - avsm 12y ago
    Using process checkpointing is indeed one good way to implement fault tolerance. The Irmin style is more explicit -- the heap itself is tree structured, and the application uses it to checkpoint itself to disk/memory as a matter of course. This ensures that only the minimal state required is stored (as opposed to the entire process heap), and also that state can be reconstructed intelligently to preserve sharing and special resources. For instance, file descriptors (if running in Unix mode) could be reified to a filename/offset and reopened, and memory mapped areas (such as the shared ring structures that Xen uses) could be re-granted from the hypervisor.
    - sitkack 12y ago
      It would be interesting to have an operating system that used a queue of transactions for IO, esp against an immutable FS somewhat like Datomic.
avsm 12y ago
We've got an experimental IMAP server using Irmin as a maildir-replacement backend, to give us e-mail provenance and undo for when mass moves go horribly wrong. It's not quite ready for open-sourcing yet, but should be over the summer. I'm switching my personal e-mail over to it soon gulp
- walterbell 12y ago
  How does lrmin compare to camlistore?
  - avsm 12y ago
    They are different design philosophies: Irmin is closer to "Sqlite-with-Git-instead-of-SQL", since it's just an OCaml library that lets you build graph-based datastructures that are persisted into memory or Git. Higher-level datastructures are built in the usual OCaml style on top of it. You could build a Camlistore-like service on top of Irmin, but this design/eval hasn't happened yet to my knowledge. It's on the TODO list for Irmin applications -- while I really like Camlistore, I do also want finer control over routing of blobs in a content-addressed network to manage the physical placement of my personal data. Another interesting aspect of Irmin is that the backend blob stores are also just another functor. In addition to the existing memory/HTTP-REST/Git backends, we've sketched out the design for a convergent-encrypted backend store as well. Other backends such as existing k/v stores like LevelDB or Arakoon shouldn't be difficult, and patches are welcome.

wordplay 12y ago

I frequently get thrown into projects with aging codebases, technical debt, and few people around who wrote the original code. In this scenario using this kind of verbose state logging starts to sound like a huge win, especially if tools exist to visualize how state mutation is different on various legs and use that information to infer dependency guesses between execution branches.

pooper 12y ago

For anyone new following at home, 'git log --pretty=online' has a typo. It is 'git log --pretty=oneline'