7 ms·

Vanadium – Create distributed applications that can run anywhere

124 points by chuhnk 10y ago

azinman2 10y ago
I ported vanadium and syncbase to iOS. Happy to answer questions. Yes indeed this is a google project but is now being taken on by its original lead, and former head of search infrastructure, now at Grail bio.
- dunkelheit 10y ago
  The project looks interesting but something about it rubs me the wrong way. Which organization is behind vanadium? Is it google? Then how is grail bio involved? Or is it a side project of google employees who now work at grail bio? I'm confused. Also, why are syncbase sources not on github?
  - azinman2 10y ago
    Everything is open source and on GitHub. This is a rare project that was 100% open source from the beginning.
  - danielsju6 10y ago
    It's all there. Just look under Go/Java/Swift https://github.com/vanadium https://github.com/vanadium
    - dunkelheit 10y ago
      Thanks, I see it now! The first part of my question is still unanswered though..
      - danielsju6 10y ago
        I'd defer to cosmosnicolaou there, as I've not been involved in some months. Check out his answer below. Also might be worth asking on our mailing list https://groups.google.com/a/v.io/forum/#!forum/vanadium-discuss https://groups.google.com/a/v.io/forum/#!forum/vanadium-disc...
sethammons 10y ago
Quickly looked over this and the thing that caught my eye is that it says you can set up distributed systems that are available offline through device to device communication. At least, I think that is what it does. I could not quickly find more information on that. A diagram or something would be helpful to visualize this, with an explanation of the protocols that are used. Maybe I missed the examples or case study section.
oori 10y ago
this vs. firebase for sync ?
- azinman2 10y ago
  Firebase doesn't have offline p2p at all, let alone sync. That said, right now I'd build using firebase unless you have very specific needs.
niftich 10y ago
After 10 minutes, here's what I found out: - The authors are either Google and/or heavily affiliated with Google - On their website, the two highlighted projects/products are Syncbase and Vanadium Core. - Syncbase is a distributed, works-offline datastore, with well-defined ways of doing reconciliation. Think, git merges, or collaborative docs, or y'know anything with vector clocks but approached from the side of the UX they want to target. - Vanadium Core is "the discovery, RPC, and security layer underlying Syncbase", including its custom binary data serialization format 'VOM' and its IDL called 'VDL'.
- the_duke 10y ago
  How did you find the association with Google? But you are probably right: page is in material design, the repos are mostly Dart or Go.
  - deleted 10y ago
    [deleted]
  - sly010 10y ago
    Their intro video is done by 2 Google engineers.
    - avighnay 10y ago
      And they say 'working on here at Google' in the video
  - niftich 10y ago
    I'm so glad you asked! - I read the Vanadium ToS [1] - In the video embedded from the page on 'Syncbase' [2] everyone shown works at Google. - At this point, the association with Google looks very likely. I searched for 'syncbase google' and found a ton of intriguing stuff, including the prospect that it's being used in production settings by Google. - I searched for 'vanadium google' and found this Twitter thread [3] [1] https://vanadium.github.io/tos.html https://vanadium.github.io/tos.html [2] https://vanadium.github.io/syncbase/ https://vanadium.github.io/syncbase/ [3] https://twitter.com/chuhnk/status/685455321465237504 https://twitter.com/chuhnk/status/685455321465237504
Mao_Zedang 10y ago
I didnt know io sold single letter domains, sexy.
- inopinatus 10y ago
  So does .org, see http://www.project94.org/ http://www.project94.org/.
  - Mao_Zedang 10y ago
    I wonder how much they cost.
    - inopinatus 10y ago
      I believe the first round auctions (a couple of years ago) started at $50,000 for single-letter domains. As an additional barrier, it may also be that PIR require some kind of public-interest usage commitment.
marknadal 10y ago
Wonderful to hear about more P2P database sync systems! I would love to hear how you compare to Deepstream and us ( http://gun.js.org/ http://gun.js.org/ ), and http://Horizon.io http://Horizon.io ? Edit: Also how on earth did you snag `v.io`? That is amazing and I thought single-letter domains weren't allowed?
- HodGreeley 10y ago
  I'm assuming Horizon is in limbo along with RethinkDB?
makmanalp 10y ago
Can anyone comment on the disadvantages of last-write-wins? Either that or how tricky it is to always have to define your own merge policies?
mehrdada 10y ago
The cool thing I really like about Vanadium is a unified worldview it provides with the ability to access every object in the Vanadium ecosystem through its powerful naming/mounting mechanism. In Vanadium, you can "mount" things on your namespace (think Unix VFS) on a hierarchical namespace and invoke RPC methods on the objects. The method invocation is mutually authenticated with ed25519 keys and end-to-end encrypted on the wire. I sometimes jokingly called it "Plan 10". It is certainly not just another "synchronizing database" though that is a subset of it. Disclosure: I very briefly worked on it at Google and I can attest the team behind it is brilliant beyond imagination.
curiousmonkey90 10y ago
It seems like it is no longer under active development. I hope I am wrong, but does anyone happen to know? It looks like a lot of the Googlers previously working on the project has now moved to Grail bio.
- danielsju6 10y ago
  I was on the team working on Vanadium @ Google and have since moved on to Firebase. I can't speak to what level of commitment to the project those that have moved on to Grail have; you should direct questions to https://news.ycombinator.com/user?id=cosmosnicolaou https://news.ycombinator.com/user?id=cosmosnicolaou or refer to his comments. I actually recently gave a presentation / live demo of Vanadium at Drinks on Tap SF. It's not production ready, but a cool glimpse into a possible new paradigm ;)
- leeeroyjenkins 10y ago
  "a lot"? who else do you see besides Cosmos?
rixed 10y ago
Is this similar to Irmin (https://github.com/mirage/irmin https://github.com/mirage/irmin) ?
allan_s 10y ago
In the documentation I see Note Requiring a cloud Syncbase is temporary. We are planning to allow the API to be used without a cloud Syncbase soon. Does it mean that the following use case will be supported: I have two devices in the same LAN with no internet access, and they will be able to synchronize ?
- azinman2 10y ago
  Correct.
botw 10y ago
besides wifi, what other type of p2p communication channels are supported in Vanadium? bluetooth? USB? radio?
- danielsju6 10y ago
  Prototypes also use BLE / Bluetooth Classic but the vision being that it's transport agnostic.
cosmosnicolaou 10y ago
Let me provide a brief history/overview and answer the questions that have been asked as best I can. I started vanadium about three years ago to build a secure rpc and naming system that would make it easier to build distributed applications, including those on mobile and embedded devices/IoT. Security was a central goal. Distributed, synchronzsed storage has long been a pain point, especially when devices are often disconnected and we built syncbase to address that need. I left Google in May of this year and joined Grail, where we're developing an NGS-based blood test to detect cancer early, when it can still be cured. We're using Vanadium for the infrastructure we need to support our internal development and analysis pipelines. Since I left Google Vanadium was split into a 'core' component (security, rpc, naming) and syncbase separately. The core component is very stable, syncbase is likely stable but not quite complete. We're currently using only Vanadium core. To answer some point questions: - yes, this is the project Jeff Nichol's is referring to - and here, https://arxiv.org/pdf/1604.06959v2.pdf https://arxiv.org/pdf/1604.06959v2.pdf

124 points by chuhnk 10y ago

I ported vanadium and syncbase to iOS. Happy to answer questions. Yes indeed this is a google project but is now being taken on by its original lead, and former head of search infrastructure, now at Grail bio.

dunkelheit 10y ago
The project looks interesting but something about it rubs me the wrong way. Which organization is behind vanadium? Is it google? Then how is grail bio involved? Or is it a side project of google employees who now work at grail bio? I'm confused. Also, why are syncbase sources not on github?
- azinman2 10y ago
  Everything is open source and on GitHub. This is a rare project that was 100% open source from the beginning.
- danielsju6 10y ago
  It's all there. Just look under Go/Java/Swift https://github.com/vanadium https://github.com/vanadium
  - dunkelheit 10y ago
    Thanks, I see it now! The first part of my question is still unanswered though..
    - danielsju6 10y ago
      I'd defer to cosmosnicolaou there, as I've not been involved in some months. Check out his answer below. Also might be worth asking on our mailing list https://groups.google.com/a/v.io/forum/#!forum/vanadium-discuss https://groups.google.com/a/v.io/forum/#!forum/vanadium-disc...

sethammons 10y ago

Quickly looked over this and the thing that caught my eye is that it says you can set up distributed systems that are available offline through device to device communication. At least, I think that is what it does. I could not quickly find more information on that. A diagram or something would be helpful to visualize this, with an explanation of the protocols that are used. Maybe I missed the examples or case study section.

oori 10y ago

this vs. firebase for sync ?

azinman2 10y ago
Firebase doesn't have offline p2p at all, let alone sync. That said, right now I'd build using firebase unless you have very specific needs.

niftich 10y ago

After 10 minutes, here's what I found out: - The authors are either Google and/or heavily affiliated with Google - On their website, the two highlighted projects/products are Syncbase and Vanadium Core. - Syncbase is a distributed, works-offline datastore, with well-defined ways of doing reconciliation. Think, git merges, or collaborative docs, or y'know anything with vector clocks but approached from the side of the UX they want to target. - Vanadium Core is "the discovery, RPC, and security layer underlying Syncbase", including its custom binary data serialization format 'VOM' and its IDL called 'VDL'.

the_duke 10y ago
How did you find the association with Google? But you are probably right: page is in material design, the repos are mostly Dart or Go.
- deleted 10y ago
  [deleted]
- sly010 10y ago
  Their intro video is done by 2 Google engineers.
  - avighnay 10y ago
    And they say 'working on here at Google' in the video
- niftich 10y ago
  I'm so glad you asked! - I read the Vanadium ToS [1] - In the video embedded from the page on 'Syncbase' [2] everyone shown works at Google. - At this point, the association with Google looks very likely. I searched for 'syncbase google' and found a ton of intriguing stuff, including the prospect that it's being used in production settings by Google. - I searched for 'vanadium google' and found this Twitter thread [3] [1] https://vanadium.github.io/tos.html https://vanadium.github.io/tos.html [2] https://vanadium.github.io/syncbase/ https://vanadium.github.io/syncbase/ [3] https://twitter.com/chuhnk/status/685455321465237504 https://twitter.com/chuhnk/status/685455321465237504

Mao_Zedang 10y ago

I didnt know io sold single letter domains, sexy.

inopinatus 10y ago
So does .org, see http://www.project94.org/ http://www.project94.org/.
- Mao_Zedang 10y ago
  I wonder how much they cost.
  - inopinatus 10y ago
    I believe the first round auctions (a couple of years ago) started at $50,000 for single-letter domains. As an additional barrier, it may also be that PIR require some kind of public-interest usage commitment.

marknadal 10y ago

Wonderful to hear about more P2P database sync systems! I would love to hear how you compare to Deepstream and us ( http://gun.js.org/ http://gun.js.org/ ), and http://Horizon.io http://Horizon.io ? Edit: Also how on earth did you snag `v.io`? That is amazing and I thought single-letter domains weren't allowed?

HodGreeley 10y ago
I'm assuming Horizon is in limbo along with RethinkDB?

makmanalp 10y ago

Can anyone comment on the disadvantages of last-write-wins? Either that or how tricky it is to always have to define your own merge policies?

mehrdada 10y ago

The cool thing I really like about Vanadium is a unified worldview it provides with the ability to access every object in the Vanadium ecosystem through its powerful naming/mounting mechanism. In Vanadium, you can "mount" things on your namespace (think Unix VFS) on a hierarchical namespace and invoke RPC methods on the objects. The method invocation is mutually authenticated with ed25519 keys and end-to-end encrypted on the wire. I sometimes jokingly called it "Plan 10". It is certainly not just another "synchronizing database" though that is a subset of it. Disclosure: I very briefly worked on it at Google and I can attest the team behind it is brilliant beyond imagination.

curiousmonkey90 10y ago

It seems like it is no longer under active development. I hope I am wrong, but does anyone happen to know? It looks like a lot of the Googlers previously working on the project has now moved to Grail bio.

danielsju6 10y ago
I was on the team working on Vanadium @ Google and have since moved on to Firebase. I can't speak to what level of commitment to the project those that have moved on to Grail have; you should direct questions to https://news.ycombinator.com/user?id=cosmosnicolaou https://news.ycombinator.com/user?id=cosmosnicolaou or refer to his comments. I actually recently gave a presentation / live demo of Vanadium at Drinks on Tap SF. It's not production ready, but a cool glimpse into a possible new paradigm ;)
leeeroyjenkins 10y ago
"a lot"? who else do you see besides Cosmos?

rixed 10y ago

Is this similar to Irmin (https://github.com/mirage/irmin https://github.com/mirage/irmin) ?

allan_s 10y ago

In the documentation I see Note Requiring a cloud Syncbase is temporary. We are planning to allow the API to be used without a cloud Syncbase soon. Does it mean that the following use case will be supported: I have two devices in the same LAN with no internet access, and they will be able to synchronize ?

azinman2 10y ago
Correct.

botw 10y ago

besides wifi, what other type of p2p communication channels are supported in Vanadium? bluetooth? USB? radio?

danielsju6 10y ago
Prototypes also use BLE / Bluetooth Classic but the vision being that it's transport agnostic.

cosmosnicolaou 10y ago

Let me provide a brief history/overview and answer the questions that have been asked as best I can. I started vanadium about three years ago to build a secure rpc and naming system that would make it easier to build distributed applications, including those on mobile and embedded devices/IoT. Security was a central goal. Distributed, synchronzsed storage has long been a pain point, especially when devices are often disconnected and we built syncbase to address that need. I left Google in May of this year and joined Grail, where we're developing an NGS-based blood test to detect cancer early, when it can still be cured. We're using Vanadium for the infrastructure we need to support our internal development and analysis pipelines. Since I left Google Vanadium was split into a 'core' component (security, rpc, naming) and syncbase separately. The core component is very stable, syncbase is likely stable but not quite complete. We're currently using only Vanadium core. To answer some point questions: - yes, this is the project Jeff Nichol's is referring to - and here, https://arxiv.org/pdf/1604.06959v2.pdf https://arxiv.org/pdf/1604.06959v2.pdf