5 ms·

Show HN: Flatend – Quickly build microservices using P2P networking in Node/Go

145 points by lithdew 6y ago

lithdew 6y ago
Hello! OP here. One of the biggest frustrations I've had prying through Darklang [1], Firebase [2], Supabase [3], and a slew of other low-code projects is that although they do help me build backends faster, they come with a big tradeoff of having to deal with either (a) vendor lock-in, (b) tie-in to specific tools/databases/coding patterns, or (c) a large monthly bill. At the same time, coding without these tools, I would have to deal with either configuration hell [4] with Kubernetes/Istio/Consul or take the time to stitch a bunch of libraries together and spend a lot of time having to deal with devops/networking/backend architecture myself. So, I built this framework called Flatend over the last two months for Mozilla Spring MVP Labs, which: 1) lets you write functions using any tools/databases/programming languages/frameworks/hosting environments, and 2) turn them into microservices stitched together into a mesh network with service discovery/load balancing/routing using the Kademlia p2p overlay networking protocol popularized in BitTorrent/Ethereum [5]. Taking advantage of p2p networking, Flatend has taken off a lot of the pains and time spent dealing with bidirectional RPC, service discovery, load balancing, scaling, deployment patterns, and end-to-end-encryption, and has led me to spending more of my time writing productive business logic code in NodeJS/Go. Using Flatend I've now deployed four different sites/apps racking roughly 200k+ visitors on two 5$ DigitalOcean instances, doing red/blue deployments and upgrades to my microservices really easily. But that being said, Flatend still is very much experimental and is a heavy work in progress. Nonetheless, I still do hope for Flatend to be immeasurably useful for the people here at HN looking to speed up the backend development of their next product/service. [1] https://news.ycombinator.com/item?id=20985429 https://news.ycombinator.com/item?id=20985429 [2] https://news.ycombinator.com/item?id=17661391 https://news.ycombinator.com/item?id=17661391 [3] https://news.ycombinator.com/item?id=23319901 https://news.ycombinator.com/item?id=23319901 [4] https://news.ycombinator.com/item?id=21551868 https://news.ycombinator.com/item?id=21551868 [5] https://en.wikipedia.org/wiki/Kademlia https://en.wikipedia.org/wiki/Kademlia
- kodablah 6y ago
  I appreciate the dev-friendliness of this (libp2p leaves a lot to be desired last I looked from a dev perspective). How chatty is it? How resilient is it to untrusted peers (e.g. any of skademlia features built in)? Any plans to let the user provide their own net.Conns or io.Reader/Writers (e.g. I might want to run it over Tor)?
  - lithdew 6y ago
    All communication is quiescent unless some given node A sends data to node B (apart from TCP keepalive packets of course). For routing messages, it is expected O(log n) much like ordinary Kademlia. There isn't much resilience carefully integrated into the system for untrusted peers (complete S/Kademlia is in progress though! Static/dynamic crypto puzzles are fully implemented to prevent Sybil attacks on IDs, and the FIND_NODE RPC call is implemented to meet he guarantees documented in S/Kademlia's paper), since this framework is intended to be used in a closed environment for a set of microservices. The underlying library for networking, Monte, fully supports providing your own net.Conn's and io.Reader/Writers - all you need to do is return your modified net.Conn via implementing the monte.Handshaker interface here: https://github.com/lithdew/monte/blob/master/net.go#L36 https://github.com/lithdew/monte/blob/master/net.go#L36 Monte will everything else related to bidirectional RPC management. Details of the bare minimum Monte prefixes to your packets is documented on the README for it. Hope you enjoy using either Monte or Flatend!
- asim 6y ago
  I'm with you on this. I was working on something similar but just found it was a bit early for the P2P networking side of things. We're focusing on cloud first and driving local to platform development and will come back around to the P2P in a few years. A fan of anyone who can make it work. https://github.com/micro/development/blob/master/ideas/network.md https://github.com/micro/development/blob/master/ideas/netwo...
- rakoo 6y ago
  I might be a bit dense, but I don't see what's p2p about Flatend. You have a service that connects to an API gateway by hardcoding its address, then the client queries the gateway directly, and the service replies. Where is Kademlia used here ? Not trying to be negative, I'm just trying to understand
tak1827 6y ago
Can I help to support WebSocket or gRPC?
- lithdew 6y ago
  Contributions would be great! Open up a Github issue or come join us on Discord (link in the readme) and let's talk about it there :).
  - tak1827 6y ago
    Thank you for relying. I'd be glad to do.
tak1827 6y ago
Internally using this library seems to nice as well :) https://github.com/lithdew/monte https://github.com/lithdew/monte
xgenecloud 6y ago
This looks really promising. Can you think of mapping express.js routes to flatend automatically - that would be awesome. We are striving to make things simpler on backend too - pls take a look at https://github.com/xgenecloud/xgenecloud https://github.com/xgenecloud/xgenecloud
- blader_johny 6y ago
  I'm working with OP on flatend. I saw your Show HN a couple days ago, it's an extensive suite for code generation indeed. I'm seeing some crossover opportunity here. May I ask if express.js is a compulsory requirment for generating with xgenecloud?
  - xgenecloud 6y ago
    >> I'm seeing some crossover opportunity here I guess so. Happy to have a chat - https://discord.gg/5RgZmkW https://discord.gg/5RgZmkW >> May I ask if express.js is a compulsory requirment .. ? Yes, XgeneCloud's instant REST/GraphQL APIs generation relies on Express.js
    - deleted 6y ago
      [deleted]
- deleted 6y ago
  [deleted]
ykjt 6y ago
awesome.. it's a very simple.
mpfundstein 6y ago
mega interesting approach. will try it out later. this is the first time i heard that someone is using a p2p protocol for stuff like service discovery. How did you get this insight? during morning breakfast ? :p
- lithdew 6y ago
  Honestly, it was during my deep dive into p2p networking a year back that I started scratching my head and thought "Huh, you could pretty much replace most of the reasons you'd need Kubernetes and gRPC and Consul with this and make coding 10x simpler..." I only got to finally try it out just two months ago when Mozilla opened up a program for building MVPs that aim to fix problems with the Internet. Big thanks to them for that!
Ataraxy 6y ago
This is neat. Reminds me of moleculer.services to a degree.
HerbsMan 6y ago
Good job, Kenta :)

145 points by lithdew 6y ago

Hello! OP here. One of the biggest frustrations I've had prying through Darklang [1], Firebase [2], Supabase [3], and a slew of other low-code projects is that although they do help me build backends faster, they come with a big tradeoff of having to deal with either (a) vendor lock-in, (b) tie-in to specific tools/databases/coding patterns, or (c) a large monthly bill. At the same time, coding without these tools, I would have to deal with either configuration hell [4] with Kubernetes/Istio/Consul or take the time to stitch a bunch of libraries together and spend a lot of time having to deal with devops/networking/backend architecture myself. So, I built this framework called Flatend over the last two months for Mozilla Spring MVP Labs, which: 1) lets you write functions using any tools/databases/programming languages/frameworks/hosting environments, and 2) turn them into microservices stitched together into a mesh network with service discovery/load balancing/routing using the Kademlia p2p overlay networking protocol popularized in BitTorrent/Ethereum [5]. Taking advantage of p2p networking, Flatend has taken off a lot of the pains and time spent dealing with bidirectional RPC, service discovery, load balancing, scaling, deployment patterns, and end-to-end-encryption, and has led me to spending more of my time writing productive business logic code in NodeJS/Go. Using Flatend I've now deployed four different sites/apps racking roughly 200k+ visitors on two 5$ DigitalOcean instances, doing red/blue deployments and upgrades to my microservices really easily. But that being said, Flatend still is very much experimental and is a heavy work in progress. Nonetheless, I still do hope for Flatend to be immeasurably useful for the people here at HN looking to speed up the backend development of their next product/service. [1] https://news.ycombinator.com/item?id=20985429 https://news.ycombinator.com/item?id=20985429 [2] https://news.ycombinator.com/item?id=17661391 https://news.ycombinator.com/item?id=17661391 [3] https://news.ycombinator.com/item?id=23319901 https://news.ycombinator.com/item?id=23319901 [4] https://news.ycombinator.com/item?id=21551868 https://news.ycombinator.com/item?id=21551868 [5] https://en.wikipedia.org/wiki/Kademlia https://en.wikipedia.org/wiki/Kademlia

kodablah 6y ago
I appreciate the dev-friendliness of this (libp2p leaves a lot to be desired last I looked from a dev perspective). How chatty is it? How resilient is it to untrusted peers (e.g. any of skademlia features built in)? Any plans to let the user provide their own net.Conns or io.Reader/Writers (e.g. I might want to run it over Tor)?
- lithdew 6y ago
  All communication is quiescent unless some given node A sends data to node B (apart from TCP keepalive packets of course). For routing messages, it is expected O(log n) much like ordinary Kademlia. There isn't much resilience carefully integrated into the system for untrusted peers (complete S/Kademlia is in progress though! Static/dynamic crypto puzzles are fully implemented to prevent Sybil attacks on IDs, and the FIND_NODE RPC call is implemented to meet he guarantees documented in S/Kademlia's paper), since this framework is intended to be used in a closed environment for a set of microservices. The underlying library for networking, Monte, fully supports providing your own net.Conn's and io.Reader/Writers - all you need to do is return your modified net.Conn via implementing the monte.Handshaker interface here: https://github.com/lithdew/monte/blob/master/net.go#L36 https://github.com/lithdew/monte/blob/master/net.go#L36 Monte will everything else related to bidirectional RPC management. Details of the bare minimum Monte prefixes to your packets is documented on the README for it. Hope you enjoy using either Monte or Flatend!
asim 6y ago
I'm with you on this. I was working on something similar but just found it was a bit early for the P2P networking side of things. We're focusing on cloud first and driving local to platform development and will come back around to the P2P in a few years. A fan of anyone who can make it work. https://github.com/micro/development/blob/master/ideas/network.md https://github.com/micro/development/blob/master/ideas/netwo...
rakoo 6y ago
I might be a bit dense, but I don't see what's p2p about Flatend. You have a service that connects to an API gateway by hardcoding its address, then the client queries the gateway directly, and the service replies. Where is Kademlia used here ? Not trying to be negative, I'm just trying to understand

tak1827 6y ago

Can I help to support WebSocket or gRPC?

lithdew 6y ago
Contributions would be great! Open up a Github issue or come join us on Discord (link in the readme) and let's talk about it there :).
- tak1827 6y ago
  Thank you for relying. I'd be glad to do.

tak1827 6y ago

Internally using this library seems to nice as well :) https://github.com/lithdew/monte https://github.com/lithdew/monte

xgenecloud 6y ago

This looks really promising. Can you think of mapping express.js routes to flatend automatically - that would be awesome. We are striving to make things simpler on backend too - pls take a look at https://github.com/xgenecloud/xgenecloud https://github.com/xgenecloud/xgenecloud

blader_johny 6y ago
I'm working with OP on flatend. I saw your Show HN a couple days ago, it's an extensive suite for code generation indeed. I'm seeing some crossover opportunity here. May I ask if express.js is a compulsory requirment for generating with xgenecloud?
- xgenecloud 6y ago
  >> I'm seeing some crossover opportunity here I guess so. Happy to have a chat - https://discord.gg/5RgZmkW https://discord.gg/5RgZmkW >> May I ask if express.js is a compulsory requirment .. ? Yes, XgeneCloud's instant REST/GraphQL APIs generation relies on Express.js
  - deleted 6y ago
    [deleted]
deleted 6y ago
[deleted]

ykjt 6y ago

awesome.. it's a very simple.

mpfundstein 6y ago

mega interesting approach. will try it out later. this is the first time i heard that someone is using a p2p protocol for stuff like service discovery. How did you get this insight? during morning breakfast ? :p

lithdew 6y ago
Honestly, it was during my deep dive into p2p networking a year back that I started scratching my head and thought "Huh, you could pretty much replace most of the reasons you'd need Kubernetes and gRPC and Consul with this and make coding 10x simpler..." I only got to finally try it out just two months ago when Mozilla opened up a program for building MVPs that aim to fix problems with the Internet. Big thanks to them for that!

Ataraxy 6y ago

This is neat. Reminds me of moleculer.services to a degree.

HerbsMan 6y ago

Good job, Kenta :)