8 ms·

Lambdoku – AWS Lambda with Heroku-like Experience

128 points by kubek2k 10y ago

oldopsguy 10y ago
Another nice project that helps to ease lambda workflows is https://github.com/garnaat/kappa https://github.com/garnaat/kappa
- mullen 10y ago
  There is similar project called Zappa. pip install zappa
  - Mizza 10y ago
    Zappa actually uses some of Kappa under the hood, but with many many more features. Learn more here: https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa
    - mullen 10y ago
      Is Zappa a fork of Kappa? I have not used Kappa at all but it looks like it a whole lot.
stcredzero 10y ago
I'm wondering if it would be possible to create something like Lambda, but optimized for multiplayer servers with game loops? What if there was a platform that let you write pure functions to create player accounts, spawn game instances, and handle the transformation of player and game state from tick to tick? (You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.) Clojure, Javascript, and Golang would be good languages for this. The environment possibly would let you inject and replace functions live, lending to a Light Table like experience, but while developing a multiplayer game.
- keithwhor 10y ago
  You could, but the latency issues (network hops) would make the solution untenable. Function as a Service platforms are ideal for solutions where scalability is a huge concern, latency not so much. These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up. You'd probably want architecture extremely specialized to a game loop, and probably your game loop specifically, if you wanted low-latency, high-availability FaaS for this purpose.
  - stcredzero 10y ago
    You could, but the latency issues (network hops) would make the solution untenable. ...ok... Did you read my comment? These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up. Yeah, of course that's not going to work. I've basically built a specialized "container" system, but instead of using kernel virtualization, it's actually a specific Go "object" residing in an already running server process. These "containers" (really instances) start running in an environment where we already have a virtual server with a process with a server/game loop up and running. I have a cluster of processes that does this, and idempotently spawns an instance of star system in a multiplayer game consisting of 2^87 star systems. www.emergencevector.com Such a system can still be built on Docker style containers, it's just that those Docker style containers represent an entire sub-cluster of many server processes that can each host many "instance" containers which are where the functions that transform from tick n to tick n+1 reside. You'd probably want architecture extremely specialized to a game loop That is what I said in my comment. and probably your game loop specifically I'm saying that the game loop could be customized/built up by a developer by injecting functions into such a system. (Or, possibly, by injecting such functions into a dev version running locally, then exporting them to a server in the cloud.)
- wolfgang42 10y ago
  Your basic description sounds like LambdaMOO (written in 1990, the name is a coincidence). The LambdaMOO server has a main event loop which handles user events, then schedules tasks which execute functions ('verbs' in LambdaMOO parlance) to modify the game state. Verbs are not pure per se but can be thought of as a transaction which executes in an atomic, consistent, and isolated manner. Verbs are attached to objects in the game, and can be modified in-game on the fly by players with their programmer bit set. (Programmers can only modify verbs on objects they own.) The LambdaMOO language is a lot like JavaScript with a Luaesque syntax. It has prototypical inheritance, which meshes well with its concept of objects as physical things. For example, to get a new instance of a duck one would create a Generic Duck named "My Duck" ('Generic' being the conventional prefix for an object specifically intended to be used as a prototype.) There was a time in the late 1990s when it regularly supported 300 active users at once (with a certan amount of lag); on modern hardware it could probably handle significantly more than that. If you want a modern scalable system then LambdaMOO itself is probably not for you; the server is single-threaded (in fact the whole thing is remarkably similar to NodeJS with async/await support) and state is saved in-memory and checkpointed periodically (hence the Durability missing from the above description of verbs). However, the basic design (prototypal inheritance, in-database verbs, ACI[D] tasks, etc) is extremely well proven, and would lend itself well to being copied into a modern distributed system like AWS Lambda. As long as you had an ACID datastore to back it up, you could scale the task execution as far as you wanted, and with a bit of clever work you could probably come up with a partitioning scheme to make the database scale as well. In fact, I've partially implemented this already; what's stopped me has primarily been that I'm more interested in the technical aspects and have no idea what I would actually use it for once I built it. If you have a use case for this sort of thing I'd love to hear about it.
michaelbuckbee 10y ago
Very cool project, but it has me thinking what/if it would be possible to build out more of the infrastructure management that Heroku has to make it even easier (domain setup, addon store, etc.)
mifreewil 10y ago
Related: Apex https://www.github.com/apex/apex https://www.github.com/apex/apex
coenhyde 10y ago
This is great. I was going to create something similar to manage lambda functions across environments. Awesome to see you're already using the newly released environment variables for lambdas.
- kubek2k 10y ago
  thanks! - please give it a try and let me know in the issues if something doesn't work as expected.
keithwhor 10y ago
Hey! Great project. We actually built stdlib [1] to capture an experience similar to Heroku's for building microservices, and we'll be presenting at AWS Re:invent on Tuesday [2]. Our workflows with our public API are accessible via our open source library, would love contributors such as yourself (and anybody else that finds it interesting!). We have a diverse team working on the project - Xooglers, support from early Heroku team members, and even some YC alumni lending a hand. :) [1] https://github.com/poly/stdlib https://github.com/poly/stdlib [2] https://www.portal.reinvent.awsevents.com/connect/sessionDetail.ww?SESSION_ID=8092 https://www.portal.reinvent.awsevents.com/connect/sessionDet...
josegonzalez 10y ago
For a second there, I thought someone implemented a push-to-deploy experience for Heroku (with some of the other Heroku niceties). I actually wrote a blog post about that this week: http://josediazgonzalez.com/2016/11/21/push-to-deploy-lambda-applications/ http://josediazgonzalez.com/2016/11/21/push-to-deploy-lambda...
- kubek2k 10y ago
  really good read. thx! :bow: I didn't plan to do the push-to-deploy thing, mostly because that would totally complicate things. The rationale was more that we would use the CI that would automatically build and deploy things to DEV env that would be then promoted around.
charlesetc 10y ago
You could easily prefix your special environment variable with 'lambdoku_' so you don't have to fear that people could use it..
- kubek2k 10y ago
  :bow: https://github.com/kubek2k/lambdoku/issues/8 https://github.com/kubek2k/lambdoku/issues/8
pepicon 10y ago
Lambdoku = lick from ass in portuguese
- fiatjaf 10y ago
  ahahah, I thought things wouldn't get worse than "Heroku" and "Dokku".
fiatjaf 10y ago
The guide on the README doesn't tell me how am I supposed to push and update a function. Also, does it help with dependencies or something like that? Does it manage packaging files?
- kubek2k 10y ago
  the initial idea was that CI deploys lambda, and you promote it from stage to stage. I might consider upload sometime. From what you write I think apex would be better suited for you
ohstopitu 10y ago
How is this any different from Serverless? (not trolling, I could not find any differences in the README)
- kubek2k 10y ago
  I don't mind being trolled :) Serverless is way much more advanced machinery than this tool. My point was to create a tool, not a whole ecosystem around AWS Lambda. When it comes to serverless - my main concern (as it was with Lambda without env variables), is with storing configuration data along the code in the repo. Additionally the reason for creating was mostly pipelines.
brilliantcode 10y ago
what I'd really like is ability to take a swagger spec and click a button to deploy it on AWS Lambda. This is great for internal facing REST Api. I don't know about scaling but hypothetically serverless means it should handle public requests fine.
- kubek2k 10y ago
  I guess you mean the API-Gates + Lambda. Not sure - I think thats something that should be handled by some separate tool (APIGatesgger? ;) - lets hope its not another dirty word)
  - brilliantcode 10y ago
    that's right swagger -> api gateway triggering lambda functions. I was using Chalice but seems like Zappas is another one a HNer pointed out. I couldn't get Chalice to work and sort of gave up on the serverless-api-gatway-lambda idea.
tjholowaychuk 10y ago
Looks cool! I like the release message stuff, I wanted to add similar to apex(1) at some point, not having a clue what the releases actually are in Lambda makes rollbacks a little tricky.
- kubek2k 10y ago
  thats a wonderful improvement of apex, thats for sure :)
aghillo 10y ago
Has anyone characterised server less architectures? I'm interested in testing / exploring the limits with some predictive modelling work and implementing (mathematical) functions as FAAS? No doubt, it 'depends' (e.g. nature of data and its locality or otherwise) but interested to hear if anyone is looking at very fine-granularity server less systems?
stcredzero 10y ago
You acted like no one offers a PaaS for MMOs (and gave it a ridiculous name). I was talking something specifically like Lambda, but where pure functions could be injected into an already running game loop. The name was intentionally ridiculous. Also, you suggested that my proposal would suffer from the latency of lambda requests going over the network, which shows you didn't read my original comment. with your cloud functions being called on triggers Yeah, I pretty much covered that this was not what I was proposing, when I wrote "(You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.)" As far as I knew (which might be out of date) Photon was developing something that would be suitable for a persistent world MMORPG, but not quite ready with it. What I'm proposing would be suitable for an MMORPG, not just MMO game sessions.
- BoorishBears 10y ago
  PaaSes for MMOs with persistent worlds already exist. >...Also, before someone makes the mistake of assuming I'm as stupid as they are... Since you insulted me, I frankly don't care what further squirming you feel like doing to convince yourself you have a novel idea that serves any purpose that is not already served. And I'll gladly dance in the mud with the pig that is yourself and say I think someone who talks like yourself is probably actually pretty stupid, much too stupid to compete with those solutions. In fact, I don't think you have the braincells to rub together and even try implement an existing solution in an MMO, let alone implement a new one. Have a nice life. Actually I was so annoyed by your insult that I had to look you up Peter. You're every bit as pedestrian as I imagined, from your cringeworthy blog to your embarrassment of a GitHub profile.
  - stcredzero 10y ago
    I frankly don't care what further squirming you feel like doing to convince yourself you have a novel idea Excuse me, but where ever did I say that I had a novel idea? I merely pointed out that 1) I wasn't talking about exactly what you thought I was talking about (and after 4 iterations, you still don't get that) and 2) you went and attributed to me a bad idea I had explicitly disavowed in the 1st place. Actually I was so annoyed by your insult that I had to look you up Peter. You're clearly a real class act and a careful reader.

128 points by kubek2k 10y ago

Another nice project that helps to ease lambda workflows is https://github.com/garnaat/kappa https://github.com/garnaat/kappa

mullen 10y ago
There is similar project called Zappa. pip install zappa
- Mizza 10y ago
  Zappa actually uses some of Kappa under the hood, but with many many more features. Learn more here: https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa
  - mullen 10y ago
    Is Zappa a fork of Kappa? I have not used Kappa at all but it looks like it a whole lot.

stcredzero 10y ago

I'm wondering if it would be possible to create something like Lambda, but optimized for multiplayer servers with game loops? What if there was a platform that let you write pure functions to create player accounts, spawn game instances, and handle the transformation of player and game state from tick to tick? (You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.) Clojure, Javascript, and Golang would be good languages for this. The environment possibly would let you inject and replace functions live, lending to a Light Table like experience, but while developing a multiplayer game.

keithwhor 10y ago
You could, but the latency issues (network hops) would make the solution untenable. Function as a Service platforms are ideal for solutions where scalability is a huge concern, latency not so much. These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up. You'd probably want architecture extremely specialized to a game loop, and probably your game loop specifically, if you wanted low-latency, high-availability FaaS for this purpose.
- stcredzero 10y ago
  You could, but the latency issues (network hops) would make the solution untenable. ...ok... Did you read my comment? These functions are all running inside of containers that have been optimized to be "warmed up" extremely quickly, but that warm-up takes time, and the more robust the solution, the longer the warm-up. Yeah, of course that's not going to work. I've basically built a specialized "container" system, but instead of using kernel virtualization, it's actually a specific Go "object" residing in an already running server process. These "containers" (really instances) start running in an environment where we already have a virtual server with a process with a server/game loop up and running. I have a cluster of processes that does this, and idempotently spawns an instance of star system in a multiplayer game consisting of 2^87 star systems. www.emergencevector.com Such a system can still be built on Docker style containers, it's just that those Docker style containers represent an entire sub-cluster of many server processes that can each host many "instance" containers which are where the functions that transform from tick n to tick n+1 reside. You'd probably want architecture extremely specialized to a game loop That is what I said in my comment. and probably your game loop specifically I'm saying that the game loop could be customized/built up by a developer by injecting functions into such a system. (Or, possibly, by injecting such functions into a dev version running locally, then exporting them to a server in the cloud.)
wolfgang42 10y ago
Your basic description sounds like LambdaMOO (written in 1990, the name is a coincidence). The LambdaMOO server has a main event loop which handles user events, then schedules tasks which execute functions ('verbs' in LambdaMOO parlance) to modify the game state. Verbs are not pure per se but can be thought of as a transaction which executes in an atomic, consistent, and isolated manner. Verbs are attached to objects in the game, and can be modified in-game on the fly by players with their programmer bit set. (Programmers can only modify verbs on objects they own.) The LambdaMOO language is a lot like JavaScript with a Luaesque syntax. It has prototypical inheritance, which meshes well with its concept of objects as physical things. For example, to get a new instance of a duck one would create a Generic Duck named "My Duck" ('Generic' being the conventional prefix for an object specifically intended to be used as a prototype.) There was a time in the late 1990s when it regularly supported 300 active users at once (with a certan amount of lag); on modern hardware it could probably handle significantly more than that. If you want a modern scalable system then LambdaMOO itself is probably not for you; the server is single-threaded (in fact the whole thing is remarkably similar to NodeJS with async/await support) and state is saved in-memory and checkpointed periodically (hence the Durability missing from the above description of verbs). However, the basic design (prototypal inheritance, in-database verbs, ACI[D] tasks, etc) is extremely well proven, and would lend itself well to being copied into a modern distributed system like AWS Lambda. As long as you had an ACID datastore to back it up, you could scale the task execution as far as you wanted, and with a bit of clever work you could probably come up with a partitioning scheme to make the database scale as well. In fact, I've partially implemented this already; what's stopped me has primarily been that I'm more interested in the technical aspects and have no idea what I would actually use it for once I built it. If you have a use case for this sort of thing I'd love to hear about it.

michaelbuckbee 10y ago

Very cool project, but it has me thinking what/if it would be possible to build out more of the infrastructure management that Heroku has to make it even easier (domain setup, addon store, etc.)

mifreewil 10y ago

Related: Apex https://www.github.com/apex/apex https://www.github.com/apex/apex

coenhyde 10y ago

This is great. I was going to create something similar to manage lambda functions across environments. Awesome to see you're already using the newly released environment variables for lambdas.

kubek2k 10y ago
thanks! - please give it a try and let me know in the issues if something doesn't work as expected.

keithwhor 10y ago

Hey! Great project. We actually built stdlib [1] to capture an experience similar to Heroku's for building microservices, and we'll be presenting at AWS Re:invent on Tuesday [2]. Our workflows with our public API are accessible via our open source library, would love contributors such as yourself (and anybody else that finds it interesting!). We have a diverse team working on the project - Xooglers, support from early Heroku team members, and even some YC alumni lending a hand. :) [1] https://github.com/poly/stdlib https://github.com/poly/stdlib [2] https://www.portal.reinvent.awsevents.com/connect/sessionDetail.ww?SESSION_ID=8092 https://www.portal.reinvent.awsevents.com/connect/sessionDet...

josegonzalez 10y ago

For a second there, I thought someone implemented a push-to-deploy experience for Heroku (with some of the other Heroku niceties). I actually wrote a blog post about that this week: http://josediazgonzalez.com/2016/11/21/push-to-deploy-lambda-applications/ http://josediazgonzalez.com/2016/11/21/push-to-deploy-lambda...

kubek2k 10y ago
really good read. thx! :bow: I didn't plan to do the push-to-deploy thing, mostly because that would totally complicate things. The rationale was more that we would use the CI that would automatically build and deploy things to DEV env that would be then promoted around.

charlesetc 10y ago

You could easily prefix your special environment variable with 'lambdoku_' so you don't have to fear that people could use it..

kubek2k 10y ago
:bow: https://github.com/kubek2k/lambdoku/issues/8 https://github.com/kubek2k/lambdoku/issues/8

pepicon 10y ago

Lambdoku = lick from ass in portuguese

fiatjaf 10y ago
ahahah, I thought things wouldn't get worse than "Heroku" and "Dokku".

fiatjaf 10y ago

The guide on the README doesn't tell me how am I supposed to push and update a function. Also, does it help with dependencies or something like that? Does it manage packaging files?

kubek2k 10y ago
the initial idea was that CI deploys lambda, and you promote it from stage to stage. I might consider upload sometime. From what you write I think apex would be better suited for you

ohstopitu 10y ago

How is this any different from Serverless? (not trolling, I could not find any differences in the README)

kubek2k 10y ago
I don't mind being trolled :) Serverless is way much more advanced machinery than this tool. My point was to create a tool, not a whole ecosystem around AWS Lambda. When it comes to serverless - my main concern (as it was with Lambda without env variables), is with storing configuration data along the code in the repo. Additionally the reason for creating was mostly pipelines.

brilliantcode 10y ago

what I'd really like is ability to take a swagger spec and click a button to deploy it on AWS Lambda. This is great for internal facing REST Api. I don't know about scaling but hypothetically serverless means it should handle public requests fine.

kubek2k 10y ago
I guess you mean the API-Gates + Lambda. Not sure - I think thats something that should be handled by some separate tool (APIGatesgger? ;) - lets hope its not another dirty word)
- brilliantcode 10y ago
  that's right swagger -> api gateway triggering lambda functions. I was using Chalice but seems like Zappas is another one a HNer pointed out. I couldn't get Chalice to work and sort of gave up on the serverless-api-gatway-lambda idea.

tjholowaychuk 10y ago

Looks cool! I like the release message stuff, I wanted to add similar to apex(1) at some point, not having a clue what the releases actually are in Lambda makes rollbacks a little tricky.

kubek2k 10y ago
thats a wonderful improvement of apex, thats for sure :)

aghillo 10y ago

Has anyone characterised server less architectures? I'm interested in testing / exploring the limits with some predictive modelling work and implementing (mathematical) functions as FAAS? No doubt, it 'depends' (e.g. nature of data and its locality or otherwise) but interested to hear if anyone is looking at very fine-granularity server less systems?

stcredzero 10y ago

You acted like no one offers a PaaS for MMOs (and gave it a ridiculous name). I was talking something specifically like Lambda, but where pure functions could be injected into an already running game loop. The name was intentionally ridiculous. Also, you suggested that my proposal would suffer from the latency of lambda requests going over the network, which shows you didn't read my original comment. with your cloud functions being called on triggers Yeah, I pretty much covered that this was not what I was proposing, when I wrote "(You'd have to choose a particular architecture for synchronization with the client, of course. Also, before someone makes the mistake of assuming I'm as stupid as they are -- I'm not proposing that a function be transmitted, compiled, then evaluated at each tick over the network! Functions would be installed into virtualized servers.)" As far as I knew (which might be out of date) Photon was developing something that would be suitable for a persistent world MMORPG, but not quite ready with it. What I'm proposing would be suitable for an MMORPG, not just MMO game sessions.

BoorishBears 10y ago
PaaSes for MMOs with persistent worlds already exist. >...Also, before someone makes the mistake of assuming I'm as stupid as they are... Since you insulted me, I frankly don't care what further squirming you feel like doing to convince yourself you have a novel idea that serves any purpose that is not already served. And I'll gladly dance in the mud with the pig that is yourself and say I think someone who talks like yourself is probably actually pretty stupid, much too stupid to compete with those solutions. In fact, I don't think you have the braincells to rub together and even try implement an existing solution in an MMO, let alone implement a new one. Have a nice life. Actually I was so annoyed by your insult that I had to look you up Peter. You're every bit as pedestrian as I imagined, from your cringeworthy blog to your embarrassment of a GitHub profile.
- stcredzero 10y ago
  I frankly don't care what further squirming you feel like doing to convince yourself you have a novel idea Excuse me, but where ever did I say that I had a novel idea? I merely pointed out that 1) I wasn't talking about exactly what you thought I was talking about (and after 4 iterations, you still don't get that) and 2) you went and attributed to me a bad idea I had explicitly disavowed in the 1st place. Actually I was so annoyed by your insult that I had to look you up Peter. You're clearly a real class act and a careful reader.