7 ms·

Show HN: I made a local development, debugging and hotreload tool for AWS Lambda

111 points by sozal 4y ago

MerLoc is a live AWS Lambda function development and debugging tool. MerLoc allows you to run AWS Lambda functions on your local while they are still part of a flow in the AWS cloud remote. MerLoc makes it possible to - test your function locally without deploy to the AWS Lambda environment (so no wait for build, package and deploy) - debug your function by putting breakpoints from your IDE - hot-reload updated function on your local automatically to apply changes automatically (so again no wait for build, package and deploy) - run the individual function locally while it is still part of flow shown above and use real requests from the AWS Lambda environment

cube2222 4y ago
That looks cool! Though it looks like it only works for Java-based Lambdas, unfortunately. Does it only work with HTTP Lambdas / manual invocations, or does it also play well with others, like SQS subscriptions? I think that's the biggest pain point for local Lambda development with the AWS-provided tooling for me right now, as detailed in a blog post I wrote a while ago[0]. All the official tooling only lets you do HTTP calls and manual invocations for locally running Lambdas, which means you can't really test architectures comprised of multiple Lambdas, or where Lambdas are just a part of the whole setup. [0]: https://spacelift.io/blog/aws-lambda-migration https://spacelift.io/blog/aws-lambda-migration
- sozal 4y ago
  Hi, Yes it is only for Java runtime but Node.js runtime is also in our near-term roadmap. If you are looking for Node.js runtime, could you please upvote issue here: https://github.com/thundra-io/merloc/issues/1 https://github.com/thundra-io/merloc/issues/1 or create a new one if you have any other request. It works for *all* Lambda invocation triggers: Manual, HTTP, SQS, SNS, Kinesis etc ...
heisen_berg 4y ago
Is this project open source? If so, is there any limitation to install any local environment?
- sozal 4y ago
  Yes, it is open-source on Github. You can clone repository and install the platform to your own AWS account and run it inside your own network.
aoozdemir 4y ago
Does the hotreload feature comes out of the box? Is there any limitation regarding what can we reload while debugging? Could you expand that feature a bit?
- sozal 4y ago
  Yes, hot-reload features comes out of the box and you don't need install any other tool or service. There is no limitation for debugging. When you changed any source code, it is compiled and then related functions are reloaded on your local automatically. So when you apply a change or fix for your function, you can check whether it works or not without deploy to AWS Lambda environment. So it increases serverless application development productivity by helping you developing and debugging faster.
somethingAlex 4y ago
I find the debugging and testing process pretty slow with AWS SAM so I may have to try this out when you have a Node runtime.
- sozal 4y ago
  Yes, there any many serverless people looking for Node support. Could you please thumbs-up its feature request here: https://github.com/thundra-io/merloc/issues/1 https://github.com/thundra-io/merloc/issues/1
  - sozal 4y ago
    [Disclaimer] I am the co-founder and CTO at Thundra which is serverless monitoring and debugging solution. We also have developed another debugging tool to debug AWS Lambda function on remote AWS Lambda environment from your local IDE: https://www.serverlessdebugger.com/ https://www.serverlessdebugger.com/ It doesn't support hot-reload but very useful for debugging functions.
- tunesmith 4y ago
  Doesn't something like localstack.cloud help with this need? I've just recently inherited some NodeJS lambdas and am thinking of looking into that to debug/develop locally.
  - edgyquant 4y ago
    Local stack seems to be invite only
    - kadoban 4y ago
      I suspect you misunderstood. It's Open Source, at least the free version. Or maybe I'm missing something?
      - trwhite 4y ago
        There's a free trial that lasts 2 weeks. After that I think it's paid.
gee_totes 4y ago
This is awesome and something I definitely would use for my docker and node based lambdas. Looking forward to this project adding more labmda backends!
- sozal 4y ago
  Hi, Thanks. There any many people looking for Node support as it is the most common serverless runtime. Could you please thumbs-up its feature request here: https://github.com/thundra-io/merloc/issues/1 https://github.com/thundra-io/merloc/issues/1
_gabe_ 4y ago
It still amazes me that AWS doesn't provide good local debugging/development support out of the box. I feel like we're slowly going back in time. Iteration cycles are very high, debugging cloud dev is essentially placing a bunch of print statements and trying to trace the code, build times take forever... Edit: Here's some more conversations I found on the topic of the terrible workflow for cloud dev if anyone is interested https://news.ycombinator.com/item?id=26855037 https://news.ycombinator.com/item?id=26855037
- sozal 4y ago
  You are right, local development and debugging is still one of the biggest pains of serverless world. This was our motivation to develop MerLoc
  - _gabe_ 4y ago
    It looks awesome :) Unfortunately we have a full Javascript stack at my company, but I'll keep an eye on this in case anybody adds more dev environments!
- user3939382 4y ago
  I just got xdebug working in ECS. It took really understanding xdebug config in depth but it’s amazing now that it’s working.
- tlarkworthy 4y ago
  Yeah, you are right. I made https://webcode.run https://webcode.run also for this reason (also hot code reload and debugging but for JavaScript computational notebooks)
- deleted 4y ago
  [deleted]
- scarface74 4y ago
  There is “sam local”. But honestly, all a Lambda is a script with a function that takes two parameters - an event and a context. You can scaffold testing and development just like you would a controller action in an MVC framework - use a test harness that calls the method with the correct JSON payload and usually a null event context.
vasinov 4y ago
How is MerLoc different from SST?
- sozal 4y ago
  Hi, Current version of MerLoc is similar to SST except some minor details. The biggest difference so far is that you can use MerLoc with any deployment tool like AWS CDK, AWS CloudFormation, AWS SAM, Serverless Framework, SST, Terraform, etc ... And we are also working on an improvement which will let you use the same network with the remote function on your local.
kulikalov 4y ago
You are missing the point. Lambda functions are made for tiny scripts or relaying events that barely need any debugging. Also, they are highly integrated with other cloud services so it doesn’t make much sense to bother about complex dev tools for them.
- phailhaus 4y ago
  That's no longer the case. Lambdas have fantastic applications to more complex use cases, so better debugging capabilities are now necessary.
  - sozal 4y ago
    You are definitely right. Serverless is much more than 5 years before
- linsomniac 4y ago
  Thus spake the Master Programmer: "Though a program be but three lines long, someday it will have to be maintained."
- icedchai 4y ago
  Some people have gone full serverless, right or wrong (and I happen to think you are correct: Lambdas are best for simple scripts.)
spatley 4y ago
this looks like a great solution to problems in serverless application development. I would be interested in how this does or does not overlap with the AWS toolkit: https://aws.amazon.com/intellij/ https://aws.amazon.com/intellij/ https://aws.amazon.com/pycharm/ https://aws.amazon.com/pycharm/ https://aws.amazon.com/visualstudiocode/ https://aws.amazon.com/visualstudiocode/ I have used the Pycharm in Python Lambda development and I was able to set a breakpoint in my IDE and hit it and see the status of all my variables. Is MerLoc an alternative to that?
- sozal 4y ago
  Hi, As far as I see, yhe plugins you mentioned just runs the function as a standalone app. But MerLoc runs it on your local as a part of AWS cloud infr This means that function is triggered from real events, runs againts real AWS services and again response is returned to real AWS services
isuckatcoding 4y ago
Are Lambda functions doing so much now that they they can’t just be run locally? I get that if you’re trying to access some AWS service, this could be helpful but in that case I thought the general consensus is basically mock things out.
- hk1337 4y ago
  Some people cram their entire application into a single lambda. Others are a little better and have their application spread out amongst multiple lambdas.
- icedchai 4y ago
  The lambda developer experience is pretty awful. I generally put "main" methods in all my lambdas so I can test them locally, but otherwise hitting actual AWS APIs. I'm not a big fan of mocking. Doing any sort of edit/deploy cycle is way too slow.
- phillu 4y ago
  I had great success with unit testing the lambda code and mocking external/aws services. For serious software you want a proper local/unit test setup anyway. Using this in your dev cycles is just the natural next step. Instead of waiting for the cdk/cfn deploy to finish, which really is way too slow, you just execute 'jest' or whatever test framework you have. Sure, figuring out how to mock everything takes some upfront time, but it pays off in the long run.
- xsmasher 4y ago
  I always write a remote function as just "an async function that takes type A and returns type B" - then I can write tests for it very easily and run it locally, IF it doesn't use any AWS services like databases. For deployment I have a generic "wrapper" that makes it lambda-compatible by converting a webEvent -> Json and my outputJSON -> web response event.
whummer 4y ago
Disclaimer: I'm the co-founder and CTO of LocalStack (https://localstack.cloud https://localstack.cloud) This looks awesome, congrats Serkan and team! Indeed, looks similar to SST, with the extensions you've mentioned. I can see this becoming very useful if you want the local debuggability of Lambdas in your IDE, with code running remotely. (i.e., not leveraging locally emulated services) Looking forward to giving it a whirl and see how it can integrate with (or even run against) LocalStack..! :) Keep up the great work!
- sozal 4y ago
  Thanks Waldemar. In fact it is already integrated with LocalStack
- deleted 4y ago
  [deleted]
scarface74 4y ago
What does this do that SAM local can’t do for testing Lambdas locally and “sam sync” for deploying a single function? https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-sync.html https://docs.aws.amazon.com/serverless-application-model/lat...
- sozal 4y ago
  Hi, The difference is that MerLoc runs functions on your local as a part of AWS cloud infa. This means that function is triggered from real events, runs againts real AWS services and again response is returned to real AWS services.
  - scarface74 4y ago
    Isn’t that slower than just mocking up your event payload?
    - sozal 4y ago
      Actually not. It is just one time setup and no mocking is needed anymore
yrgulation 4y ago
This is actually a tool i’d need to use - thank you for creating it. However i cant help but notice how large corps, such as amazon, build products on top of freely available software, intentionally leave gaps in their offering (aws local dev is horrible, amazon doesnt provide a local env so we wont build an alternative - but is happy to use ours) and then the community steps in and fills in the gaps. I yet have to understand how they get away with such things. Hopefully amazon will have the decency to sponsor such projects.

111 points by sozal 4y ago

cube2222 4y ago

That looks cool! Though it looks like it only works for Java-based Lambdas, unfortunately. Does it only work with HTTP Lambdas / manual invocations, or does it also play well with others, like SQS subscriptions? I think that's the biggest pain point for local Lambda development with the AWS-provided tooling for me right now, as detailed in a blog post I wrote a while ago[0]. All the official tooling only lets you do HTTP calls and manual invocations for locally running Lambdas, which means you can't really test architectures comprised of multiple Lambdas, or where Lambdas are just a part of the whole setup. [0]: https://spacelift.io/blog/aws-lambda-migration https://spacelift.io/blog/aws-lambda-migration

sozal 4y ago
Hi, Yes it is only for Java runtime but Node.js runtime is also in our near-term roadmap. If you are looking for Node.js runtime, could you please upvote issue here: https://github.com/thundra-io/merloc/issues/1 https://github.com/thundra-io/merloc/issues/1 or create a new one if you have any other request. It works for *all* Lambda invocation triggers: Manual, HTTP, SQS, SNS, Kinesis etc ...

heisen_berg 4y ago

Is this project open source? If so, is there any limitation to install any local environment?

sozal 4y ago
Yes, it is open-source on Github. You can clone repository and install the platform to your own AWS account and run it inside your own network.

aoozdemir 4y ago

Does the hotreload feature comes out of the box? Is there any limitation regarding what can we reload while debugging? Could you expand that feature a bit?

sozal 4y ago
Yes, hot-reload features comes out of the box and you don't need install any other tool or service. There is no limitation for debugging. When you changed any source code, it is compiled and then related functions are reloaded on your local automatically. So when you apply a change or fix for your function, you can check whether it works or not without deploy to AWS Lambda environment. So it increases serverless application development productivity by helping you developing and debugging faster.

somethingAlex 4y ago

I find the debugging and testing process pretty slow with AWS SAM so I may have to try this out when you have a Node runtime.

sozal 4y ago
Yes, there any many serverless people looking for Node support. Could you please thumbs-up its feature request here: https://github.com/thundra-io/merloc/issues/1 https://github.com/thundra-io/merloc/issues/1
- sozal 4y ago
  [Disclaimer] I am the co-founder and CTO at Thundra which is serverless monitoring and debugging solution. We also have developed another debugging tool to debug AWS Lambda function on remote AWS Lambda environment from your local IDE: https://www.serverlessdebugger.com/ https://www.serverlessdebugger.com/ It doesn't support hot-reload but very useful for debugging functions.
tunesmith 4y ago
Doesn't something like localstack.cloud help with this need? I've just recently inherited some NodeJS lambdas and am thinking of looking into that to debug/develop locally.
- edgyquant 4y ago
  Local stack seems to be invite only
  - kadoban 4y ago
    I suspect you misunderstood. It's Open Source, at least the free version. Or maybe I'm missing something?
    - trwhite 4y ago
      There's a free trial that lasts 2 weeks. After that I think it's paid.

gee_totes 4y ago

This is awesome and something I definitely would use for my docker and node based lambdas. Looking forward to this project adding more labmda backends!

sozal 4y ago
Hi, Thanks. There any many people looking for Node support as it is the most common serverless runtime. Could you please thumbs-up its feature request here: https://github.com/thundra-io/merloc/issues/1 https://github.com/thundra-io/merloc/issues/1

_gabe_ 4y ago

It still amazes me that AWS doesn't provide good local debugging/development support out of the box. I feel like we're slowly going back in time. Iteration cycles are very high, debugging cloud dev is essentially placing a bunch of print statements and trying to trace the code, build times take forever... Edit: Here's some more conversations I found on the topic of the terrible workflow for cloud dev if anyone is interested https://news.ycombinator.com/item?id=26855037 https://news.ycombinator.com/item?id=26855037

sozal 4y ago
You are right, local development and debugging is still one of the biggest pains of serverless world. This was our motivation to develop MerLoc
- _gabe_ 4y ago
  It looks awesome :) Unfortunately we have a full Javascript stack at my company, but I'll keep an eye on this in case anybody adds more dev environments!
user3939382 4y ago
I just got xdebug working in ECS. It took really understanding xdebug config in depth but it’s amazing now that it’s working.
tlarkworthy 4y ago
Yeah, you are right. I made https://webcode.run https://webcode.run also for this reason (also hot code reload and debugging but for JavaScript computational notebooks)
deleted 4y ago
[deleted]
scarface74 4y ago
There is “sam local”. But honestly, all a Lambda is a script with a function that takes two parameters - an event and a context. You can scaffold testing and development just like you would a controller action in an MVC framework - use a test harness that calls the method with the correct JSON payload and usually a null event context.

vasinov 4y ago

How is MerLoc different from SST?

sozal 4y ago
Hi, Current version of MerLoc is similar to SST except some minor details. The biggest difference so far is that you can use MerLoc with any deployment tool like AWS CDK, AWS CloudFormation, AWS SAM, Serverless Framework, SST, Terraform, etc ... And we are also working on an improvement which will let you use the same network with the remote function on your local.

kulikalov 4y ago

You are missing the point. Lambda functions are made for tiny scripts or relaying events that barely need any debugging. Also, they are highly integrated with other cloud services so it doesn’t make much sense to bother about complex dev tools for them.

phailhaus 4y ago
That's no longer the case. Lambdas have fantastic applications to more complex use cases, so better debugging capabilities are now necessary.
- sozal 4y ago
  You are definitely right. Serverless is much more than 5 years before
linsomniac 4y ago
Thus spake the Master Programmer: "Though a program be but three lines long, someday it will have to be maintained."
icedchai 4y ago
Some people have gone full serverless, right or wrong (and I happen to think you are correct: Lambdas are best for simple scripts.)

spatley 4y ago

this looks like a great solution to problems in serverless application development. I would be interested in how this does or does not overlap with the AWS toolkit: https://aws.amazon.com/intellij/ https://aws.amazon.com/intellij/ https://aws.amazon.com/pycharm/ https://aws.amazon.com/pycharm/ https://aws.amazon.com/visualstudiocode/ https://aws.amazon.com/visualstudiocode/ I have used the Pycharm in Python Lambda development and I was able to set a breakpoint in my IDE and hit it and see the status of all my variables. Is MerLoc an alternative to that?

sozal 4y ago
Hi, As far as I see, yhe plugins you mentioned just runs the function as a standalone app. But MerLoc runs it on your local as a part of AWS cloud infr This means that function is triggered from real events, runs againts real AWS services and again response is returned to real AWS services

isuckatcoding 4y ago

Are Lambda functions doing so much now that they they can’t just be run locally? I get that if you’re trying to access some AWS service, this could be helpful but in that case I thought the general consensus is basically mock things out.

hk1337 4y ago
Some people cram their entire application into a single lambda. Others are a little better and have their application spread out amongst multiple lambdas.
icedchai 4y ago
The lambda developer experience is pretty awful. I generally put "main" methods in all my lambdas so I can test them locally, but otherwise hitting actual AWS APIs. I'm not a big fan of mocking. Doing any sort of edit/deploy cycle is way too slow.
phillu 4y ago
I had great success with unit testing the lambda code and mocking external/aws services. For serious software you want a proper local/unit test setup anyway. Using this in your dev cycles is just the natural next step. Instead of waiting for the cdk/cfn deploy to finish, which really is way too slow, you just execute 'jest' or whatever test framework you have. Sure, figuring out how to mock everything takes some upfront time, but it pays off in the long run.
xsmasher 4y ago
I always write a remote function as just "an async function that takes type A and returns type B" - then I can write tests for it very easily and run it locally, IF it doesn't use any AWS services like databases. For deployment I have a generic "wrapper" that makes it lambda-compatible by converting a webEvent -> Json and my outputJSON -> web response event.

whummer 4y ago

Disclaimer: I'm the co-founder and CTO of LocalStack (https://localstack.cloud https://localstack.cloud) This looks awesome, congrats Serkan and team! Indeed, looks similar to SST, with the extensions you've mentioned. I can see this becoming very useful if you want the local debuggability of Lambdas in your IDE, with code running remotely. (i.e., not leveraging locally emulated services) Looking forward to giving it a whirl and see how it can integrate with (or even run against) LocalStack..! :) Keep up the great work!

sozal 4y ago
Thanks Waldemar. In fact it is already integrated with LocalStack
deleted 4y ago
[deleted]

scarface74 4y ago

What does this do that SAM local can’t do for testing Lambdas locally and “sam sync” for deploying a single function? https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-sync.html https://docs.aws.amazon.com/serverless-application-model/lat...

sozal 4y ago
Hi, The difference is that MerLoc runs functions on your local as a part of AWS cloud infa. This means that function is triggered from real events, runs againts real AWS services and again response is returned to real AWS services.
- scarface74 4y ago
  Isn’t that slower than just mocking up your event payload?
  - sozal 4y ago
    Actually not. It is just one time setup and no mocking is needed anymore

yrgulation 4y ago

This is actually a tool i’d need to use - thank you for creating it. However i cant help but notice how large corps, such as amazon, build products on top of freely available software, intentionally leave gaps in their offering (aws local dev is horrible, amazon doesnt provide a local env so we wont build an alternative - but is happy to use ours) and then the community steps in and fills in the gaps. I yet have to understand how they get away with such things. Hopefully amazon will have the decency to sponsor such projects.