11 ms·

New for AWS Lambda – Environment Variables and Serverless Application Model

264 points by luhn 10y ago

consultutah 10y ago
Interesting. Slowly carving away at the reasons that serverless.com was created.
- Everhusk 10y ago
  Very slowly, Serverless had this a year ago. By the time AWS gets something like a 'sls function deploy' it's pretty likely that serverless will have support multiple cloud providers, in which case they should be pretty safe.
  - cagataygurturk 10y ago
    With cloudformation you can deploy full stack serverless apps. IMHO You don't need serverless any more.
    - workleg 10y ago
      I beg to differ. CloudFormation is great, but crafting CF templates for non-trivial backends is not for the faint of heart. Also iterating on them while tweaking things in CF templates and corresponding logic in micro-services running on Lambda is not the shortest/fastest feedback loop with the current AWS tooling. Also things are pretty hard to test. After over a year of struggling with the current tooling, I've started working on a framework [0] that would allow writing serverless infrastructures at a higher abstraction level. [0]: http://qmu.li http://qmu.li
iwintermute 10y ago
Can anyone share their experiences using AWS Lambda in Production?
- hn_user2 10y ago
  We tried about a year ago and gave up after too much wasted effort on getting any kind of a good dev/CI/CD pipeline going. Mainly due to lack of tooling. Things like environment variables, etc. Taking another look now on some side projects, to get a sense of if all the of the issues have been addressed. But the big ones were lack of env variables, and the tooling was atrocious. I really like the concept, and the current serverless (serverless.com) framework has been simplified, runs based on cloud formation, and also allows the project to be run as an express server, and potentially with additional plugins on a competing serverless architecture.
- avitzurel 10y ago
  I've been using Lambda in production for multiple things. I'm not using Apex or any of the other frameworks, just plain Go code wrapped by the JS function. It's really awesome to work with, you just deploy a function and quit worrying about it. Things like callbacks, message monitoring and stuff like that. Example: Monitor SNS messages for auto-scaling and send a message to slack when something fails. https://github.com/KensoDev/sns-lambda-notifier-golang https://github.com/KensoDev/sns-lambda-notifier-golang
  - jtwaleson 10y ago
    Same here, deployed two functions. One thumbnail converter after s3 uploads which is used a couple of times per day, running for over a year. The other is a log aggregation script that runs thousands of times per hour, running for a few months now. Both run flawlessly, never had to worry about them after deploying.
- Nimsical 10y ago
  I've used Lambda in production under pretty high throughput conditions before. It's fairly stable – given that we were using Kinesis to feed into it (which was a pain to setup). Recently, I've been using stdlib (https://stdlib.com https://stdlib.com) to do function-services since it's a lot easier to get started with it a bit more intuitive for newcomers to understand.
jtwaleson 10y ago
Does anyone know if the 256 character limit for the value is in place like in ECS? Super annoying.
- nitrogen 10y ago
  You could probably split longer data into multiple variables and concatenate them, but maybe having such large environment variables is a sign that you might be better served with a different mechanism. For example, instead of passing in a big JSON blob, one could split the JSON keys into separate environment variables.
  - jtwaleson 10y ago
    I wanted to add a 2K public key, could be split up but annoying solution. Used a IAM role with rights to an S3 object in the end.
    - chronial 10y ago
      If you can, you should switch to elliptic curve keys – they are less than 256 bytes.
- ranman 10y ago
  I just tested and didn't run into a 256 character limit. Looks like you can use anything and any number of variables as long as the total size is <4k.
djhworld 10y ago
The environment variables thing was desperately needed, I raised this as a Feature Request with AWS over a year ago (my work has enterprise support) I remember them telling us a way to make a lambda function distinguish whether it is in a DEV/TEST or PROD environment was to do some sort of regex on the function name, which was suboptimal, especially if you have Lambda's created via CloudFormation. We "got around" this problem by creating tables in DynamoDB as our DEV/PROD environments are in separate AWS accounts, so the dynamo tables contained simple key/value pairs that you would read once when the lambda container started up. Another option would be to have a file in S3 or something, but you still have to write code to manage the retrieval of those resources. Looking forward to dropping all that infrastructure and using this feature instead.
- scrollaway 10y ago
  Did you happen to raise a feature request for Python 3 as well? With any luck, we'll get that before Python 4 is released... :)
  - djhworld 10y ago
    We primarily use Java 8, but I'd imagine a lot of people have raised it
- adjohn 10y ago
  Why aren't you using aliases[1] for denoting dev, prod, etc? [1] http://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html http://docs.aws.amazon.com/lambda/latest/dg/versioning-alias...
  - djhworld 10y ago
    This was before AWS introduced aliases and versioning, think back to summer 2015. Aliases are still a half solution to this problem though, you would still need logic in your code to say "if alias == PROD then ... else ..." for configuration, something I'd rather do without. EDIT: by configuration I mean configuration that's specific to an environment. Imagine you had a Lambda that wrote data to an S3 bucket, you wouldn't want to accidentally mix TEST/LIVE data in the same location, so you would use a configuration property to inform the Lambda where to write.
vmarsy 10y ago
Nice, This could be useful also if you could have "secret" variables as well, similar to [1]. [1] https://www.visualstudio.com/en-us/docs/build/define/variables#secret-variables https://www.visualstudio.com/en-us/docs/build/define/variabl...
schappim 10y ago
Do you use / know of any services like AWS Lambda that supports Ruby out of the box? I've been playing with AWS Lambda using travelling Ruby and Mruby, but have hit issues (with native gems etc). I have used Iron Worker previously, but they seem to be going up market and don't even display pricing on their site. Thanks in advance for your input!
- empath75 10y ago
  My suggestion is to just use Python.
- emidln 10y ago
  You can probably use JRuby without too much effort.
- 33degrees 10y ago
  There are a few libraries for using JRuby (like https://github.com/c9katayama/aws-lambda-jruby https://github.com/c9katayama/aws-lambda-jruby) But they may also have issues with native gems
- kaishiro 10y ago
  I haven't used it it, but the new (like, really new) Iron Functions may be what you're looking for. I've only used the Workers/Cache so far, but they're dockerized so pretty much language agnostic. I'm making an assumption that Functions is similar.
- ch4s3 10y ago
  I wonder if you could use opal?
  - schappim 10y ago
    yeah that was one of my very early thoughts...
    - ch4s3 10y ago
      I'm not normally interested in Opal per se, but it would be great for this. I'd much rather write Ruby for simple call/response APIs than JS.
tedmiston 10y ago
This is a very welcome improvement to an awesome service that I use daily. Now if we could just get a smoother bundling / deployment workflow or a way to edit uploaded bundles on the AWS Lambda dashboard.
brilliantcode 10y ago
Has anyone been able to get Chalice to work? https://github.com/awslabs/chalice https://github.com/awslabs/chalice Basically, I'm interested to know how the Swagger template file will look a like. It'd be nice if you could use Swagger to quickly create REST api on Lambda stack. Being able to quickly generate scaffold CRUD REST api on Lambda behind Authentication + DynamoDB would be an absolute killer app. I'd imagine that Azure is not far behind. But one thing that's killing it for Azure is the Visual Studio IDE. Edit my ASP web app, one click deploy to Azure from inside VS is a killer app as well. I really wish AWS came up with their own IDE where it would be tightly integrated with AWS. Imagine if you could write code and deploy it instantly without setting things up yourself through the AWS console (not that it's bad or anything).
- adjohn 10y ago
  I suspect an Amazon IDE is coming. They did acquire[1] Cloud 9 not too long ago. [1] https://www.google.com/amp/www.forbes.com/sites/janakirammsv/2016/07/18/the-master-plan-behind-amazons-acquisition-of-cloud9-ide/?client=safari https://www.google.com/amp/www.forbes.com/sites/janakirammsv...
- Mizza 10y ago
  I'm biased saying this, but Chalice is a half-baked rip-off of Zappa[0], deliberately nerfed by Amazon to lock you into the AWS stack. If you want to see an example of how easy it is to use Zappa + Lambda + DyanamoDB, you can check out Zappa BitTorrent Tracker, which can use S3 or DynamoDB as a back-end.[1] As a bonus, Zappa has had both local and remote environment variables as a feature for months - although it's pretty cool that this new announcement can use KMS, although that will of course mean more vendor-lock in if you choose to go that route. [0] https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa [1] https://github.com/Miserlou/zappa-bittorrent-tracker https://github.com/Miserlou/zappa-bittorrent-tracker
  - ranman 10y ago
    (I work for AWS) I've been livestreaming building a zappa app on twitch.tv/aws ! daysuntilreinvent.com is powered by zappa! I will say that chalice wasn't meant to copy zappa. Chalice and Zappa both started in Jan of 2016. Zappa has way more features and is a wonderful piece of software. One thing Chalice does well that I'd love to see in Zappa is automatically figuring out what kind of IAM policy/permissions you need by analyzing the code.
skoocda 10y ago
Has anyone been able to use Lambda for relatively high-memory load applications? (~2GB+ RAM) That's our biggest restraint at the moment, so far I haven't seen any good options.
- djhworld 10y ago
  Lambda only supports up to a maximum of 1536mb of RAM right now. I've been involved with developing lambda functions that consume roughly 1.2gb of ram each time, but the memory usage is easy to predict as the function is triggered by files in S3 that are about the same size. They say to break your problem down into smaller chunks to fit into the memory - is that possible in your case?
  - skoocda 10y ago
    Not easily! I'm doing speech recognition; I might be able to run a segment through two separate models, with half the lexicon available in each run, then just combine the results. However I don't think the overall savings will be enough to justify running it twice. It's getting close though, once they get the RAM up more, I'll try it out.
forrestbrazeal 10y ago
My team has been using Lambda/Dynamo/API Gateway/Cognito for a couple of web services this year and really loving it. The addition of CloudWatch Events to the mix was really the final piece of the puzzle for us, because now we can launch lambda functions on a schedule to perform asynchronous background tasks (think data crunching, email notifications, etc). Lambda is maturing so fast that I almost feel bad complaining about it, but of course we have run into our fair share of issues, too. One thing that's making our life more difficult right now is the fact that when you launch a Lambda function in a VPC via CloudFormation, the function's ENI doesn't get attached until the function runs the first time, so CloudFormation doesn't know anything about the ENI. Thus, when you tear down the CF stack, the ENI gets orphaned and hangs around in a detached state. Throw some automation into the mix and you can start eating up IP addresses in your subnet really fast. I have no doubt they will fix this soon.
- manojlds 10y ago
  Interesting. I want to see how Terraform fares in this aspect.
tszming 10y ago
Environments (dev, prod etc) support would be great, currently need to use apex to simulate.
packetslave 10y ago
my current annoyance with Lambda is the lack of support for timezones in the scheduler. we use it for spinning up/down EC2 instances at particular times, and having to manually work around DST is a pain.

264 points by luhn 10y ago

consultutah 10y ago

Interesting. Slowly carving away at the reasons that serverless.com was created.

Everhusk 10y ago
Very slowly, Serverless had this a year ago. By the time AWS gets something like a 'sls function deploy' it's pretty likely that serverless will have support multiple cloud providers, in which case they should be pretty safe.
- cagataygurturk 10y ago
  With cloudformation you can deploy full stack serverless apps. IMHO You don't need serverless any more.
  - workleg 10y ago
    I beg to differ. CloudFormation is great, but crafting CF templates for non-trivial backends is not for the faint of heart. Also iterating on them while tweaking things in CF templates and corresponding logic in micro-services running on Lambda is not the shortest/fastest feedback loop with the current AWS tooling. Also things are pretty hard to test. After over a year of struggling with the current tooling, I've started working on a framework [0] that would allow writing serverless infrastructures at a higher abstraction level. [0]: http://qmu.li http://qmu.li

iwintermute 10y ago

Can anyone share their experiences using AWS Lambda in Production?

hn_user2 10y ago
We tried about a year ago and gave up after too much wasted effort on getting any kind of a good dev/CI/CD pipeline going. Mainly due to lack of tooling. Things like environment variables, etc. Taking another look now on some side projects, to get a sense of if all the of the issues have been addressed. But the big ones were lack of env variables, and the tooling was atrocious. I really like the concept, and the current serverless (serverless.com) framework has been simplified, runs based on cloud formation, and also allows the project to be run as an express server, and potentially with additional plugins on a competing serverless architecture.
avitzurel 10y ago
I've been using Lambda in production for multiple things. I'm not using Apex or any of the other frameworks, just plain Go code wrapped by the JS function. It's really awesome to work with, you just deploy a function and quit worrying about it. Things like callbacks, message monitoring and stuff like that. Example: Monitor SNS messages for auto-scaling and send a message to slack when something fails. https://github.com/KensoDev/sns-lambda-notifier-golang https://github.com/KensoDev/sns-lambda-notifier-golang
- jtwaleson 10y ago
  Same here, deployed two functions. One thumbnail converter after s3 uploads which is used a couple of times per day, running for over a year. The other is a log aggregation script that runs thousands of times per hour, running for a few months now. Both run flawlessly, never had to worry about them after deploying.
Nimsical 10y ago
I've used Lambda in production under pretty high throughput conditions before. It's fairly stable – given that we were using Kinesis to feed into it (which was a pain to setup). Recently, I've been using stdlib (https://stdlib.com https://stdlib.com) to do function-services since it's a lot easier to get started with it a bit more intuitive for newcomers to understand.

jtwaleson 10y ago

Does anyone know if the 256 character limit for the value is in place like in ECS? Super annoying.

nitrogen 10y ago
You could probably split longer data into multiple variables and concatenate them, but maybe having such large environment variables is a sign that you might be better served with a different mechanism. For example, instead of passing in a big JSON blob, one could split the JSON keys into separate environment variables.
- jtwaleson 10y ago
  I wanted to add a 2K public key, could be split up but annoying solution. Used a IAM role with rights to an S3 object in the end.
  - chronial 10y ago
    If you can, you should switch to elliptic curve keys – they are less than 256 bytes.
ranman 10y ago
I just tested and didn't run into a 256 character limit. Looks like you can use anything and any number of variables as long as the total size is <4k.

djhworld 10y ago

The environment variables thing was desperately needed, I raised this as a Feature Request with AWS over a year ago (my work has enterprise support) I remember them telling us a way to make a lambda function distinguish whether it is in a DEV/TEST or PROD environment was to do some sort of regex on the function name, which was suboptimal, especially if you have Lambda's created via CloudFormation. We "got around" this problem by creating tables in DynamoDB as our DEV/PROD environments are in separate AWS accounts, so the dynamo tables contained simple key/value pairs that you would read once when the lambda container started up. Another option would be to have a file in S3 or something, but you still have to write code to manage the retrieval of those resources. Looking forward to dropping all that infrastructure and using this feature instead.

scrollaway 10y ago
Did you happen to raise a feature request for Python 3 as well? With any luck, we'll get that before Python 4 is released... :)
- djhworld 10y ago
  We primarily use Java 8, but I'd imagine a lot of people have raised it
adjohn 10y ago
Why aren't you using aliases[1] for denoting dev, prod, etc? [1] http://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html http://docs.aws.amazon.com/lambda/latest/dg/versioning-alias...
- djhworld 10y ago
  This was before AWS introduced aliases and versioning, think back to summer 2015. Aliases are still a half solution to this problem though, you would still need logic in your code to say "if alias == PROD then ... else ..." for configuration, something I'd rather do without. EDIT: by configuration I mean configuration that's specific to an environment. Imagine you had a Lambda that wrote data to an S3 bucket, you wouldn't want to accidentally mix TEST/LIVE data in the same location, so you would use a configuration property to inform the Lambda where to write.

vmarsy 10y ago

Nice, This could be useful also if you could have "secret" variables as well, similar to [1]. [1] https://www.visualstudio.com/en-us/docs/build/define/variables#secret-variables https://www.visualstudio.com/en-us/docs/build/define/variabl...

schappim 10y ago

Do you use / know of any services like AWS Lambda that supports Ruby out of the box? I've been playing with AWS Lambda using travelling Ruby and Mruby, but have hit issues (with native gems etc). I have used Iron Worker previously, but they seem to be going up market and don't even display pricing on their site. Thanks in advance for your input!

empath75 10y ago
My suggestion is to just use Python.
emidln 10y ago
You can probably use JRuby without too much effort.
33degrees 10y ago
There are a few libraries for using JRuby (like https://github.com/c9katayama/aws-lambda-jruby https://github.com/c9katayama/aws-lambda-jruby) But they may also have issues with native gems
kaishiro 10y ago
I haven't used it it, but the new (like, really new) Iron Functions may be what you're looking for. I've only used the Workers/Cache so far, but they're dockerized so pretty much language agnostic. I'm making an assumption that Functions is similar.
ch4s3 10y ago
I wonder if you could use opal?
- schappim 10y ago
  yeah that was one of my very early thoughts...
  - ch4s3 10y ago
    I'm not normally interested in Opal per se, but it would be great for this. I'd much rather write Ruby for simple call/response APIs than JS.

tedmiston 10y ago

This is a very welcome improvement to an awesome service that I use daily. Now if we could just get a smoother bundling / deployment workflow or a way to edit uploaded bundles on the AWS Lambda dashboard.

brilliantcode 10y ago

Has anyone been able to get Chalice to work? https://github.com/awslabs/chalice https://github.com/awslabs/chalice Basically, I'm interested to know how the Swagger template file will look a like. It'd be nice if you could use Swagger to quickly create REST api on Lambda stack. Being able to quickly generate scaffold CRUD REST api on Lambda behind Authentication + DynamoDB would be an absolute killer app. I'd imagine that Azure is not far behind. But one thing that's killing it for Azure is the Visual Studio IDE. Edit my ASP web app, one click deploy to Azure from inside VS is a killer app as well. I really wish AWS came up with their own IDE where it would be tightly integrated with AWS. Imagine if you could write code and deploy it instantly without setting things up yourself through the AWS console (not that it's bad or anything).

adjohn 10y ago
I suspect an Amazon IDE is coming. They did acquire[1] Cloud 9 not too long ago. [1] https://www.google.com/amp/www.forbes.com/sites/janakirammsv/2016/07/18/the-master-plan-behind-amazons-acquisition-of-cloud9-ide/?client=safari https://www.google.com/amp/www.forbes.com/sites/janakirammsv...
Mizza 10y ago
I'm biased saying this, but Chalice is a half-baked rip-off of Zappa[0], deliberately nerfed by Amazon to lock you into the AWS stack. If you want to see an example of how easy it is to use Zappa + Lambda + DyanamoDB, you can check out Zappa BitTorrent Tracker, which can use S3 or DynamoDB as a back-end.[1] As a bonus, Zappa has had both local and remote environment variables as a feature for months - although it's pretty cool that this new announcement can use KMS, although that will of course mean more vendor-lock in if you choose to go that route. [0] https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa [1] https://github.com/Miserlou/zappa-bittorrent-tracker https://github.com/Miserlou/zappa-bittorrent-tracker
- ranman 10y ago
  (I work for AWS) I've been livestreaming building a zappa app on twitch.tv/aws ! daysuntilreinvent.com is powered by zappa! I will say that chalice wasn't meant to copy zappa. Chalice and Zappa both started in Jan of 2016. Zappa has way more features and is a wonderful piece of software. One thing Chalice does well that I'd love to see in Zappa is automatically figuring out what kind of IAM policy/permissions you need by analyzing the code.

skoocda 10y ago

Has anyone been able to use Lambda for relatively high-memory load applications? (~2GB+ RAM) That's our biggest restraint at the moment, so far I haven't seen any good options.

djhworld 10y ago
Lambda only supports up to a maximum of 1536mb of RAM right now. I've been involved with developing lambda functions that consume roughly 1.2gb of ram each time, but the memory usage is easy to predict as the function is triggered by files in S3 that are about the same size. They say to break your problem down into smaller chunks to fit into the memory - is that possible in your case?
- skoocda 10y ago
  Not easily! I'm doing speech recognition; I might be able to run a segment through two separate models, with half the lexicon available in each run, then just combine the results. However I don't think the overall savings will be enough to justify running it twice. It's getting close though, once they get the RAM up more, I'll try it out.

forrestbrazeal 10y ago

My team has been using Lambda/Dynamo/API Gateway/Cognito for a couple of web services this year and really loving it. The addition of CloudWatch Events to the mix was really the final piece of the puzzle for us, because now we can launch lambda functions on a schedule to perform asynchronous background tasks (think data crunching, email notifications, etc). Lambda is maturing so fast that I almost feel bad complaining about it, but of course we have run into our fair share of issues, too. One thing that's making our life more difficult right now is the fact that when you launch a Lambda function in a VPC via CloudFormation, the function's ENI doesn't get attached until the function runs the first time, so CloudFormation doesn't know anything about the ENI. Thus, when you tear down the CF stack, the ENI gets orphaned and hangs around in a detached state. Throw some automation into the mix and you can start eating up IP addresses in your subnet really fast. I have no doubt they will fix this soon.

manojlds 10y ago
Interesting. I want to see how Terraform fares in this aspect.

tszming 10y ago

Environments (dev, prod etc) support would be great, currently need to use apex to simulate.

packetslave 10y ago

my current annoyance with Lambda is the lack of support for timezones in the scheduler. we use it for spinning up/down EC2 instances at particular times, and having to manually work around DST is a pain.