5 ms·

Effe, a building block for an open source AWS lambda

42 points by siscia 11y ago

wsh91 11y ago
As someone who's using Lambda in production today (successfully, at relatively high volume), I'd like to suggest that what most would-be Lambda alternatives miss is the infrastructure piece. Talk of lock-in notwithstanding, running your code on Lambda requires minor changes at most (assuming it's already structured as a callback). The real value is not having to manage EC2 boxes via any abstraction other than the Lambda service itself. You set one knob (CPU/RAM). It does the rest. That's the part that adds value for me as a developer and an operator.
ninjakeyboard 11y ago
"Bring to you by RedBeardLab" I would have someone proof read this for you. It's hard to go through this introduction as it is.
- lsaferite 11y ago
  Since they are likely non-native English speakers (from Italy), a few grammatical mistakes are forgivable.
alpb 11y ago
This looks interesting but the screencast video kind of failed to make the point in my opinion. Take a look at serverless talks or apex intro https://medium.com/@tjholowaychuk/introducing-apex-800824ffaa70 https://medium.com/@tjholowaychuk/introducing-apex-800824ffa... for how to prepare a better illustration that proves your point. > Isolation is not a real issues today, containers can provide complete isolation between runtimes and much more. Most people will not necessarily agree. Containers don’t provide complete isolation and from a theoretical standpoint they never will (unless we rewrite the Linux kernel). There have been security vulnerabilities in the past about this and there will be, it’s just the way it is. Your argument here should probably be ”it's safe for single tenant use“ i.e. all the code runs on effe is your company's code. Also coming back to the video again, I just can't fully understand what this project does, it just looks like a Go web server framework.
- _jezell_ 11y ago
  You could use the same logic to say virtualization isn't safe, but it doesn't stop cloud providers from putting multiple people's VMs on the same instance. Regardless of the technologies you use in your datacenter, there will be vulnerabilities that will need to be found, patched, and fixed.
- siscia 11y ago
  The point is to have a single binary for every single endpoint you need. Now you can deploy those binaries in whatever platform makes sense for you and scale those endpoint independently. All being open source.
hassy 11y ago
The power of Lambda comes from being able to use other AWS services. Lambdas on their own are not very interesting. A true Lambda alternative, open-source or otherwise, would also have to offer alternatives to DynamoDB, S3, SNS/SQS etc. Without those you're basically reinventing CGI.
- npx 11y ago
  I've been giving some thought to an AWS Lambda equivalent on Joyent Triton (largely because it's open source). You'd have RethinkDB (DynamoDB), Manta (S3), your choice of queue, TensorFlow, etc. I'm generally a huge fan of the Joyent engineering ethos (not affiliated, just a fan). Python 3.5 with DTrace probes in an AWS Lambda equivalent would be sweet.
  - _jezell_ 11y ago
    Something running on Kubernetes is probably going to get more traction at this point than something running on Triton. 1.2 added a task API, probably not high performance enough yet for this, but I think it's only a matter of time till the ability to execute ad hoc tasks is a core platform expectation.
- siscia 11y ago
  All of those services have very valid Open Source alternative, why be locked in a closed environment ?
  - nl 11y ago
    So the point of Lambda is that you don't have to run any infrastructure. If Lambda had an open source equivalent, the "other pieces" are also needed to make it useful. The "Open Sourceness" is only one factor in that environment - it is at least as important that you can somehow call them from a stateless service and get adequate performance. That tends to push towards all the services being run in the same environment (although it doesn't have to be like that - it's just easier)
  - hassy 11y ago
tacos 11y ago
If you're tied to AWS then Lambda is great. If you don't want to be tied to AWS or if you need more control over resource allocation/longer running jobs then that's what all the emerging "container engine" services are for.
jorgecurio 11y ago
> In November 2014 AWS Lambda was released and it was nothing short than a revolution. still not good enough to be reliable in production, more expensive than running on micro instance or digitalocean. Lambda needs a serious makeover soon or it risks developer abandonment. It's great to lock yourself in with a cloud provider. That's the only serious benefit you get. You still need to figure out how to scale properly, getting on AWS helps but it's not a plug and play scalable thing. I've largely abandoned development on Lambda because you have to upload your code every time which is annoying. Not only that, they randomly take 5~10 seconds to respond. Also unhelpful are service error messages with no explanation or help from discussion boards. I was told to purchase their support license. So it seems like AWS is locking in developers and companies with a non user friendly documentation and arcane hidden pitfalls then leaving them no choice but to purchase support since the cost is cheap (pay to play right?) until you add up the effort of integrating with AWS and the cost of lock-in.... AWS lambda is great for prototyping but it doesn't have the tools or a proper development guideline to utilize it. There doesn't seem to be much enthusiasm from AWS either, and if they pull the plug, well hope you didn't build any mission critical apps on lambda backends because you are going to have some considerable downtime.
- wsh91 11y ago
  Can you share some more details about what you're trying to do with us? There's a latency floor when using the API Gateway of about 200 ms, which is unacceptable for a lot of APIs, but other than that we largely haven't been running into the issues you're dealing with. And if Meetup is any indication, AWS seems really invested in Lambda. (In terms of cost, it really would depend on what you're using it for.)
  - jorgecurio 11y ago
    well I built a user authentication and it takes 5~6 seconds for a real simple operation. it happens randomly and no way to debug it. the opacity of being able to peer at what's going on in the backend is a real problem with AWS Lambda. I think that this lack of insight is going to bite down the road. Even now, I'm getting 'service error' with NO EXPLANATION what happened.
manojlds 11y ago
Is the "Bring to you by RedBeardLab" intentional or something?

42 points by siscia 11y ago

wsh91 11y ago

As someone who's using Lambda in production today (successfully, at relatively high volume), I'd like to suggest that what most would-be Lambda alternatives miss is the infrastructure piece. Talk of lock-in notwithstanding, running your code on Lambda requires minor changes at most (assuming it's already structured as a callback). The real value is not having to manage EC2 boxes via any abstraction other than the Lambda service itself. You set one knob (CPU/RAM). It does the rest. That's the part that adds value for me as a developer and an operator.

ninjakeyboard 11y ago

"Bring to you by RedBeardLab" I would have someone proof read this for you. It's hard to go through this introduction as it is.

lsaferite 11y ago
Since they are likely non-native English speakers (from Italy), a few grammatical mistakes are forgivable.

alpb 11y ago

This looks interesting but the screencast video kind of failed to make the point in my opinion. Take a look at serverless talks or apex intro https://medium.com/@tjholowaychuk/introducing-apex-800824ffaa70 https://medium.com/@tjholowaychuk/introducing-apex-800824ffa... for how to prepare a better illustration that proves your point. > Isolation is not a real issues today, containers can provide complete isolation between runtimes and much more. Most people will not necessarily agree. Containers don’t provide complete isolation and from a theoretical standpoint they never will (unless we rewrite the Linux kernel). There have been security vulnerabilities in the past about this and there will be, it’s just the way it is. Your argument here should probably be ”it's safe for single tenant use“ i.e. all the code runs on effe is your company's code. Also coming back to the video again, I just can't fully understand what this project does, it just looks like a Go web server framework.

_jezell_ 11y ago
You could use the same logic to say virtualization isn't safe, but it doesn't stop cloud providers from putting multiple people's VMs on the same instance. Regardless of the technologies you use in your datacenter, there will be vulnerabilities that will need to be found, patched, and fixed.
siscia 11y ago
The point is to have a single binary for every single endpoint you need. Now you can deploy those binaries in whatever platform makes sense for you and scale those endpoint independently. All being open source.

hassy 11y ago

The power of Lambda comes from being able to use other AWS services. Lambdas on their own are not very interesting. A true Lambda alternative, open-source or otherwise, would also have to offer alternatives to DynamoDB, S3, SNS/SQS etc. Without those you're basically reinventing CGI.

npx 11y ago
I've been giving some thought to an AWS Lambda equivalent on Joyent Triton (largely because it's open source). You'd have RethinkDB (DynamoDB), Manta (S3), your choice of queue, TensorFlow, etc. I'm generally a huge fan of the Joyent engineering ethos (not affiliated, just a fan). Python 3.5 with DTrace probes in an AWS Lambda equivalent would be sweet.
- _jezell_ 11y ago
  Something running on Kubernetes is probably going to get more traction at this point than something running on Triton. 1.2 added a task API, probably not high performance enough yet for this, but I think it's only a matter of time till the ability to execute ad hoc tasks is a core platform expectation.
siscia 11y ago
All of those services have very valid Open Source alternative, why be locked in a closed environment ?
- nl 11y ago
  So the point of Lambda is that you don't have to run any infrastructure. If Lambda had an open source equivalent, the "other pieces" are also needed to make it useful. The "Open Sourceness" is only one factor in that environment - it is at least as important that you can somehow call them from a stateless service and get adequate performance. That tends to push towards all the services being run in the same environment (although it doesn't have to be like that - it's just easier)
- hassy 11y ago

tacos 11y ago

If you're tied to AWS then Lambda is great. If you don't want to be tied to AWS or if you need more control over resource allocation/longer running jobs then that's what all the emerging "container engine" services are for.

jorgecurio 11y ago

> In November 2014 AWS Lambda was released and it was nothing short than a revolution. still not good enough to be reliable in production, more expensive than running on micro instance or digitalocean. Lambda needs a serious makeover soon or it risks developer abandonment. It's great to lock yourself in with a cloud provider. That's the only serious benefit you get. You still need to figure out how to scale properly, getting on AWS helps but it's not a plug and play scalable thing. I've largely abandoned development on Lambda because you have to upload your code every time which is annoying. Not only that, they randomly take 5~10 seconds to respond. Also unhelpful are service error messages with no explanation or help from discussion boards. I was told to purchase their support license. So it seems like AWS is locking in developers and companies with a non user friendly documentation and arcane hidden pitfalls then leaving them no choice but to purchase support since the cost is cheap (pay to play right?) until you add up the effort of integrating with AWS and the cost of lock-in.... AWS lambda is great for prototyping but it doesn't have the tools or a proper development guideline to utilize it. There doesn't seem to be much enthusiasm from AWS either, and if they pull the plug, well hope you didn't build any mission critical apps on lambda backends because you are going to have some considerable downtime.

wsh91 11y ago
Can you share some more details about what you're trying to do with us? There's a latency floor when using the API Gateway of about 200 ms, which is unacceptable for a lot of APIs, but other than that we largely haven't been running into the issues you're dealing with. And if Meetup is any indication, AWS seems really invested in Lambda. (In terms of cost, it really would depend on what you're using it for.)
- jorgecurio 11y ago
  well I built a user authentication and it takes 5~6 seconds for a real simple operation. it happens randomly and no way to debug it. the opacity of being able to peer at what's going on in the backend is a real problem with AWS Lambda. I think that this lack of insight is going to bite down the road. Even now, I'm getting 'service error' with NO EXPLANATION what happened.

manojlds 11y ago

Is the "Bring to you by RedBeardLab" intentional or something?