10 ms·

How we built Hamiltix.net for less than $1 a month on AWS

155 points by gregorymichael 9y ago

dshuang 9y ago
For those interested, it looks like the aws stack diagram was created with cloudcraft.
isuckatcoding 9y ago
Great work! I love reading stories like this. Lambda isn’t for every usecase but I’d say a large majority of CRUD apps can use this architecture and reduce costs.
- davidgf 9y ago
  For the large majority of CRUD apps the least cost you should be concerned about is hosting, but development time. This kind of architecture adds an extra layer of complexity that is trivially solved with most of the web frameworks out there. I love Serverless, which I chose to design and build a couple of applications, but for some use cases it might not be the best solution.
  - nzoschke 9y ago
    I’m starting to find a FaaS architecture can remove complexity for the developer. The CRUD functions themselves are easy to write and test. Then everything else is handled by the infrastructure layer. Auth, security policies, rate limiting, request timeouts, autoscaling, logs, tracing, etc. With Rails the developer has to build a lot of this into the application layer. And the dev has to do more operations over time to maintain the app and database. I am working on a boilerplate app that demonstrates all the functionality you get from “serverless” and documents all of the things we no longer have to worry about: https://github.com/nzoschke/gofaas https://github.com/nzoschke/gofaas
    - davidgf 9y ago
      Rails is a great framework that prevents you from dealing with a lot of boilerplate when it comes to web applications and to focus only on the business logic. There are some great platforms out there (Heroku, for example) that deal with most of the stuff you mentioned (logs, tracing, autoscaling, etc.), and no one prevents you from using a third party service for auth in Rails as well (Auth0, for example). FaaS is awesome, but you've got to code a lot of stuff that Rails gives you for free. Besides, pushing some logic down to infrastructure locks you in the provider you chose. I'm not saying that's bad, because as I've mentioned before I run some apps fully Serverless in AWS, but there's some stuff you've got to take into consideration.
  - mseebach 9y ago
    In my (limited) experience with Lambda, there's nothing inherently complex about it. Yes, there's a learning curve, as with anything, but that's a one-off cost. Once you know how, you write the function in a very similar environment you would in any non-serverless setup, and you call it like you would any API endpoint, serverless or not. Once you have made this investment, you get scale (almost) for free (at least as concerns development/sys admin time). No need to worry about your $5 DO conking out in the middle of the night because you got Slashdotted or whatever the kids call it these days.
    - davidgf 9y ago
      Yes, once you get used to the development environment, coding Serverless is pretty smooth. However, when you need something a bit more convoluted than executing a single function, but rather orchestrating a bunch of them across different services, you have to think about stuff like service discovery. Making the right decisions and designing an event driven architecture is going to take time, which is worth money that could pay a lot of servers. With Serverless you get scalability almost for free, but there are trade-offs, so the first you should ask yourself if you're really going to need that scalability or would rather getting to market quicker. In some cases it makes more sense going for Serverless straight away, in some others just using the web framework you're experienced with and in some others a combination of both.
nzoschke 9y ago
Very nice. I built a boilerplate Lambda app that puts a lot of this cost effective architecture into practice: https://github.com/nzoschke/gofaas https://github.com/nzoschke/gofaas One difference is that I use SNS for cheap email or SMS error notifications.
- badsectorlabs 9y ago
  This is awesome and will save people tons of time setting up something similar!
sscarduzio 9y ago
Wait but where do you host gitlab? You need an ec2 instance for that, right?
- dorfsmay 9y ago
  Not OP, but I suspect on gitlab.com itself (free private repo for small teams).
  - jamietanna 9y ago
    Not just small teams - unlimited contributors
- badsectorlabs 9y ago
  gitlab.com is free for private repos!
everdev 9y ago
> After the free-tier expires it will be less than $5, but the fact remains: it's crazy cheap. I like the idea behind serverless development, but DO is $5/month and is easy to setup and maintain. From what I understand as your business grows AWS costs skyrocket and just starting out it's more complicated than a DO solution. The idea is great, but looking forward to the day where serverless is the default solution because it's easier and cheaper. Not sure we're there yet.
- sudhirj 9y ago
  The costs skyrocket if you’re not doing it right - otherwise it tends to be linear with request count. The nice thing, though, is that you don’t have to panic or do anything at any scale - it’ll just keep running along. Once you hit high levels of traffic, instead of panicking and trying to add more servers and fixing DB structure, you do a cost benefit analysis and see what to do next. Either work on reducing your cost slope, or make it logarithmic by moving to regular servers.
  - vidarh 9y ago
    The problem is the per request cost is going to be astronomical compared to what it needs to be once you e.g. get above the initial 1 GB/month tier. Bandwidth at AWS costs about 50 times as much as with Hetzner for example. > Once you hit high levels of traffic, instead of panicking and trying to add more servers and fixing DB structure, you do a cost benefit analysis and see what to do next. Either work on reducing your cost slope, or make it logarithmic by moving to regular servers. ...and that's a good reason to try to avoid the lock-in of AWS specific services. Relatively entry level setups on alternative platforms will let you scale well past the system where AWS costs starts to get painful and people start panicking about how to cut costs. When you hit that point you really don't want to be facing a major re-engineering effort to be able to even use other options to bring the cost down just as the costs are soaring.
    - phoboslab 9y ago
      > Bandwidth at AWS costs about 50 times as much as with Hetzner for example. It's actually closer to 70x Hetzner: 1.25 USD per TB (after 30 TB free per server) [1] AWS: 90 USD per TB [2] So, 100 TB at Hetzner will cost you 125 USD (not counting the 30 TB free). You would pay 9000 USD at AWS. [1] https://wiki.hetzner.de/index.php/Traffic/en https://wiki.hetzner.de/index.php/Traffic/en [2] https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/
onion2k 9y ago
Using Codenvy for development, Github for code, Netlify for deployment and hosting, and Firebase for a database, in total I spend exactly nothing on my side projects.
- everdev 9y ago
  All great services except Firebase. It's hard to version control development and it's a lock in platform that's hard to migrate away from.
  - marknadal 9y ago
    Try https://github.com/amark/gun https://github.com/amark/gun , it is a popular (~7.5K stars) truly open source (MIT/ZLIB/Apache 2) database that has Firebase like features (realtime sync) plus a lot more, like graph/table/document/relational data and a P2P/decentralized architecture, by yours truly. :)
    - onion2k 9y ago
      I've used gun. It's really cool.
      - marknadal 9y ago
        Thanks! Firebase still has a ton of features we don't have, what would you say are the most important things for us to work on?
        everdev 9y ago
        I love the plug and play nature of Firebase for simple CRUD apps, so the simplicity of Firebase user auth was a huge win. Is that on the roadmap?
        marknadal 9y ago
        Yes, we just announced/released the alpha of our auth framework a while ago, called SEA (Security, Encryption, Authorization). Because it is based on P2P/decentralized cryptography, it is actually pretty hard and cutting-edge to implement (you get end-to-end encryption automatically with it!), but we've made it ridiculously easy for developers to use: - Easy getting started guide: https://hackernoon.com/so-you-want-to-build-a-p2p-twitter-with-e2e-encryption-f90505b2ff8 https://hackernoon.com/so-you-want-to-build-a-p2p-twitter-wi... - 4 minute nad 40LOC interactive coding tutorial: https://scrimba.com/c/c2gBgt4 https://scrimba.com/c/c2gBgt4 It is alpha though, still has several more features that need to be added to make it usable for different types of app (so you might want to stick with Firebase till then). Please let me know if you have any questions or anything else I can help with!
tuananh 9y ago
how is aws lambda pricing at scale?
- sudhirj 9y ago
  Same as pricing at non scale. It’s per 100ms of execution time. Could do a large scale discount if you’re scaley enough.
outworlder 9y ago
"Getting a push alert any time there is an error helps us respond to issues as soon as they come up. " Oh god no. Maybe if this system is not expected to ever grow in complexity. Otherwise this will become a big source of frustration really soon. And even if it does not, it's very easy to get a flood of non-actionable alerts for a transient condition. It would be better to collect error rates and alert on that. Also the other golden signals.
- alex_duf 9y ago
  yeah but to each stage of a project its solution. If that works at that scale why not?
  - lugg 9y ago
    You know for people who live in an ever changing environment and are constantly editing our own reality - so much so that we need to track it in "source control" - developers are kind of funny how they get stuck on doing things "the right way."
    - BoorishBears 9y ago
      Do they really get stuck on doing things "the right way", or do they get stuck on making off-the-cuff remarks about doing things "the right way" for easy karma?
- zimbatm 9y ago
  Or better yet, forward the errors to https://sentry.io/ https://sentry.io/ and let them deal with error grouping.
- badsectorlabs 9y ago
  A valid concern. However even after reddit and hacker news has had their way with the site, my phone has been impressively quiet. These push notifications are only for cases where the lambda functions thrown an error that isn't handled anywhere else (i.e. something is very wrong).
pbreit 9y ago
Would be curious to understand how this might work on a plain ole virtual server (ie, Vultr $2.50).
nodomain 9y ago
I can modify the price in the checkout by changing the URL parameter. Really?
- veb 9y ago
  Can also use a negative price! I wonder what would happen if you went ahead with that. Presumably the payment gateway wouldn't pay me $5. Seems a strange choice to be having those parameters in a GET.
  - throwaway1909 9y ago
    Why stop at $5? Try setting quantity or price to -9e999999999. How will you spend your $∞ dollars?
    - mlvljr 9y ago
      On the most stellar lawyer there is?
  - paulie_a 9y ago
    > Presumably the payment gateway wouldn't pay me $5 Having dealt with payment gateways, I would not assume that.
  - badsectorlabs 9y ago
    I agree, the param passing is not ideal. Session storage/local storage is another option.
    - stdclass 9y ago
      Everything that can be modified by the user is not suitable. The price must be stored on the server side, everything on the client side is subject to modification.
      - badsectorlabs 9y ago
Kagerjay 9y ago
Not familiar with amazon Lambda, is the analogy developer toolset : amazon lambda business toolset: zapier personal toolset: ifttt correct here?
Jyaif 9y ago
Or use GAE to do the same for ∞ less.
alokdhari 9y ago
What year did you make it in??
yorby 9y ago
You can get below $1/month if you use a cheap VPS [1] ... 1. http://www.lowendstock.com/ http://www.lowendstock.com/
deckarep 9y ago
Impressive but my side projects are even cheaper to run because 1/3 of the way through I lose excitement, never finish them and then don’t do anything with them. ;) Nice post!
- segmondy 9y ago
  Or do it like it was 1999. Host it yourself. I had a $20 DO, I've downgraded it to the $5/month plan. My next goal is to move that shut down the VM sometimes this year and host it on a raspberry pi myself.
gowld 9y ago
I don't get it. Is hamiltix scalping tickets? I thought the ticket sellers control re-distribution. How can it accept payment for tickets someone else is holding?
pards 9y ago
> We also made the decision to not use a javascript framework for the front end, mostly because they are incredebly [sic] complex and some people suggest they are all terrible (or maybe great?). Not to mention that for back-end developers, the learning curve is steep and ever-changing especially when one considers the "build" process. Is it grunt, or gulp these days, or webpack, or NPM, or some Frankenstein combination of all of them?

For those interested, it looks like the aws stack diagram was created with cloudcraft.

Great work! I love reading stories like this. Lambda isn’t for every usecase but I’d say a large majority of CRUD apps can use this architecture and reduce costs.

davidgf 9y ago
For the large majority of CRUD apps the least cost you should be concerned about is hosting, but development time. This kind of architecture adds an extra layer of complexity that is trivially solved with most of the web frameworks out there. I love Serverless, which I chose to design and build a couple of applications, but for some use cases it might not be the best solution.
- nzoschke 9y ago
  I’m starting to find a FaaS architecture can remove complexity for the developer. The CRUD functions themselves are easy to write and test. Then everything else is handled by the infrastructure layer. Auth, security policies, rate limiting, request timeouts, autoscaling, logs, tracing, etc. With Rails the developer has to build a lot of this into the application layer. And the dev has to do more operations over time to maintain the app and database. I am working on a boilerplate app that demonstrates all the functionality you get from “serverless” and documents all of the things we no longer have to worry about: https://github.com/nzoschke/gofaas https://github.com/nzoschke/gofaas
  - davidgf 9y ago
    Rails is a great framework that prevents you from dealing with a lot of boilerplate when it comes to web applications and to focus only on the business logic. There are some great platforms out there (Heroku, for example) that deal with most of the stuff you mentioned (logs, tracing, autoscaling, etc.), and no one prevents you from using a third party service for auth in Rails as well (Auth0, for example). FaaS is awesome, but you've got to code a lot of stuff that Rails gives you for free. Besides, pushing some logic down to infrastructure locks you in the provider you chose. I'm not saying that's bad, because as I've mentioned before I run some apps fully Serverless in AWS, but there's some stuff you've got to take into consideration.
- mseebach 9y ago
  In my (limited) experience with Lambda, there's nothing inherently complex about it. Yes, there's a learning curve, as with anything, but that's a one-off cost. Once you know how, you write the function in a very similar environment you would in any non-serverless setup, and you call it like you would any API endpoint, serverless or not. Once you have made this investment, you get scale (almost) for free (at least as concerns development/sys admin time). No need to worry about your $5 DO conking out in the middle of the night because you got Slashdotted or whatever the kids call it these days.
  - davidgf 9y ago
    Yes, once you get used to the development environment, coding Serverless is pretty smooth. However, when you need something a bit more convoluted than executing a single function, but rather orchestrating a bunch of them across different services, you have to think about stuff like service discovery. Making the right decisions and designing an event driven architecture is going to take time, which is worth money that could pay a lot of servers. With Serverless you get scalability almost for free, but there are trade-offs, so the first you should ask yourself if you're really going to need that scalability or would rather getting to market quicker. In some cases it makes more sense going for Serverless straight away, in some others just using the web framework you're experienced with and in some others a combination of both.

nzoschke 9y ago

Very nice. I built a boilerplate Lambda app that puts a lot of this cost effective architecture into practice: https://github.com/nzoschke/gofaas https://github.com/nzoschke/gofaas One difference is that I use SNS for cheap email or SMS error notifications.

badsectorlabs 9y ago
This is awesome and will save people tons of time setting up something similar!

sscarduzio 9y ago

Wait but where do you host gitlab? You need an ec2 instance for that, right?

dorfsmay 9y ago
Not OP, but I suspect on gitlab.com itself (free private repo for small teams).
- jamietanna 9y ago
  Not just small teams - unlimited contributors
badsectorlabs 9y ago
gitlab.com is free for private repos!

everdev 9y ago

> After the free-tier expires it will be less than $5, but the fact remains: it's crazy cheap. I like the idea behind serverless development, but DO is $5/month and is easy to setup and maintain. From what I understand as your business grows AWS costs skyrocket and just starting out it's more complicated than a DO solution. The idea is great, but looking forward to the day where serverless is the default solution because it's easier and cheaper. Not sure we're there yet.

sudhirj 9y ago
The costs skyrocket if you’re not doing it right - otherwise it tends to be linear with request count. The nice thing, though, is that you don’t have to panic or do anything at any scale - it’ll just keep running along. Once you hit high levels of traffic, instead of panicking and trying to add more servers and fixing DB structure, you do a cost benefit analysis and see what to do next. Either work on reducing your cost slope, or make it logarithmic by moving to regular servers.
- vidarh 9y ago
  The problem is the per request cost is going to be astronomical compared to what it needs to be once you e.g. get above the initial 1 GB/month tier. Bandwidth at AWS costs about 50 times as much as with Hetzner for example. > Once you hit high levels of traffic, instead of panicking and trying to add more servers and fixing DB structure, you do a cost benefit analysis and see what to do next. Either work on reducing your cost slope, or make it logarithmic by moving to regular servers. ...and that's a good reason to try to avoid the lock-in of AWS specific services. Relatively entry level setups on alternative platforms will let you scale well past the system where AWS costs starts to get painful and people start panicking about how to cut costs. When you hit that point you really don't want to be facing a major re-engineering effort to be able to even use other options to bring the cost down just as the costs are soaring.
  - phoboslab 9y ago
    > Bandwidth at AWS costs about 50 times as much as with Hetzner for example. It's actually closer to 70x Hetzner: 1.25 USD per TB (after 30 TB free per server) [1] AWS: 90 USD per TB [2] So, 100 TB at Hetzner will cost you 125 USD (not counting the 30 TB free). You would pay 9000 USD at AWS. [1] https://wiki.hetzner.de/index.php/Traffic/en https://wiki.hetzner.de/index.php/Traffic/en [2] https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/

onion2k 9y ago

Using Codenvy for development, Github for code, Netlify for deployment and hosting, and Firebase for a database, in total I spend exactly nothing on my side projects.

everdev 9y ago
All great services except Firebase. It's hard to version control development and it's a lock in platform that's hard to migrate away from.
- marknadal 9y ago
  Try https://github.com/amark/gun https://github.com/amark/gun , it is a popular (~7.5K stars) truly open source (MIT/ZLIB/Apache 2) database that has Firebase like features (realtime sync) plus a lot more, like graph/table/document/relational data and a P2P/decentralized architecture, by yours truly. :)
  - onion2k 9y ago
    I've used gun. It's really cool.
    - marknadal 9y ago
      Thanks! Firebase still has a ton of features we don't have, what would you say are the most important things for us to work on?
      - everdev 9y ago
        I love the plug and play nature of Firebase for simple CRUD apps, so the simplicity of Firebase user auth was a huge win. Is that on the roadmap?
        marknadal 9y ago
        Yes, we just announced/released the alpha of our auth framework a while ago, called SEA (Security, Encryption, Authorization). Because it is based on P2P/decentralized cryptography, it is actually pretty hard and cutting-edge to implement (you get end-to-end encryption automatically with it!), but we've made it ridiculously easy for developers to use: - Easy getting started guide: https://hackernoon.com/so-you-want-to-build-a-p2p-twitter-with-e2e-encryption-f90505b2ff8 https://hackernoon.com/so-you-want-to-build-a-p2p-twitter-wi... - 4 minute nad 40LOC interactive coding tutorial: https://scrimba.com/c/c2gBgt4 https://scrimba.com/c/c2gBgt4 It is alpha though, still has several more features that need to be added to make it usable for different types of app (so you might want to stick with Firebase till then). Please let me know if you have any questions or anything else I can help with!

tuananh 9y ago

how is aws lambda pricing at scale?

sudhirj 9y ago
Same as pricing at non scale. It’s per 100ms of execution time. Could do a large scale discount if you’re scaley enough.

outworlder 9y ago

"Getting a push alert any time there is an error helps us respond to issues as soon as they come up. " Oh god no. Maybe if this system is not expected to ever grow in complexity. Otherwise this will become a big source of frustration really soon. And even if it does not, it's very easy to get a flood of non-actionable alerts for a transient condition. It would be better to collect error rates and alert on that. Also the other golden signals.

alex_duf 9y ago
yeah but to each stage of a project its solution. If that works at that scale why not?
- lugg 9y ago
  You know for people who live in an ever changing environment and are constantly editing our own reality - so much so that we need to track it in "source control" - developers are kind of funny how they get stuck on doing things "the right way."
  - BoorishBears 9y ago
    Do they really get stuck on doing things "the right way", or do they get stuck on making off-the-cuff remarks about doing things "the right way" for easy karma?
zimbatm 9y ago
Or better yet, forward the errors to https://sentry.io/ https://sentry.io/ and let them deal with error grouping.
badsectorlabs 9y ago
A valid concern. However even after reddit and hacker news has had their way with the site, my phone has been impressively quiet. These push notifications are only for cases where the lambda functions thrown an error that isn't handled anywhere else (i.e. something is very wrong).

pbreit 9y ago

Would be curious to understand how this might work on a plain ole virtual server (ie, Vultr $2.50).

nodomain 9y ago

I can modify the price in the checkout by changing the URL parameter. Really?

veb 9y ago
Can also use a negative price! I wonder what would happen if you went ahead with that. Presumably the payment gateway wouldn't pay me $5. Seems a strange choice to be having those parameters in a GET.
- throwaway1909 9y ago
  Why stop at $5? Try setting quantity or price to -9e999999999. How will you spend your $∞ dollars?
  - mlvljr 9y ago
    On the most stellar lawyer there is?
- paulie_a 9y ago
  > Presumably the payment gateway wouldn't pay me $5 Having dealt with payment gateways, I would not assume that.
- badsectorlabs 9y ago
  I agree, the param passing is not ideal. Session storage/local storage is another option.
  - stdclass 9y ago
    Everything that can be modified by the user is not suitable. The price must be stored on the server side, everything on the client side is subject to modification.
    - badsectorlabs 9y ago

Kagerjay 9y ago

Not familiar with amazon Lambda, is the analogy developer toolset : amazon lambda business toolset: zapier personal toolset: ifttt correct here?

Jyaif 9y ago

Or use GAE to do the same for ∞ less.

alokdhari 9y ago

What year did you make it in??

yorby 9y ago

You can get below $1/month if you use a cheap VPS [1] ... 1. http://www.lowendstock.com/ http://www.lowendstock.com/

deckarep 9y ago

Impressive but my side projects are even cheaper to run because 1/3 of the way through I lose excitement, never finish them and then don’t do anything with them. ;) Nice post!

segmondy 9y ago
Or do it like it was 1999. Host it yourself. I had a $20 DO, I've downgraded it to the $5/month plan. My next goal is to move that shut down the VM sometimes this year and host it on a raspberry pi myself.

gowld 9y ago

I don't get it. Is hamiltix scalping tickets? I thought the ticket sellers control re-distribution. How can it accept payment for tickets someone else is holding?

pards 9y ago

> We also made the decision to not use a javascript framework for the front end, mostly because they are incredebly [sic] complex and some people suggest they are all terrible (or maybe great?). Not to mention that for back-end developers, the learning curve is steep and ever-changing especially when one considers the "build" process. Is it grunt, or gulp these days, or webpack, or NPM, or some Frankenstein combination of all of them?