15 ms·

How we got our AWS bill to around 2% of revenue

298 points by grwthckrmstr 6y ago

fxtentacle 6y ago
TLDR: Use Lightsail, Amazon's approximation of a dedicated server. In my opinion, that approach sacrifices pretty much all the benefits that cloud proponents usually talk about, like only paying for what you actually use or scaling up and down on demand. In fact, I'm not sure what the actual difference would be between Lightsail and a good dedicated hoster that provides backup and failover services.
- jakozaur 6y ago
  I disagree. Start with LightSail customize later. Too many startups starts with Kubernetes, microservices putting a lot of engineering into infrastruture when they should worry about product market fit and simple server would just do fine.
  - fxtentacle 6y ago
    You could just as well start with one cheap dedicated server and then migrate to the cloud once you know that you need it. For example, one of my companies is doing hosting and image delivery services for photographers. I started it on one dedicated server, and now 10+ years later it's running on 3 dedicated servers. My hoster does backups and OS upgrades, so there was never any reason to go into the cloud. Except maybe, if I wanted to burn cash on slow performance ;)
  - kristopolous 6y ago
    Complex solutions to simple problems pad resumes. It's important. I've been looking for a job recently and my lack of experience with excitedly building expensive overly engineered big piles of stuff to do simple things is really hurting me. When I suggest simple solutions proportional to the problem size people look at me like I'm some kind of simpleminded ninny. And then when I describe working systems I've made using the approach on companies that exited north of 10 million, then they usually just think I'm a bullshitter. 10,000 users? Just do a simple lamp stack on a $10 month vps and don't code it like a moron, it's not hard. And it can be delegated to almost anyone, that's important for longevity. That's it, move on with life. I've even offered to show logs, I mean, it's remarkably bad how hard it is to find work as a cost-cutting product-focused engineer, nobody wants that shit, really. They want autoscale, prometheus, travis, blah blah blah... as fancy and crazy as possible, an empire of infrastructure instead of infrastructure for an empire... It's a damn cult. I'm thinking about giving up and giving in. Maybe spend a month and drop $10,000 or whatever on a bunch of the tech just so I can talk about these things. --- The second thing I'm constantly surprised by is how many obvious things are formalized with fancy names, math equations, papers, etc. Stuff I'd think like literally anyone with half a brain would do. Recently I found out about "CRDT". Apparently I've been doing it for a decade... It's just super simple stuff wrapped in needless formalism, like back when design "patterns" were all the rage. I used to think "why is there a name for that?" it's such basic stuff. So saying "dude, the emperor has no clothes, here, take my coat" - bad interview strategy and it's hard to rectify because there's like 50 naked emperors and silicon valley culture is on a constant rotating basis of seeing the nakedness of 5 of them and ignoring the other 45. Maybe what I should do is just a couple throwaway interviews to focus on finding out the current fads and then be able to give the right cultural signals for the ones I actually want.
    - mulhoon 6y ago
      Great response. We’re all obsessed with infrastructure and over-engineering, myself included. Startups like to think they’re working on the next big thing, so they want to use the tech that the big boys use and bask in their anticipation of scale. You can easily save cost and time to validate your business with smaller hosting approaches, though it does look amateur from the outside. It’s not amateur, it’s smart.
      - kristopolous 6y ago
        Yeah I've been in LA for 12 years which is much more practical for the most part. And it's been fine, profitable, revenue generating, sure. Mysql, php, a bit of python, easy stuff. Honestly I've been wanting to move and try out silicon valley again though and it's been hard because I don't speak that language and frankly I'm skeptical and hostile to it - bad idea. That bad attitude needs to go. I've been in practical startup land too long so I keep getting stonewalled from getting in to the sv world. I'm trying to contribute to some companies open source projects recently and then basically earn my way in through labor since I can't do the virtue signaling. It's only been a few weeks, nothing yet. I gotta learn how to be along for the ride without always trying to reach for the steering wheel. Hopefully by the end of summer I'll find a lead somewhere. It's not about the money, it's kinda actually how I want to spend my time for the next year or two.
- antonvs 6y ago
  FTA: > We are a team of two people Articles like this reflect the fact that one size does not fit all. > a good dedicated hoster that provides backup and failover services. In theory that sounds reasonable. In practice the ability to completely manage the system via a web console and APIs makes a big difference.
  - fxtentacle 6y ago
    I have never needed it, but the hosting provider which I use actually has an API that you can use to reboot servers into different OS images, or trigger hardware resets, or do an kvm access. https://wiki.hetzner.de/index.php/Robot_Webservice/en https://wiki.hetzner.de/index.php/Robot_Webservice/en But honestly, if you have only one server, what do you need an API for? It'll be much faster to just configure things by hand. It's not like you'll switch servers very often. Or at least the original article says that they committed to monthly cancelation terms.
  - dx034 6y ago
    Even Hetzner as the cheapest VPS and dedicated server provider offers an API to completely manage servers.
    - isatty 6y ago
      You can definitely do cheaper than hetzner (like OVH).
- redis_mlc 6y ago
  > all the benefits that cloud proponents usually talk about, like only paying for what you actually use or scaling up and down on demand. That's a trope that I have never seen in practise. A company might have some small percentage of clusters that autoscale, but usually there's a bunch of servers that don't. Also, when AWS has control plane problems, autoscaling doesn't work, you get swamped, your health checks fail, and you lose all your servers and you're hosed. So it's better to always over-provision in the first place.
  - fxtentacle 6y ago
    Same here. Heroku offers auto-scaling for the web dynos, but they advise you to do near-realtime tasks like the thumbnails for image uploads using background workers, for which there is no Auto-scaling.
- alain_gilbert 6y ago
  The difference would be: - They could have much better service - And bring down their expenses by at least half again I used lightsail, and it's a terrible service. It is very expensive (still) compared to other hosting providers. And if your instance ever go out of memory, it become unresponsive for as long as you don't go manually restart it. On other providers that I use, the OS would just "sacrifice child" (kill the process) and restart it. It's not ideal, but much better than having to go there yourself to restart the whole thing.
  - oefrha 6y ago
    Sounds like OOM killer is disabled by default on Lightsail. Shouldn't be hard to turn it back on by modifying a line or two in sysctl.conf.
  - fxtentacle 6y ago
    I've been using cgroups since way before docker made them cool exactly for this: if one process goes rogue on memory use, make sure only exactly that one crashes. And yes, I fully agree with you. In addition to the other disadvantages, Lightsail also shifts the burden of process monitoring and management onto you.
    - alain_gilbert 6y ago
      I use docker. And yes, it normally just take care of restarting the app whenever something goes wrong. But somehow on lightsail (only), the machine just goes completely unresponsive instead.
- Proven 6y ago
  I think so too. I just had a quick look at the Lightsail pages and thought "This is just plain old hosting, with hooks into a highly proprietary cloud environment."
whack 6y ago
I appreciate the disclaimers at the end: > I would like to put emphasis on the fact that we are a micro-SaaS product that solves a small and specific use case and therefore this kind of AWS setup worked for us. This may not work for big organisations or products where the traffic is erratic. > This setup will also not work for folks who have a ton of stuff to do already and would prefer to use managed services and not take the additional headache of monitoring, maintaining and provisioning hardware resources on a regular basis because this has a time cost to it. It is refreshing to see a post that doesn't pretend as though its recommendations are the holy grail for anyone and everyone. I personally prefer saving time and effort, over a ~$500 monetary discount. But it's nice to read posts like these and learn more about alternatives such as Lightsail.
- tehnicaorg 6y ago
  > I personally prefer saving time and effort I guess it depends where you live and how much your time costs (and if you have any time to spare).
  - ashwinm 6y ago
    Yup, This is some high quality jugaad. ;)
- ldoughty 6y ago
  > I personally prefer saving time and effort, over a ~$500 monetary discount. Completely agree. As the general "Ops" manager, I hate getting the call at 2am because something broke... I hate it even more when it's not really fixable by me in the first place (e.g. black box appliance or software).. but I feel a heck of a lot better when I can point the finger at an external team that's paid and supposed to be experts... If you outsourced Redis or you DB, and someone like AWS or MongoDB themselves has an issue... Will they pay people a lot of money to simply provide that service reliably... I get paid to keep 20 things connected, not be the subject matter expert in all of them.
  - amelius 6y ago
KorematsuFred 6y ago
Tldr; Author is talking about few hundred dollars here.
- sushshshsh 6y ago
  At that rate a raspi on starbucks wifi would get them to 0.2%
ChicagoDave 6y ago
I’m wondering if Lambdas and DynamoDB would be competitive to this setup.
- aritraghosh007 6y ago
  Since they are serving about 250 requests per second, Lambdas might end up being more expensive, factor in an expensive DynamoDB and that monthly AWS bill looks scary.
  - root993 6y ago
    Hello, author here. I just want to confirm that this is one of the reasons why we did not go with Lambda and the other reason being I was not fully aware of and versed with serverless when we started this venture
  - ChicagoDave 6y ago
    I’m thinking otherwise. If the payloads are small, DynamoDB is going to be cheaper and nothing will come close to the savings from Lambda compute.
- PaulKeeble 6y ago
  Almost certainly. A lot of groups have shown low volume systems have been remarkably cheap using lambda's compared to trying to do the same thing in a VM with EC2/lightsail. Whether that is genuinely because it is that much cheaper for Amazon or whether it is trying to lock people into the platform I don't know.
- redis_mlc 6y ago
  > I’m wondering if Lambdas and DynamoDB would be competitive to this setup. Depends, but it's important to know the details. Lambdas are used for 2 things in AWS: 1) Glue for managing services and events. (Required for advanced AWS mgmt.) 2) User applications. Lambdas have improved over time, but initially had all kinds of limitations (startup time, database access, 15 minute job limits, etc.) DynamoDB has also improved a lot. Initially hot keys could cost you a lot of money. Some companies have designed their applications to use it in a way that absolutely no administration is required, reducing DBAs and Ops people. Source: DBA.
PaulKeeble 6y ago
If you end up in the lightsail world and you are not utilising Amazon's other services then its probably cheaper to do this with another provider. Someone like Contabo or Hetzner will get you VMs at substantially less cost with similar fixed hardware and just give you the box but for a lot less a month. At these low scales with a completely open source stack Amazon isn't good value in my opinion. It is as you grow and it provides the scale that it becomes valuable.
- robertlagrant 6y ago
  This is interesting. Do you have (or know of) a published comparison somewhere?
  - EGreg 6y ago
    Would like to see this too
  - deleted 6y ago
    [deleted]
  - dx034 6y ago
    I don't have a comparison but Hetzner is pretty much unbeatable in price if you can live with having your servers in Europe. If not, OVH, DigitalOcean, Linote or Vultr are good alternatives with datacenters around the globe and prices below that of AWS.
    - mxschumacher 6y ago
      I know that hetzner is affordable, but is digital ocean really cheaper than AWS?
      - uaas 6y ago
        That would largely depend on your usecase.
    - callmeal 6y ago
mattbillenstein 6y ago
Their setup is trivial - they could do it at 0.2% of revenue with a cheap vps on Linode or other...
- nordsieck 6y ago
  > Their setup is trivial - they could do it at 0.2% of revenue with a cheap vps on Linode or other... Lightsail has pretty comparable pricing to Linode. I'm sure they could re-architect their app to use fewer instances, but they could do that and stay on Lightsail as well. Moving to linode isn't going to give them a 90% savings.
  - wolco 6y ago
    The price seems on par. Are there hidden costs on aws?
    - deleted 6y ago
      [deleted]
    - dijit 6y ago
      I'm assuming that you might not be aware of AWS pricing. It's actually incredibly complicated and often very difficult (if not impossible) to predict. We have a dedicated part of our organisation which exists solely to figure out costs ahead of time for project. I'm not sure if it's intentionally obfuscated, I would suspect not, because "pay for what you use" can be broken down into many areas.
      - scarface74 6y ago
        What is incredibly complicated about Lightsail? That’s kind of the point of it.
      - radicalbyte 6y ago
        Azure is the same and Microsoft spent 20 years making their software licencing more and more unpredictable. It absolutely is a strategic choice and not an accident.
samstave 6y ago
I have a question: (it may not be related to this article in any other sense aside from "really lightweight" use of AWS... I would like to have a workflow of really lightweight QR/barcode transactions... but with what seems to me currently a complex implementation: 1. you have a prescribed barcode ParentA 2. you scan that barcode and create a QR code ParentB 2.a: you ascribe that QR to an object ChildA 3.n: you have that ChildA go through many iterations 3.n+: ChildA may be conjoined with ChildB,C,D etc... -- but I want an iterative QR code that follows the family tree back to parent A and keeps history of all transitions... (I know I am wording this poorly... just thinking out loud...)
- unixhero 6y ago
  So what about the grandparents, and how would ChildN go to kindergarden?
  - samstave 6y ago
    What a fucking stupid response. The workflow that I am talking about would apply to taking the nightmare out of METRC tracking system for cannabis -- the METRC system is a piece of shit - but wants complete chain of history of how plantA becomes productA - and it is a bullshit workflow - and I have a way to make it easier - and by using a system similar in the way described in this post, it could be done very cheaply...
    - unixhero 6y ago
      For what is this workflow? Just curious. If it's barcodes being used by humans in the real world, get a software package to handle it, check the solution from Zebra https://www.zebra.com/us/en/products.html https://www.zebra.com/us/en/products.html
barrenko 6y ago
Slightly off-topic, but is there a book to learn about this kind of stuff, preferrably without having to sign up for AWS, but that does have code samples and whatall?
- ramraj07 6y ago
  There's only so much you can learn about this without signing up and actually getting your hands dirty unfortunately.
- root993 6y ago
  There is one book that I would highly recommend - https://gumroad.com/l/aws-good-parts https://gumroad.com/l/aws-good-parts
ramraj07 6y ago
I dabble extensively in being dirt cheap with my monthly cloud spend on personal projects, and after much experimentation, I have settled on the following: 1. elastic beanstalk, no docker: EB comes with really nice defaults so that you can quickly whip up a flask app, upload it and it just works. It provisions a small ec2 instance by default which iirc costs 10ish a month at best. Importantly, any operation you do with EB will by definition be ready for continuous deployment since you don't get an option to ssh into the machine to deploy. It's extensible enough to add whatever extra stuff you need as well. Only thing it can't do is simple caching ( if you scale to more than one instance that is), but that can be solved by having a separate eb deployment for a worker that can take care of all these aux stuff (elasticache is expensive I think). In a pinch, it also scales well (though the default cheap deployment does not have a load balancer). 2. Just suck it up and go with RDS postgres. Again, I see too many things going wrong with spinning up your own db in an ec2 instance especially if that instance goes down. Im too lazy to write backup scripts and keep track of them! The cheapest RDS postgres costs 13 a month or so, but I just suck it up to power whatever side projects I do. Postgres means I'm working with something I know, and I get full text search and pubsub for free. And whatever I write is not locked code in anyform, and can be scaled up if needed as well. More importantly I'm only spending my time in technologies that are relevant for me in my day job so that's a win. 3. Github Actions to deploy to eb. just a few lines of yaml and you instantly get continuous deployment directly from your repo for free! Really can't beat that. I have meant to try out heroku since it could be cheaper from what I have read. But I couldn't figure out what their S3 alternative is or how different they are from the canonical cloud offerings. I'm sure it can be so much more cheaper, but I'm not good at advanced networking or sysadmin, and I'm too lazy / bored / disorganised to write deploy scripts or sshing into remote machines. I'm also always afraid of if/how long I need to re-provision a vps/ec2 if it goes down. Not that they do, but they can, and that scares me.
- root993 6y ago
  Hello, author here. 1. When I started working on this I was not fully aware of the large array of services that AWS provides and therefore our setup may not be the best possible one. So I would be checking out Elastic beanstalk for sure and see if it is feasible to use for my next venture. 2. Just to clarify, we are not running mysql on a lightsail instance, rather we are using a managed database provided by lightsail which automatically takes backups on its own and also has the option to restore a new DB from an existing backup. 3. Thanks for the insight on EB. Will be looking into it for sure. The thought of any of these instances going down scares me as well. but I would like to believe that I have set up enough alerts everywhere so I can take immediate action :)
omnifischer 6y ago
Text from about me: > where I learnt how to scale an application to serve ads to millions of users a day Please lets stop this 'race'.
lend000 6y ago
A bit off topic, but I bet someone here knows. When running an EC2 instance and not using all of the cores on the socket (for example, using a c5.large instance instead of a c5.12x large, which gives you all the cores on the socket), you presumably are sharing your L3 cache with your neighbors on the same socket, because that's how the processor is designed. Is there a way that the hypervisor allocates a dedicated portion of the shared L3 cache to just your instance, or is it a free for all for all of the L3 cache space against potentially noisy neighbors?
- Keyframe 6y ago
  Sounds like a textbook example of where and how a side channel attack would look like.
- nisten 6y ago
  Yeah that is true, you are sharing L3 cache. In order to mitigate some of recent intel issues I think AWS actuallly has their own chip now on newer motherboards to handle the hypervisor duties securely. Otherwise, they'd do it in software patches for older CPUs and take the performance hit of the patch. I'm not sure how much the hypervisor would reserve off of the L3, it is likely to be free for all however you'd still have quite a bit of dedicated L2 and L1 on most xenons. With AMD's first gen EPYC it's a little bit different because clusters of cores share a cache and you can get weirdly high latencies depending on which cores you're using, (i.e. cores 8 and 9 being too far apart) Also according to this anandtech article, the average total CPU load for physical aws machines is ~60% and is actively balanced out by them. And yes, running benchmarks on a machine without noisy neighbors yields very significant improvements, up to 2x better on the benchmark scores. They measured this by comparing renting out all the cores of a machine vs only renting out the 4 or however they needed . https://www.anandtech.com/print/15578/cloud-clash-amazon-graviton2-arm-against-intel-and-amd https://www.anandtech.com/print/15578/cloud-clash-amazon-gra... I'm assuming they'd put a CPU into sleep/hibernate mode in order to save power instead of having it only run at 5% utilization.
  - lend000 6y ago
    Without any dedicated hypervisor tricks, can't a typical L3 cache eviction algorithm also evict memory that is assigned to another core and currently residing in its L2 or L1 cache? (Thereby flushing even the higher level caches if another core is really noisy.)
raverbashing 6y ago
So Lightsail is the AWS equivalent of DO/Linode/etc?
- sumedh 6y ago
  Yes, AWS probably realized that they losing money to DO/Linode so to capture some of that market they launched Lightsail.
rumanator 6y ago
> An EC2 instance with 2 virtual cores, 4GB RAM and a storage of 80GB costs roughly 37$ a month and a Lightsail instance with the exact same configuration costs 20$ a month which is almost half the cost! The author completely failed to do his homework. An a1.large instance is only 37$ if it's an on-demand instance. You pay a high premium to be able to pull the switch in the exact minute you cease to need one. If he's willing to go with AWS lightsail with it's monthly plan, the same a1.medium instance type is about 11.75$/month as a reserved instance, and can be had for about 131$/year as well.
- kijin 6y ago
  The Lightsail instance comes with 4TB/mo of free transfer. Even using a small fraction of that transfer on EC2 will cost you more than the instance itself. Also, creating an instance on Lightsail isn't exactly a monthly commitment. You are free to delete your instance at any time and only pay for the hours you used. It's a lot more flexible than the multi-year commitment you need to make in order to get the best pricing out of EC2.
  - whoisjohnkid 6y ago
    Keep in mind that lightsail charges you for inbound and outbound data transfer; whereas EC2 only charges for outbound ... so depending on your network usage the 4TB isn’t a guaranteed saving.
  - byefruit 6y ago
    Yea and 4TB is $360 which dwarfs everything else.
    - PhantomGremlin 6y ago
      Okay, I don't get this at all!???? If I want to move 400 TB per month out of Amazon, why can't I just proxy it thru 100 Lightsail instances? Isn't that much cheaper than paying directly per byte? For that much saving it's probably worth figuring out how to do this. It can't be hard. Isn't it "free" to move data internally to a lightsail instance, as long as both are in the same DC?
LogicX 6y ago
Just to be clear, this would be more appropriately titled How we saved money moving from AWS to VPS providers. It comes as little surprise to me that the AWS lightsail offering saved them money over traditional AWS services, just as they’d save even more using two dedicated servers with any reasonable provider and being able to have a hot-failover for everything. At DNSFilter we’ve gone through the evolution of Heroku -> AWS -> VPSs -> Dedicated -> and now are setting up our first colo rack. I did a price comparison recently between Dedicated, AWS, and Colo. Dedicated is 15% the cost of AWS for our needs, and colo will be 42% the cost of Dedicated for us. Now keep in mind we have dedicated DevOps staff and are running at a very different scale from OP, and such a solution is not for everyone. But I personally have never understood the folks who love to brag about having spent much time and effort optimizing AWS to spin up and spin things down, when for the same cost I could just have a 10x more powerful server sitting there, on all the time to handle a spike in load, and I can utilize the extra resources or get things done faster with faster gear at 1/10th the price.
- ies7 6y ago
  In the end, the most optimized way to save AWS cost is to call the AWS sales guy for lunch. I don't ask for anything but usually 2-3 days after that he'd call to inform that I got a few thousands of AWS credit.
- mwcampbell 6y ago
  > two dedicated servers with any reasonable provider and being able to have a hot-failover for everything. Except you have to implement that hot failover yourself, rather than using something like EC2 auto-scaling groups or an RDS multi-AZ deployment. I'm not confident that I'd get it right, and the middle of a night is a hell of a time to discover that you got it wrong.
  - marcosdumay 6y ago
    How much uptime do you need, and how much does EC2 provide? Things will depend on that. But from what I've seen, the point may be moot. Odds are that the EC2 uptime is around the same as you'll get on a single machine VPS anywhere anyway.
elcomet 6y ago
It's the difference between pets and cattle. They are using Lightsail to create pets, it has not much configuration. When you need a cattle, you need more config and complex setup which costs money. As they say in the end, this is because they are a micro startup and don't need a huge scalable infrastructure. It's funny as it seems to be the inverse of economy of scale. The more you grow, the higher the marginal cost. But I think that Amazon gives discount to large users
pachico 6y ago
I read a lot of "you are wrong", "you didn't think about this", etc, which I'm not going to get into. I embrace these posts as an invitation to re-evaluate, with your own data and use cases, your technical decisions and for that I'm always grateful. On the same note, in case someone is digging into how to reduce CDN bills, I wanted to share that we are quite happy with BelugaCDN. It distributes objects stored in S3 using, in a hacky way, referrals as authentication method. Lots of money saved there.
t0mas88 6y ago
I don't understand why this is so special? Ours is about 4% for a very compute heavy service handling millions of events per day across four continents.
Software202 6y ago
Really good one.
longtermdd 6y ago
AWS expense/ revenue is an insane anti-metric to track.
scarface74 6y ago
For context, I know the ins ands outs of most of the core AWS services really well from the dev, Devops, and ops side. But, my advice tends to be Lambda first if it is really low volume, LightSail second, and full AWS third. As far as Lambda, I often recommend proxy integration, where you can just use the standard API framework for whichever language you choose (Django, Flask, Express, ASP.Net, etc) add three lines of code and push the entire thing into Lambdas. This gives you the flexibility to deploy to a VM later with no code changes. For your static assets use S3, except for the case of Lightsail where you get plenty of free bandwidth.
- root993 6y ago
  Hello, author here. I totally agree on Lambda first. The only reason we did not do that is because when we started this product I was not well versed with Lambda and serverless and preferred to work with something that I dealt with previously. If I could go back in time, I would set up all our applications on serverless.
  - scarface74 6y ago
    Disclaimer: I work for AWS Professional Services but I just started. All of my experience comes from working at outside companies. From the perspective of an outside, boots on the ground Developer/architect, I’ve never worried about “vendor lock in”, I believe you should choose your infrastructure wisely and go all in. But, I do worry about “Lambda lock-in” for APIs. I like the optionality of being able to deploy my APIs anywhere just by changing the CI/CD pipeline. That’s why I recommend using proxy integration. Every language supported by Lambda has a method to just throw your standard API in lambda without tying yourself to it. Here is an example for Node/Express https://github.com/awslabs/aws-serverless-express https://github.com/awslabs/aws-serverless-express Python/Flask (ignore the DDB part): https://www.serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/ https://www.serverless.com/blog/flask-python-rest-api-server... C#/Web API https://aws.amazon.com/blogs/developer/deploy-an-existing-asp-net-core-web-api-to-aws-lambda/ https://aws.amazon.com/blogs/developer/deploy-an-existing-as...
- lowmemcpu 6y ago
euph0ria 6y ago
How do you solve high availability with the nginx setup? What software for heartbeats?
pezo1919 6y ago
Here is how I do: no revenue, no bill. ;D
- mcintyre1994 6y ago
  That’s significantly more profitable than most startups on AWS without revenue ;)
- random_visitor 6y ago
  I've first hand, witnessed people having no revenue and a long AWS bill. A direct consequence of using all the hip services when they didn't need any. I'd say you're doing great (to be clear this is not sarcasm)
axegon_ 6y ago
It's no secret that I've had my fair amount of criticism towards AWS's billing and how unpredictable it becomes, even if you read every fine print and take everything into account. Especially if you are a heavy user, "correction invoices" are not uncommon. That said, cloud services can be incredibly cheap if you are very careful and smart about it. Especially if you combine several of them.
KenCochrane 6y ago
Maybe I missed something, but how do you handle the fact that your nginx server is a single point of failure? If that goes down, traffic can’t get to your web servers. Do you have more than one, and DNS load balance, or do you just live with the risk? One of the main reasons why I use an ALB/ELB is so that I don’t have that SPOF. If you found a way around that, please share, I would love to know, so I can save some money :)
- root993 6y ago
  I hate to admit it but the nginx server infact is a SPOF. We currently handle it by setting up alerts all over the place so I can take quick action if something goes sideways, but other than this I have not really found a way around it. We also have latest snapshots ready of all our instances so that I can get another server running ASAP during a calamity.
- n_u_l_l 6y ago
  His database also is. I think it's highly unprofessional to use a setup like this in production. Looking at his product, it seems like a product whose downtime has a big impact on their clients.
reledi 6y ago
Another approach to consider, that's so straightforward and quick that most will consider it boring and overlook it. - kill services with low usage - downgrade instance size - downgrade instance type - merge databases - schedule services with obvious usage patterns to shut down when not used - use EC2 spot instances Most importantly, it requires an aggressive culling mindset. If drastically reducing the AWS bill means staying afloat, then make bold choices.
elondaits 6y ago
One problem of VPS solutions is that they're easy to maintain until you have to do it. At some point the technical debt comes back and you need to upgrade the stack components and eventually the OS as well, without sacrificing uptime. ... if you did things well, it's just starting another instance and installing / copying things over with some minimum downtime. But if you don't have a 100% documented stack, you don't know which configurations you touched to make things work, you have files lying around, then you're probably going to pay back all your savings and more in the workdays needed to migrate. At my current employer we never have time for maintenance, and we don't have professional expertise (the "I worked exclusively at this for many years" kind, I mean) at webhosting, security, system admin or dba, so I heavily lean towards more "managed" and cloud solutions.
- disgruntledphd2 6y ago
  I completely agree (it's always the small changes that make things work that cause problems), but for this product which is greenfield and has a very small team, it seems like a good fit. Like, I wouldn't run Shopify like this, or anything really large, but it may get them to a better place right now. What I really liked about this post was the discussion of what they actually needed and how to achieve their goals. It's a tradeoff of time against money, and I wish more people documented their thought processes and approaches to this kind of stuff.
- amelius 6y ago
  But cloud solutions can change their UIs and APIs anytime they want. They can also drop features as they please.
  - briffle 6y ago
    Has any major cloud provider ever done a major change like that without 6+ months of advanced notice?
m3kw9 6y ago
You said you basically use Shopify as CDN, what’s stopping everyone from just setting it up and abuse it?
- root993 6y ago
  Hello, author here. Like I mentioned in the post, there is no way to programmatically upload anything to the Shopify CDN and the CDN cache cannot be invalidated once you upload a file. If you want to update an existing resource, all you can do really is upload a new one. This ofcourse still does not completely prevent people from abusing it, but it does restrict usage to a large extent. There is also the issue of giving out a cdn.shopify.com link to your customers instead of something that has you company branding on it. This is not a problem for us because our customers do not have to manually add this snippet to their website and we do it via an API instead, so this link is not apparent to our customers
ykevinator 6y ago
You have a beautifully designed site, thanks for the write up
deleted 6y ago
[deleted]
darsoli 6y ago
For the mathematically disinclined (such as myself), the total savings are at least $229/mo .. not bad! Curious if you could use Cloudformation to provision these instances and setups? https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-cloudformation-stacks https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...
person_of_color 6y ago
Another day, another SaaS that I can’t believe has value.
mark_mart 6y ago
As a summary: They use AWS, see huge bills, then decide to not to use AWS services and switch to redis, nginx etc.
upgoingstar 6y ago
How do you go around the Data Transfer pricing? That area is just painful for me every single time I try to process some good amount of data.
peterwwillis 6y ago
You can shop for deals on VPSes at lowendbox.com. But if you're trying to run a business, this is a waste of time. Find a provider which is highly reliable, which can also automatically rebuild all failed infrastructure with no intervention needed. That eliminates 99.999% of the providers out there. You're paying a premium to never have to think about your tech again, so you can focus on the business. Besides using the free tier and other AWS services which are practically free at low uses, you can use cost effective options like Fargate Spot Instances and EC2 Reserved Instances. I highly recommend Fargate over running instances. Use Lambda with CloudWatch triggers if you need to schedule occasional jobs (or use Fargate's feature for that). Try to avoid heavy reliance on caches, ElastiCache is kind of a rip off. Move as much content to static as possible, use CloudFlare to reduce bandwidth costs. If you're gonna serve over S3, you might as well front it with CloudFront as it's actually cheaper due to caching at the edge, and also more reliable. ALBs are expensive but very useful for APIs as well as autoscaling (if you have to run instances, run them with an ASG, which also means having versioned AMIs)
aiibe 6y ago
Definitely off topic, are you using Twilio/Whatsapp at superlemon.xyz ? I wonder how do you cover the cost for unlimited outbound messages in your free plan... Thanks
jorangreef 6y ago
Do Hetzner or xneelo reflash the firmware on server hardware when they recycle them across users?
anthony_barker 6y ago
When I did data center stuff for a large company the biggest cost was network - not servers. Specifically network security and redundancy. Its amazing to me that this is all included for basically free except for Application level filtering by these providers.
- EdwardDiego 6y ago
  Yep, I was consulting for a firm with an already sizeable infrastructure investment/dependence on AWS, and their architects were proud of their best practice usage of multiple AZs for redundancy, but weren't know that inter-AZ traffic cost also. So I imagine they're going to have fun trying to incorporate that into their infrastructure layout. In my day job we saved $600 a day by cutting down on needless inter-AZ traffic.
sandGorgon 6y ago
The general rule of thumb to drop AWS bill is spot instance. Which is very hard to do unless you build your application to be tolerant of server kills.
tzs 6y ago
> An EC2 instance with 2 virtual cores, 4GB RAM and a storage of 80GB costs roughly 37$ a month and a Lightsail instance with the exact same configuration costs 20$ a month which is almost half the cost! In general I'm not sure you can really compare instances from different service by just look at cores, RAM, and disk. I used to have my small website and email server at Rackspace, on the smallest, cheapest instance. I ended up getting out of mail hosting (moved that to Fastmail), and putting the small website on Lightsail. The Lightsail instance and the old Rackspace instance have the same nominal specs for virtual cores, RAM, and disk. (Actually, Lightsail may be better on disk--I don't recall if the Rackspace one was SSD like Lightsail). The main thing the website actually does is host some graphs showing the temperature in my house. An ESP8266 temperature monitor I made uploads a sample once a minute. A script was running once a minute on the server that using gnuplot to make graphs of temperature over the last hour, 3 hours, 12 hours, 48 hours, and over all samples. At Rackspace it ran that fine, while at the same time gathering my mail from several services via fetchmail, receiving mail for my domain on its smtp server, and running spam filtering. On Lightsail just handling the website it was locking up every few hours. I managed to catch one pre-lockup and found the load average was something like 300. What was happening was sometimes that once a minute graph generation task was taking longer than a minute, and that would slow down the next one, and so on. Oddly, it didn't seem to be gnuplot that was taking too long, but rather the script that took the file containing all the samples, and extracted just the samples newer than a specific threshold. At least, running everything by hand that was the only step I ever saw take unusually long. I changed it to every 5 minutes to temporarily stop the frequent hanging, and then added a check to my script to skip regenerating the graphs if a previous instance of the script was still running to fix things permanently. I also changed it from storing the data in just a big file of samples sorted by time to an sqlite DB.
ravedave5 6y ago
A lot of this is penny wise pound foolish. Managed services cost more, but they provide more. RDS is just such a simple slam dunk for what you get for the cost. How much does it cost to spend 8-20 hours a week handling maintenance and upgrades and scaling for these services? I guess I'm bitter working on a project that was built with all sorts of crazy custom deployment stuff too conserve costs that's a total OPs nightmare. Moving from Redis on EC2s to elasticache was such an upgrade. We constantly had little issues here and there. Edit - Alsocontainerize your app from the start so that if it does take off you can slam it in ecs or k8s.
barnabask 6y ago
I'm interested in the managed DB part of this writeup, specifically that OP chose Lightsail. The last time I looked into it, Lightsail was MySQL only, so that was good to know. I wrote a PostgreSQL DBaaS Calculator that got some traction here a while ago, and I just updated it this evening to add Lightsail to see how it stacks up: https://barnabas.me/articles/postgres-dbaas.html#calculator https://barnabas.me/articles/postgres-dbaas.html#calculator No surprise, Lightsail is similar pricing to RDS with a 1-year commitment, but it's month-to-month. It's a pretty good deal until you need more than 8 GB memory, 240 GB storage, 2 cores ($115/month Standard plan). But Azure or AWS RDS are the ones to beat.

298 points by grwthckrmstr 6y ago

TLDR: Use Lightsail, Amazon's approximation of a dedicated server. In my opinion, that approach sacrifices pretty much all the benefits that cloud proponents usually talk about, like only paying for what you actually use or scaling up and down on demand. In fact, I'm not sure what the actual difference would be between Lightsail and a good dedicated hoster that provides backup and failover services.

jakozaur 6y ago
I disagree. Start with LightSail customize later. Too many startups starts with Kubernetes, microservices putting a lot of engineering into infrastruture when they should worry about product market fit and simple server would just do fine.
- fxtentacle 6y ago
  You could just as well start with one cheap dedicated server and then migrate to the cloud once you know that you need it. For example, one of my companies is doing hosting and image delivery services for photographers. I started it on one dedicated server, and now 10+ years later it's running on 3 dedicated servers. My hoster does backups and OS upgrades, so there was never any reason to go into the cloud. Except maybe, if I wanted to burn cash on slow performance ;)
- kristopolous 6y ago
  Complex solutions to simple problems pad resumes. It's important. I've been looking for a job recently and my lack of experience with excitedly building expensive overly engineered big piles of stuff to do simple things is really hurting me. When I suggest simple solutions proportional to the problem size people look at me like I'm some kind of simpleminded ninny. And then when I describe working systems I've made using the approach on companies that exited north of 10 million, then they usually just think I'm a bullshitter. 10,000 users? Just do a simple lamp stack on a $10 month vps and don't code it like a moron, it's not hard. And it can be delegated to almost anyone, that's important for longevity. That's it, move on with life. I've even offered to show logs, I mean, it's remarkably bad how hard it is to find work as a cost-cutting product-focused engineer, nobody wants that shit, really. They want autoscale, prometheus, travis, blah blah blah... as fancy and crazy as possible, an empire of infrastructure instead of infrastructure for an empire... It's a damn cult. I'm thinking about giving up and giving in. Maybe spend a month and drop $10,000 or whatever on a bunch of the tech just so I can talk about these things. --- The second thing I'm constantly surprised by is how many obvious things are formalized with fancy names, math equations, papers, etc. Stuff I'd think like literally anyone with half a brain would do. Recently I found out about "CRDT". Apparently I've been doing it for a decade... It's just super simple stuff wrapped in needless formalism, like back when design "patterns" were all the rage. I used to think "why is there a name for that?" it's such basic stuff. So saying "dude, the emperor has no clothes, here, take my coat" - bad interview strategy and it's hard to rectify because there's like 50 naked emperors and silicon valley culture is on a constant rotating basis of seeing the nakedness of 5 of them and ignoring the other 45. Maybe what I should do is just a couple throwaway interviews to focus on finding out the current fads and then be able to give the right cultural signals for the ones I actually want.
  - mulhoon 6y ago
    Great response. We’re all obsessed with infrastructure and over-engineering, myself included. Startups like to think they’re working on the next big thing, so they want to use the tech that the big boys use and bask in their anticipation of scale. You can easily save cost and time to validate your business with smaller hosting approaches, though it does look amateur from the outside. It’s not amateur, it’s smart.
    - kristopolous 6y ago
      Yeah I've been in LA for 12 years which is much more practical for the most part. And it's been fine, profitable, revenue generating, sure. Mysql, php, a bit of python, easy stuff. Honestly I've been wanting to move and try out silicon valley again though and it's been hard because I don't speak that language and frankly I'm skeptical and hostile to it - bad idea. That bad attitude needs to go. I've been in practical startup land too long so I keep getting stonewalled from getting in to the sv world. I'm trying to contribute to some companies open source projects recently and then basically earn my way in through labor since I can't do the virtue signaling. It's only been a few weeks, nothing yet. I gotta learn how to be along for the ride without always trying to reach for the steering wheel. Hopefully by the end of summer I'll find a lead somewhere. It's not about the money, it's kinda actually how I want to spend my time for the next year or two.
antonvs 6y ago
FTA: > We are a team of two people Articles like this reflect the fact that one size does not fit all. > a good dedicated hoster that provides backup and failover services. In theory that sounds reasonable. In practice the ability to completely manage the system via a web console and APIs makes a big difference.
- fxtentacle 6y ago
  I have never needed it, but the hosting provider which I use actually has an API that you can use to reboot servers into different OS images, or trigger hardware resets, or do an kvm access. https://wiki.hetzner.de/index.php/Robot_Webservice/en https://wiki.hetzner.de/index.php/Robot_Webservice/en But honestly, if you have only one server, what do you need an API for? It'll be much faster to just configure things by hand. It's not like you'll switch servers very often. Or at least the original article says that they committed to monthly cancelation terms.
- dx034 6y ago
  Even Hetzner as the cheapest VPS and dedicated server provider offers an API to completely manage servers.
  - isatty 6y ago
    You can definitely do cheaper than hetzner (like OVH).
redis_mlc 6y ago
> all the benefits that cloud proponents usually talk about, like only paying for what you actually use or scaling up and down on demand. That's a trope that I have never seen in practise. A company might have some small percentage of clusters that autoscale, but usually there's a bunch of servers that don't. Also, when AWS has control plane problems, autoscaling doesn't work, you get swamped, your health checks fail, and you lose all your servers and you're hosed. So it's better to always over-provision in the first place.
- fxtentacle 6y ago
  Same here. Heroku offers auto-scaling for the web dynos, but they advise you to do near-realtime tasks like the thumbnails for image uploads using background workers, for which there is no Auto-scaling.
alain_gilbert 6y ago
The difference would be: - They could have much better service - And bring down their expenses by at least half again I used lightsail, and it's a terrible service. It is very expensive (still) compared to other hosting providers. And if your instance ever go out of memory, it become unresponsive for as long as you don't go manually restart it. On other providers that I use, the OS would just "sacrifice child" (kill the process) and restart it. It's not ideal, but much better than having to go there yourself to restart the whole thing.
- oefrha 6y ago
  Sounds like OOM killer is disabled by default on Lightsail. Shouldn't be hard to turn it back on by modifying a line or two in sysctl.conf.
- fxtentacle 6y ago
  I've been using cgroups since way before docker made them cool exactly for this: if one process goes rogue on memory use, make sure only exactly that one crashes. And yes, I fully agree with you. In addition to the other disadvantages, Lightsail also shifts the burden of process monitoring and management onto you.
  - alain_gilbert 6y ago
    I use docker. And yes, it normally just take care of restarting the app whenever something goes wrong. But somehow on lightsail (only), the machine just goes completely unresponsive instead.
Proven 6y ago
I think so too. I just had a quick look at the Lightsail pages and thought "This is just plain old hosting, with hooks into a highly proprietary cloud environment."

whack 6y ago

I appreciate the disclaimers at the end: > I would like to put emphasis on the fact that we are a micro-SaaS product that solves a small and specific use case and therefore this kind of AWS setup worked for us. This may not work for big organisations or products where the traffic is erratic. > This setup will also not work for folks who have a ton of stuff to do already and would prefer to use managed services and not take the additional headache of monitoring, maintaining and provisioning hardware resources on a regular basis because this has a time cost to it. It is refreshing to see a post that doesn't pretend as though its recommendations are the holy grail for anyone and everyone. I personally prefer saving time and effort, over a ~$500 monetary discount. But it's nice to read posts like these and learn more about alternatives such as Lightsail.

tehnicaorg 6y ago
> I personally prefer saving time and effort I guess it depends where you live and how much your time costs (and if you have any time to spare).
- ashwinm 6y ago
  Yup, This is some high quality jugaad. ;)
ldoughty 6y ago
> I personally prefer saving time and effort, over a ~$500 monetary discount. Completely agree. As the general "Ops" manager, I hate getting the call at 2am because something broke... I hate it even more when it's not really fixable by me in the first place (e.g. black box appliance or software).. but I feel a heck of a lot better when I can point the finger at an external team that's paid and supposed to be experts... If you outsourced Redis or you DB, and someone like AWS or MongoDB themselves has an issue... Will they pay people a lot of money to simply provide that service reliably... I get paid to keep 20 things connected, not be the subject matter expert in all of them.
- amelius 6y ago

KorematsuFred 6y ago

Tldr; Author is talking about few hundred dollars here.

sushshshsh 6y ago
At that rate a raspi on starbucks wifi would get them to 0.2%

ChicagoDave 6y ago

I’m wondering if Lambdas and DynamoDB would be competitive to this setup.

aritraghosh007 6y ago
Since they are serving about 250 requests per second, Lambdas might end up being more expensive, factor in an expensive DynamoDB and that monthly AWS bill looks scary.
- root993 6y ago
  Hello, author here. I just want to confirm that this is one of the reasons why we did not go with Lambda and the other reason being I was not fully aware of and versed with serverless when we started this venture
- ChicagoDave 6y ago
  I’m thinking otherwise. If the payloads are small, DynamoDB is going to be cheaper and nothing will come close to the savings from Lambda compute.
PaulKeeble 6y ago
Almost certainly. A lot of groups have shown low volume systems have been remarkably cheap using lambda's compared to trying to do the same thing in a VM with EC2/lightsail. Whether that is genuinely because it is that much cheaper for Amazon or whether it is trying to lock people into the platform I don't know.
redis_mlc 6y ago
> I’m wondering if Lambdas and DynamoDB would be competitive to this setup. Depends, but it's important to know the details. Lambdas are used for 2 things in AWS: 1) Glue for managing services and events. (Required for advanced AWS mgmt.) 2) User applications. Lambdas have improved over time, but initially had all kinds of limitations (startup time, database access, 15 minute job limits, etc.) DynamoDB has also improved a lot. Initially hot keys could cost you a lot of money. Some companies have designed their applications to use it in a way that absolutely no administration is required, reducing DBAs and Ops people. Source: DBA.

PaulKeeble 6y ago

If you end up in the lightsail world and you are not utilising Amazon's other services then its probably cheaper to do this with another provider. Someone like Contabo or Hetzner will get you VMs at substantially less cost with similar fixed hardware and just give you the box but for a lot less a month. At these low scales with a completely open source stack Amazon isn't good value in my opinion. It is as you grow and it provides the scale that it becomes valuable.

robertlagrant 6y ago
This is interesting. Do you have (or know of) a published comparison somewhere?
- EGreg 6y ago
  Would like to see this too
- deleted 6y ago
  [deleted]
- dx034 6y ago
  I don't have a comparison but Hetzner is pretty much unbeatable in price if you can live with having your servers in Europe. If not, OVH, DigitalOcean, Linote or Vultr are good alternatives with datacenters around the globe and prices below that of AWS.
  - mxschumacher 6y ago
    I know that hetzner is affordable, but is digital ocean really cheaper than AWS?
    - uaas 6y ago
      That would largely depend on your usecase.
  - callmeal 6y ago

mattbillenstein 6y ago

Their setup is trivial - they could do it at 0.2% of revenue with a cheap vps on Linode or other...

nordsieck 6y ago
> Their setup is trivial - they could do it at 0.2% of revenue with a cheap vps on Linode or other... Lightsail has pretty comparable pricing to Linode. I'm sure they could re-architect their app to use fewer instances, but they could do that and stay on Lightsail as well. Moving to linode isn't going to give them a 90% savings.
- wolco 6y ago
  The price seems on par. Are there hidden costs on aws?
  - deleted 6y ago
    [deleted]
  - dijit 6y ago
    I'm assuming that you might not be aware of AWS pricing. It's actually incredibly complicated and often very difficult (if not impossible) to predict. We have a dedicated part of our organisation which exists solely to figure out costs ahead of time for project. I'm not sure if it's intentionally obfuscated, I would suspect not, because "pay for what you use" can be broken down into many areas.
    - scarface74 6y ago
      What is incredibly complicated about Lightsail? That’s kind of the point of it.
    - radicalbyte 6y ago
      Azure is the same and Microsoft spent 20 years making their software licencing more and more unpredictable. It absolutely is a strategic choice and not an accident.

samstave 6y ago

I have a question: (it may not be related to this article in any other sense aside from "really lightweight" use of AWS... I would like to have a workflow of really lightweight QR/barcode transactions... but with what seems to me currently a complex implementation: 1. you have a prescribed barcode ParentA 2. you scan that barcode and create a QR code ParentB 2.a: you ascribe that QR to an object ChildA 3.n: you have that ChildA go through many iterations 3.n+: ChildA may be conjoined with ChildB,C,D etc... -- but I want an iterative QR code that follows the family tree back to parent A and keeps history of all transitions... (I know I am wording this poorly... just thinking out loud...)

unixhero 6y ago
So what about the grandparents, and how would ChildN go to kindergarden?
- samstave 6y ago
  What a fucking stupid response. The workflow that I am talking about would apply to taking the nightmare out of METRC tracking system for cannabis -- the METRC system is a piece of shit - but wants complete chain of history of how plantA becomes productA - and it is a bullshit workflow - and I have a way to make it easier - and by using a system similar in the way described in this post, it could be done very cheaply...
  - unixhero 6y ago
    For what is this workflow? Just curious. If it's barcodes being used by humans in the real world, get a software package to handle it, check the solution from Zebra https://www.zebra.com/us/en/products.html https://www.zebra.com/us/en/products.html

barrenko 6y ago

Slightly off-topic, but is there a book to learn about this kind of stuff, preferrably without having to sign up for AWS, but that does have code samples and whatall?

ramraj07 6y ago
There's only so much you can learn about this without signing up and actually getting your hands dirty unfortunately.
root993 6y ago
There is one book that I would highly recommend - https://gumroad.com/l/aws-good-parts https://gumroad.com/l/aws-good-parts

ramraj07 6y ago

I dabble extensively in being dirt cheap with my monthly cloud spend on personal projects, and after much experimentation, I have settled on the following: 1. elastic beanstalk, no docker: EB comes with really nice defaults so that you can quickly whip up a flask app, upload it and it just works. It provisions a small ec2 instance by default which iirc costs 10ish a month at best. Importantly, any operation you do with EB will by definition be ready for continuous deployment since you don't get an option to ssh into the machine to deploy. It's extensible enough to add whatever extra stuff you need as well. Only thing it can't do is simple caching ( if you scale to more than one instance that is), but that can be solved by having a separate eb deployment for a worker that can take care of all these aux stuff (elasticache is expensive I think). In a pinch, it also scales well (though the default cheap deployment does not have a load balancer). 2. Just suck it up and go with RDS postgres. Again, I see too many things going wrong with spinning up your own db in an ec2 instance especially if that instance goes down. Im too lazy to write backup scripts and keep track of them! The cheapest RDS postgres costs 13 a month or so, but I just suck it up to power whatever side projects I do. Postgres means I'm working with something I know, and I get full text search and pubsub for free. And whatever I write is not locked code in anyform, and can be scaled up if needed as well. More importantly I'm only spending my time in technologies that are relevant for me in my day job so that's a win. 3. Github Actions to deploy to eb. just a few lines of yaml and you instantly get continuous deployment directly from your repo for free! Really can't beat that. I have meant to try out heroku since it could be cheaper from what I have read. But I couldn't figure out what their S3 alternative is or how different they are from the canonical cloud offerings. I'm sure it can be so much more cheaper, but I'm not good at advanced networking or sysadmin, and I'm too lazy / bored / disorganised to write deploy scripts or sshing into remote machines. I'm also always afraid of if/how long I need to re-provision a vps/ec2 if it goes down. Not that they do, but they can, and that scares me.

root993 6y ago
Hello, author here. 1. When I started working on this I was not fully aware of the large array of services that AWS provides and therefore our setup may not be the best possible one. So I would be checking out Elastic beanstalk for sure and see if it is feasible to use for my next venture. 2. Just to clarify, we are not running mysql on a lightsail instance, rather we are using a managed database provided by lightsail which automatically takes backups on its own and also has the option to restore a new DB from an existing backup. 3. Thanks for the insight on EB. Will be looking into it for sure. The thought of any of these instances going down scares me as well. but I would like to believe that I have set up enough alerts everywhere so I can take immediate action :)

omnifischer 6y ago

Text from about me: > where I learnt how to scale an application to serve ads to millions of users a day Please lets stop this 'race'.

lend000 6y ago

A bit off topic, but I bet someone here knows. When running an EC2 instance and not using all of the cores on the socket (for example, using a c5.large instance instead of a c5.12x large, which gives you all the cores on the socket), you presumably are sharing your L3 cache with your neighbors on the same socket, because that's how the processor is designed. Is there a way that the hypervisor allocates a dedicated portion of the shared L3 cache to just your instance, or is it a free for all for all of the L3 cache space against potentially noisy neighbors?

Keyframe 6y ago
Sounds like a textbook example of where and how a side channel attack would look like.
nisten 6y ago
Yeah that is true, you are sharing L3 cache. In order to mitigate some of recent intel issues I think AWS actuallly has their own chip now on newer motherboards to handle the hypervisor duties securely. Otherwise, they'd do it in software patches for older CPUs and take the performance hit of the patch. I'm not sure how much the hypervisor would reserve off of the L3, it is likely to be free for all however you'd still have quite a bit of dedicated L2 and L1 on most xenons. With AMD's first gen EPYC it's a little bit different because clusters of cores share a cache and you can get weirdly high latencies depending on which cores you're using, (i.e. cores 8 and 9 being too far apart) Also according to this anandtech article, the average total CPU load for physical aws machines is ~60% and is actively balanced out by them. And yes, running benchmarks on a machine without noisy neighbors yields very significant improvements, up to 2x better on the benchmark scores. They measured this by comparing renting out all the cores of a machine vs only renting out the 4 or however they needed . https://www.anandtech.com/print/15578/cloud-clash-amazon-graviton2-arm-against-intel-and-amd https://www.anandtech.com/print/15578/cloud-clash-amazon-gra... I'm assuming they'd put a CPU into sleep/hibernate mode in order to save power instead of having it only run at 5% utilization.
- lend000 6y ago
  Without any dedicated hypervisor tricks, can't a typical L3 cache eviction algorithm also evict memory that is assigned to another core and currently residing in its L2 or L1 cache? (Thereby flushing even the higher level caches if another core is really noisy.)

raverbashing 6y ago

So Lightsail is the AWS equivalent of DO/Linode/etc?

sumedh 6y ago
Yes, AWS probably realized that they losing money to DO/Linode so to capture some of that market they launched Lightsail.

rumanator 6y ago

> An EC2 instance with 2 virtual cores, 4GB RAM and a storage of 80GB costs roughly 37$ a month and a Lightsail instance with the exact same configuration costs 20$ a month which is almost half the cost! The author completely failed to do his homework. An a1.large instance is only 37$ if it's an on-demand instance. You pay a high premium to be able to pull the switch in the exact minute you cease to need one. If he's willing to go with AWS lightsail with it's monthly plan, the same a1.medium instance type is about 11.75$/month as a reserved instance, and can be had for about 131$/year as well.

kijin 6y ago
The Lightsail instance comes with 4TB/mo of free transfer. Even using a small fraction of that transfer on EC2 will cost you more than the instance itself. Also, creating an instance on Lightsail isn't exactly a monthly commitment. You are free to delete your instance at any time and only pay for the hours you used. It's a lot more flexible than the multi-year commitment you need to make in order to get the best pricing out of EC2.
- whoisjohnkid 6y ago
  Keep in mind that lightsail charges you for inbound and outbound data transfer; whereas EC2 only charges for outbound ... so depending on your network usage the 4TB isn’t a guaranteed saving.
- byefruit 6y ago
  Yea and 4TB is $360 which dwarfs everything else.
  - PhantomGremlin 6y ago
    Okay, I don't get this at all!???? If I want to move 400 TB per month out of Amazon, why can't I just proxy it thru 100 Lightsail instances? Isn't that much cheaper than paying directly per byte? For that much saving it's probably worth figuring out how to do this. It can't be hard. Isn't it "free" to move data internally to a lightsail instance, as long as both are in the same DC?

LogicX 6y ago

Just to be clear, this would be more appropriately titled How we saved money moving from AWS to VPS providers. It comes as little surprise to me that the AWS lightsail offering saved them money over traditional AWS services, just as they’d save even more using two dedicated servers with any reasonable provider and being able to have a hot-failover for everything. At DNSFilter we’ve gone through the evolution of Heroku -> AWS -> VPSs -> Dedicated -> and now are setting up our first colo rack. I did a price comparison recently between Dedicated, AWS, and Colo. Dedicated is 15% the cost of AWS for our needs, and colo will be 42% the cost of Dedicated for us. Now keep in mind we have dedicated DevOps staff and are running at a very different scale from OP, and such a solution is not for everyone. But I personally have never understood the folks who love to brag about having spent much time and effort optimizing AWS to spin up and spin things down, when for the same cost I could just have a 10x more powerful server sitting there, on all the time to handle a spike in load, and I can utilize the extra resources or get things done faster with faster gear at 1/10th the price.

ies7 6y ago
In the end, the most optimized way to save AWS cost is to call the AWS sales guy for lunch. I don't ask for anything but usually 2-3 days after that he'd call to inform that I got a few thousands of AWS credit.
mwcampbell 6y ago
> two dedicated servers with any reasonable provider and being able to have a hot-failover for everything. Except you have to implement that hot failover yourself, rather than using something like EC2 auto-scaling groups or an RDS multi-AZ deployment. I'm not confident that I'd get it right, and the middle of a night is a hell of a time to discover that you got it wrong.
- marcosdumay 6y ago
  How much uptime do you need, and how much does EC2 provide? Things will depend on that. But from what I've seen, the point may be moot. Odds are that the EC2 uptime is around the same as you'll get on a single machine VPS anywhere anyway.

elcomet 6y ago

It's the difference between pets and cattle. They are using Lightsail to create pets, it has not much configuration. When you need a cattle, you need more config and complex setup which costs money. As they say in the end, this is because they are a micro startup and don't need a huge scalable infrastructure. It's funny as it seems to be the inverse of economy of scale. The more you grow, the higher the marginal cost. But I think that Amazon gives discount to large users

pachico 6y ago

I read a lot of "you are wrong", "you didn't think about this", etc, which I'm not going to get into. I embrace these posts as an invitation to re-evaluate, with your own data and use cases, your technical decisions and for that I'm always grateful. On the same note, in case someone is digging into how to reduce CDN bills, I wanted to share that we are quite happy with BelugaCDN. It distributes objects stored in S3 using, in a hacky way, referrals as authentication method. Lots of money saved there.

t0mas88 6y ago

I don't understand why this is so special? Ours is about 4% for a very compute heavy service handling millions of events per day across four continents.

Software202 6y ago

Really good one.

longtermdd 6y ago

AWS expense/ revenue is an insane anti-metric to track.

scarface74 6y ago

For context, I know the ins ands outs of most of the core AWS services really well from the dev, Devops, and ops side. But, my advice tends to be Lambda first if it is really low volume, LightSail second, and full AWS third. As far as Lambda, I often recommend proxy integration, where you can just use the standard API framework for whichever language you choose (Django, Flask, Express, ASP.Net, etc) add three lines of code and push the entire thing into Lambdas. This gives you the flexibility to deploy to a VM later with no code changes. For your static assets use S3, except for the case of Lightsail where you get plenty of free bandwidth.

root993 6y ago
Hello, author here. I totally agree on Lambda first. The only reason we did not do that is because when we started this product I was not well versed with Lambda and serverless and preferred to work with something that I dealt with previously. If I could go back in time, I would set up all our applications on serverless.
- scarface74 6y ago
  Disclaimer: I work for AWS Professional Services but I just started. All of my experience comes from working at outside companies. From the perspective of an outside, boots on the ground Developer/architect, I’ve never worried about “vendor lock in”, I believe you should choose your infrastructure wisely and go all in. But, I do worry about “Lambda lock-in” for APIs. I like the optionality of being able to deploy my APIs anywhere just by changing the CI/CD pipeline. That’s why I recommend using proxy integration. Every language supported by Lambda has a method to just throw your standard API in lambda without tying yourself to it. Here is an example for Node/Express https://github.com/awslabs/aws-serverless-express https://github.com/awslabs/aws-serverless-express Python/Flask (ignore the DDB part): https://www.serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/ https://www.serverless.com/blog/flask-python-rest-api-server... C#/Web API https://aws.amazon.com/blogs/developer/deploy-an-existing-asp-net-core-web-api-to-aws-lambda/ https://aws.amazon.com/blogs/developer/deploy-an-existing-as...
lowmemcpu 6y ago

euph0ria 6y ago

How do you solve high availability with the nginx setup? What software for heartbeats?

pezo1919 6y ago

Here is how I do: no revenue, no bill. ;D

mcintyre1994 6y ago
That’s significantly more profitable than most startups on AWS without revenue ;)
random_visitor 6y ago
I've first hand, witnessed people having no revenue and a long AWS bill. A direct consequence of using all the hip services when they didn't need any. I'd say you're doing great (to be clear this is not sarcasm)

axegon_ 6y ago

It's no secret that I've had my fair amount of criticism towards AWS's billing and how unpredictable it becomes, even if you read every fine print and take everything into account. Especially if you are a heavy user, "correction invoices" are not uncommon. That said, cloud services can be incredibly cheap if you are very careful and smart about it. Especially if you combine several of them.

KenCochrane 6y ago

Maybe I missed something, but how do you handle the fact that your nginx server is a single point of failure? If that goes down, traffic can’t get to your web servers. Do you have more than one, and DNS load balance, or do you just live with the risk? One of the main reasons why I use an ALB/ELB is so that I don’t have that SPOF. If you found a way around that, please share, I would love to know, so I can save some money :)

root993 6y ago
I hate to admit it but the nginx server infact is a SPOF. We currently handle it by setting up alerts all over the place so I can take quick action if something goes sideways, but other than this I have not really found a way around it. We also have latest snapshots ready of all our instances so that I can get another server running ASAP during a calamity.
n_u_l_l 6y ago
His database also is. I think it's highly unprofessional to use a setup like this in production. Looking at his product, it seems like a product whose downtime has a big impact on their clients.

reledi 6y ago

Another approach to consider, that's so straightforward and quick that most will consider it boring and overlook it. - kill services with low usage - downgrade instance size - downgrade instance type - merge databases - schedule services with obvious usage patterns to shut down when not used - use EC2 spot instances Most importantly, it requires an aggressive culling mindset. If drastically reducing the AWS bill means staying afloat, then make bold choices.

elondaits 6y ago

One problem of VPS solutions is that they're easy to maintain until you have to do it. At some point the technical debt comes back and you need to upgrade the stack components and eventually the OS as well, without sacrificing uptime. ... if you did things well, it's just starting another instance and installing / copying things over with some minimum downtime. But if you don't have a 100% documented stack, you don't know which configurations you touched to make things work, you have files lying around, then you're probably going to pay back all your savings and more in the workdays needed to migrate. At my current employer we never have time for maintenance, and we don't have professional expertise (the "I worked exclusively at this for many years" kind, I mean) at webhosting, security, system admin or dba, so I heavily lean towards more "managed" and cloud solutions.

disgruntledphd2 6y ago
I completely agree (it's always the small changes that make things work that cause problems), but for this product which is greenfield and has a very small team, it seems like a good fit. Like, I wouldn't run Shopify like this, or anything really large, but it may get them to a better place right now. What I really liked about this post was the discussion of what they actually needed and how to achieve their goals. It's a tradeoff of time against money, and I wish more people documented their thought processes and approaches to this kind of stuff.
amelius 6y ago
But cloud solutions can change their UIs and APIs anytime they want. They can also drop features as they please.
- briffle 6y ago
  Has any major cloud provider ever done a major change like that without 6+ months of advanced notice?

m3kw9 6y ago

You said you basically use Shopify as CDN, what’s stopping everyone from just setting it up and abuse it?

root993 6y ago
Hello, author here. Like I mentioned in the post, there is no way to programmatically upload anything to the Shopify CDN and the CDN cache cannot be invalidated once you upload a file. If you want to update an existing resource, all you can do really is upload a new one. This ofcourse still does not completely prevent people from abusing it, but it does restrict usage to a large extent. There is also the issue of giving out a cdn.shopify.com link to your customers instead of something that has you company branding on it. This is not a problem for us because our customers do not have to manually add this snippet to their website and we do it via an API instead, so this link is not apparent to our customers

ykevinator 6y ago

You have a beautifully designed site, thanks for the write up

deleted 6y ago

[deleted]

darsoli 6y ago

For the mathematically disinclined (such as myself), the total savings are at least $229/mo .. not bad! Curious if you could use Cloudformation to provision these instances and setups? https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-cloudformation-stacks https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...

person_of_color 6y ago

Another day, another SaaS that I can’t believe has value.

mark_mart 6y ago

As a summary: They use AWS, see huge bills, then decide to not to use AWS services and switch to redis, nginx etc.

upgoingstar 6y ago

How do you go around the Data Transfer pricing? That area is just painful for me every single time I try to process some good amount of data.

peterwwillis 6y ago

You can shop for deals on VPSes at lowendbox.com. But if you're trying to run a business, this is a waste of time. Find a provider which is highly reliable, which can also automatically rebuild all failed infrastructure with no intervention needed. That eliminates 99.999% of the providers out there. You're paying a premium to never have to think about your tech again, so you can focus on the business. Besides using the free tier and other AWS services which are practically free at low uses, you can use cost effective options like Fargate Spot Instances and EC2 Reserved Instances. I highly recommend Fargate over running instances. Use Lambda with CloudWatch triggers if you need to schedule occasional jobs (or use Fargate's feature for that). Try to avoid heavy reliance on caches, ElastiCache is kind of a rip off. Move as much content to static as possible, use CloudFlare to reduce bandwidth costs. If you're gonna serve over S3, you might as well front it with CloudFront as it's actually cheaper due to caching at the edge, and also more reliable. ALBs are expensive but very useful for APIs as well as autoscaling (if you have to run instances, run them with an ASG, which also means having versioned AMIs)

aiibe 6y ago

Definitely off topic, are you using Twilio/Whatsapp at superlemon.xyz ? I wonder how do you cover the cost for unlimited outbound messages in your free plan... Thanks

jorangreef 6y ago

Do Hetzner or xneelo reflash the firmware on server hardware when they recycle them across users?

anthony_barker 6y ago

When I did data center stuff for a large company the biggest cost was network - not servers. Specifically network security and redundancy. Its amazing to me that this is all included for basically free except for Application level filtering by these providers.

EdwardDiego 6y ago
Yep, I was consulting for a firm with an already sizeable infrastructure investment/dependence on AWS, and their architects were proud of their best practice usage of multiple AZs for redundancy, but weren't know that inter-AZ traffic cost also. So I imagine they're going to have fun trying to incorporate that into their infrastructure layout. In my day job we saved $600 a day by cutting down on needless inter-AZ traffic.

sandGorgon 6y ago

The general rule of thumb to drop AWS bill is spot instance. Which is very hard to do unless you build your application to be tolerant of server kills.

tzs 6y ago

> An EC2 instance with 2 virtual cores, 4GB RAM and a storage of 80GB costs roughly 37$ a month and a Lightsail instance with the exact same configuration costs 20$ a month which is almost half the cost! In general I'm not sure you can really compare instances from different service by just look at cores, RAM, and disk. I used to have my small website and email server at Rackspace, on the smallest, cheapest instance. I ended up getting out of mail hosting (moved that to Fastmail), and putting the small website on Lightsail. The Lightsail instance and the old Rackspace instance have the same nominal specs for virtual cores, RAM, and disk. (Actually, Lightsail may be better on disk--I don't recall if the Rackspace one was SSD like Lightsail). The main thing the website actually does is host some graphs showing the temperature in my house. An ESP8266 temperature monitor I made uploads a sample once a minute. A script was running once a minute on the server that using gnuplot to make graphs of temperature over the last hour, 3 hours, 12 hours, 48 hours, and over all samples. At Rackspace it ran that fine, while at the same time gathering my mail from several services via fetchmail, receiving mail for my domain on its smtp server, and running spam filtering. On Lightsail just handling the website it was locking up every few hours. I managed to catch one pre-lockup and found the load average was something like 300. What was happening was sometimes that once a minute graph generation task was taking longer than a minute, and that would slow down the next one, and so on. Oddly, it didn't seem to be gnuplot that was taking too long, but rather the script that took the file containing all the samples, and extracted just the samples newer than a specific threshold. At least, running everything by hand that was the only step I ever saw take unusually long. I changed it to every 5 minutes to temporarily stop the frequent hanging, and then added a check to my script to skip regenerating the graphs if a previous instance of the script was still running to fix things permanently. I also changed it from storing the data in just a big file of samples sorted by time to an sqlite DB.

ravedave5 6y ago

A lot of this is penny wise pound foolish. Managed services cost more, but they provide more. RDS is just such a simple slam dunk for what you get for the cost. How much does it cost to spend 8-20 hours a week handling maintenance and upgrades and scaling for these services? I guess I'm bitter working on a project that was built with all sorts of crazy custom deployment stuff too conserve costs that's a total OPs nightmare. Moving from Redis on EC2s to elasticache was such an upgrade. We constantly had little issues here and there. Edit - Alsocontainerize your app from the start so that if it does take off you can slam it in ecs or k8s.

barnabask 6y ago

I'm interested in the managed DB part of this writeup, specifically that OP chose Lightsail. The last time I looked into it, Lightsail was MySQL only, so that was good to know. I wrote a PostgreSQL DBaaS Calculator that got some traction here a while ago, and I just updated it this evening to add Lightsail to see how it stacks up: https://barnabas.me/articles/postgres-dbaas.html#calculator https://barnabas.me/articles/postgres-dbaas.html#calculator No surprise, Lightsail is similar pricing to RDS with a 1-year commitment, but it's month-to-month. It's a pretty good deal until you need more than 8 GB memory, 240 GB storage, 2 cores ($115/month Standard plan). But Azure or AWS RDS are the ones to beat.