9 ms·
We saved $500k per year by rolling our own "S3"
- ch2026 1y ago[flagged]
- OsrsNeedsf2P 1y agoIt's the government who lost 850TB of citizen data with no backups[0] Because Cloud bad. [0] https://www.techradar.com/pro/security/the-south-korean-government-has-learnt-they-need-a-backup-the-hard-way https://www.techradar.com/pro/security/the-south-korean-gove...
- codedokode 1y agoStoring the data in a foreign cloud would allow foreign nation to play funny tricks on the country. What they need is not the cloud but sane backup system.
- PartiallyTyped 1y agoIsolated partitions exist.
- bpodgursky 1y agoap-northeast-2 is literally in Seoul
- sekh60 1y agoCLOUD Act.
- senectus1 1y agobecause they didnt have a decent backup.
- Havoc 1y agoTbh I feel this in one of those that would be significantly cleaner without serverless in first place. Sticking something with 2 second lifespan on disk to shoehorn it into aws serverless paradigm created problems and cost out of thin air here Good solution moving at least partially to a in memory solution though
- tcdent 1y agoYeah, so now you're basically running a heavy instance in order to get the network throughput and the RAM, but not really using that much CPU when you could probably handle the encode with the available headroom. Although the article lists TLS handshakes as being a significant source of CPU usage, I must be missing something because I don't see how that is anywhere near the top of the constraints of a system like this. Regardless, I enjoyed the article and I appreciate that people are still finding ways to build systems tailored to their workflows.
- inlined 1y agoMaybe they’re not using keepalives in their clients causing thousands of handshakes per second?
- ruperthair 1y agoYes, they mention this as a 'fix' for connection-related memory usage: > Disable keep-alive: close the connection immediately after each upload completes. Very odd idea.
- kevin_thibedeau 1y agoPossibly missing session resumption support compounding the problem.
- ahoka 1y agoTLS being a bottleneck when doing image processing is very-very weird.
- none2585 1y agoI'm curious how many engineers per year this costs to maintain
- UseofWeapons1 1y agoYes, that was my thought as well. Breakeven might be like 1 (give or take 2x)?
- hinkley 1y agoAnything worth doing needs three people. Even if they also are used for other things.
- codedokode 1y agoAnd I am curious how many engineer years it requires to port code to cloud services and deal with multiple issues you cannot even debug due to not having root privileges in the cloud. Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB. And no weird network issues to debug.
- rajamaka 1y ago> as simple as "with open(...) as f: f.write(data)" Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and... Without on-prem, saving a file is as simple as s3.put_object() !
- codedokode 1y agoWith s3, you cannot use ls, grep and other tools. > Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and... Wow that's a lot to learn before using s3... I wonder how much it costs in salaries. > With what network topology? You don't need to care about this when using SSDs/HDDs. > With what access policies? Whichever is defined in your code, no restrictions unlike in S3. No need to study complicated AWS documentation and navigate through multiple consoles (this also costs you salaries by the way). No risk of leaking files due to misconfigured cloud services. > With what backup strategy? Automatically backed up with rest of your server data, no need to spend time on this.
- DevelopingElk 1y agoVideo processing is one of those things that need caution when doing serverlessly. This solution makes sense, especially because S3s durability guarantees aren't needed.
- VladVladikoff 1y agoI’m mostly just impressed that some janky baby monitor has racked up server fees on this scale. Amazing example of absolutely horrible engineering. Also, just take an old phone from your drawer full of old phones, slap some free camera app on it, zip tie a car phone mount to the crib, and boom you have a free baby monitor.
- Huxley1 1y agoS3 certainly saves a lot of hassle, but in certain use cases, it really is prohibitively expensive. Has anyone tried self-hosted alternatives like MinIO or SeaweedFS? Or taken even more radical approaches? How do you balance between stability, maintenance overhead, and cost savings?
- ddxv 1y agoMinIO has moved away from having a free community fork, and I think it's base cost is close to $100k a year. I've been using Garage and been happy, but as a single dev and orders of magnitude smaller than the OP, so there are certainly edge cases I'm missing to compare the two.
- Cerium 1y agoI'm a fellow new Garage user. I have had a great time so far - but I also don't need much. My use case is to share data analysis results with a small team. I wanted something simple to manage that can provide an s3 like interface to work with off the shelf data analysis tools.
- Lucian6 1y ago[flagged]
- YZF 1y agoRight. Having worked on a commercial S3 compatible storage I can tell y'all that there's a lot more to it then just sticking some files on JBOD. It does depend on your specific requirements though. 1.5 FTE over 18 months sounds on the low side for everything you've described. That said the article seems to be more about an optimization of their pipeline to reduce the S3 usage by holding some objects in memory instead. That's very different than trying to build your own object store to replace S3.
- supriyo-biswas 1y agoWhy do all your comments seem LLM generated? You do clearly have something to contribute, but it’s probably better to just write what you’re talking about than going through a LLM.
- pjjpo 1y agoI don't know about the commenter specifically but in general, using LLMs to format text is a game changer in the ability for English-as-Second-Language folks to contribute to tech conversations. While I get where some of the bias against anything LLM generated comes from, I would keep it for editorial content and not community comments to be fair to a global audience.
- ashdksnndck 1y agoI’m worried that LLMs could facilitate cheap, scaled astroturfing. I understand that people encounter discrimination based on English skill, and it makes sense that people will use LLMs to help with that, especially in a professional context. On the other hand, I’d instinctively be more trusting of the authenticity of a comment with some language errors than one that reads like it was generated by ChatGPT.
- phito 1y agoIt just makes everything sound bland and soulless. You don't know which part of the message actually comes from the user's brain and which part has been added/suggested by the LLM. The latter is not an original thought and it would be disingenuous to include it, but people do because it makes them look smarter. Meanwhile, on the other side, you might as well be talking to a LLM...
- dxxvi 1y agoSo, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified? Have you ever thought of using a postgresql db (also on aws) to store those files and use CDC to publish messages about those files to a kafka topic? In your original way, we need 3 aws services: s3, lambda and sqs. With this way, we need 2: postgresql and kafka. I'm not sure how well this method works though :-)
- jrochkind1 1y agoLike put the video blobs themselves in postgres data columns? Does putting very large (relative to what you normally put in postgres) files in pg work well? Genuine question, i do not know, I've been considering it too and hesitant about it.
- dxxvi 1y agoI already checked with AI before putting the comment :-) 1GB with the bytea data type (https://www.postgresql.org/docs/current/datatype-binary.html https://www.postgresql.org/docs/current/datatype-binary.html) and 4TB with the BLOB data type (https://wiki.postgresql.org/wiki/BinaryFilesInDB https://wiki.postgresql.org/wiki/BinaryFilesInDB).
- jrochkind1 1y agoHave you done this? I can google or AI for the max size that postgres will allow, sure. I have googled in the past for whether this actually works well, and have gotten answers leaning towards most advice against it in real world scenarios. So if you have experience with this and it did work well, I'm curious to hear about it! That's why i asked about if it worked well, not about the maximum size postgres allowed in various data types. If you have no experience with it, but are just posting advice based on what AI tells you about max sizes of data allowed by pg that I can get from the same source too, then okay, fair enough, and certainly no need to give me any more of that!
- dxxvi 1y ago> I've been considering it too and hesitant about it Why hesitant? Just ask AI. It'll tell you how to do it and then you can experiment it yourself.
- ixtli 1y agoThey didn’t actually do what the headline claims. They made a memory cache which sits in front of S3 for the happy path. Cool but not nearly rolling your own S3
- ramraj07 1y agoI also didnt get why the cache had to be memory and not local name storage..
- jonasdegendt 1y agoOut of curiosity, why? My first thought is, why bother with local storage if your turnaround on video chunks is 2 seconds? What's disk going to add besides a little bit more resiliency in that 2 second time frame? This at the cost of having slower pod startups given you have to mount the PVC, and a small performance hit of writing to a filesystem instead of memory. All moot anyway given that the cameras/proxy allegedly has retries built-in, but interested to hear your thoughts.
- ghm2180 1y ago> What's disk going to add besides a little bit more resiliency Resiliency is the point. How would you protect against machine's loss/crash?
- ramraj07 1y agoreduced cost? you can have same storage in nvme (and for this purpose, sufficient performance) at a tenth or hundredth the cost of the same storage in memory in AWS.
- jrochkind1 1y agoTrue, but, if i'm following, the memory cache has to clone S3 API for existing clients that can't be easily altered. Regardless of what you title it, it's an interesting project report!
- varenc 1y agoIn HN style, I'm going to diverge from the content and rant about the company: Nanit needs this storage because they run cloud based baby cameras. Every Nanit user is uploading video and audio of their home/baby live to Nanit without any E2EE. It's a hot mic sending anything you say near it to the cloud. Their hardware essentially requires a subscription to use, even though it costs $200/camera. You must spend an additional $200 on a Nanit floor stand if you want sleep tracking. This is purely a software limitation since there's plenty of other ways to get an overhead camera mount. (I'm curious how they even detect if you're using the stand since it's just a USB-C cable. Maybe etags?) Of course Nanit is a popular and successful product that many parents swear by. It just pains me to see cloud based in-home audio/video storage being so normalized. Self-hosted video isn't that hard but no one makes a baby-monitor centric solution. I'm sure the cloud based video storage model will continue to be popular because it's easy, but also because it helps justifies a recurring subscription. edit: just noticed an irony in my comment. I'm ranting about Nanit locking users into their 3rd party cloud video storage, and the article is about Nanit's engineering team moving off a 3rd party (S3) and self-hosting their own storage. Props to them for getting off S3.
- jen20 1y agoThis is the reason I refused to buy Nanit cameras, instead opting for unconnected models. E2E encryption is table stakes.
- hshdhdhehd 1y agoBy the way you dont need a video (or hell even audio) baby monitor. Source: 2 kids.
- NetOpWibby 1y agoSame here. I wonder if the market is for first-time parents and people who work 8+ hour days.
- wltr 1y ago
- lpa22 1y agoIf anyone here uses the Nanit app in the background of their phones, it absolutely destroys battery life. I got a new phone because I thought my battery was cooked, but turns out it was just the app.
- mring33621 1y agowhile (true) { Thread.sleep(300L); checkIsSubscriptionCurrent(); }, probably
- another_twist 1y agoI mean the "S3" could be replaced with object storage. I guess thats the technical term anyway. Having said that just goes to show how cheap S3 is, if after all of this, the savings are just $500k. Definitely money saved but not a lot.
- OrangeDelonge 1y agoCouldn’t they have used S3 express one zone?
- gnarlouse 1y agobecause "How we stopped putting your kids in S3 buckets" just sounded less attractive
- szszrk 1y agoThey can't say that, as they did not stop. They made a cache in front of it.
- NetOpWibby 1y agoCloudflare R2 solves this
- bryanrasmussen 1y agothey don't seem to have factored in the cost of doing this, so not sure what their actual saving was although it was probably substantial.
- anarsdk 1y agoSounds like the title should have been > We used S3 even though it wasn’t the right service
- Gasp0de 1y agoExactly, my first thought was "Why in earth would anyone think that S3 was the right service to store millions of tiny ephemeral files?" and now it seems they have invented their own in-memory store instead of just using something like Redis. I also wonder what happens if their DIY thingy crashes, are the videos lost? Why not send to Kinesis or SQS in the first place?
- alex_smart 1y agoFrom the article, individual video segments were 2-6 MB in size and SQS and Kinesis have a 1MB limit for individual records so they couldn’t have used either service directly. At least not without breaking their segments into even smaller chunks.
- Gasp0de 1y agoYou're right, I didn't pay attention there. Still seems that there a many solutions better suited than S3. Probably a classic case of "We need an MVP fast, let's optimize later".
- 0x6c6f6c 1y agoAgreed, but this isn't always bad. Optimizing early with unclear requirements can kill time, which at an early stage is just delaying product launch. Migrate/optimize later when you're actually reaching scale is a perfectly reasonable approach. In fact, if you have a decent abstraction or standard in place (e.g. S3 API or repository pattern) you can swap it out in place.
- dmje 1y agoI’m sufficiently old / sensible (you decide) to think that uploading video of your baby (to anywhere) is fucking weird and fucking spooky and not needed anyway. This is a solution that doesn’t have a problem. Worse: it prays on parental / young parental fears. There’s nothing here - this is not a product that’s needed. You don’t need to “track” your baby, ffs. You don’t need to watch it while it sleeps. You don’t need “every breath, seen”. People have been having babies for fucking centuries without entering them into this hyper weird surveillance state at birth. What an appalling screwed up world we seem to have manufactured for ourselves.
- ramraj07 1y agoOf all the hills to die on w.r.t. how the world is screwed up, this seems like the silliest. Different folks parent differently, culture evolves. You're free to have your "old school" thoughts as are people who use services like this. Its not like they're publishing it in public. The service in discussion especially just stores it in the server only temporarily to use ML to detect things such as sleeping or crying. Sounds innocuous to me. Many of us can do the math ourselves and choose to make choices based on our own beliefs. That's true freedom.
- 4gotunameagain 1y ago> Different folks parent differently, culture evolves. You are framing it as if every change is in a positive direction, which it clearly isn't. Risking at sounding like an old man yelling at clouds, look at the kids these days. They are so dependent, and so sensitive to negative stimuli or emotions. Parents need to realise that they need to grow adults, not perpetual kids.
- Capricorn2481 1y ago> look at the kids these days. They are so dependent, and so sensitive to negative stimuli or emotions. I think many of us have found people writing comments like this are not interacting with children very much. More just reading the takes of others who also don't interact with children. And if this was a legitimate problem to address, you would not address it by taking away baby monitors.
- anshumankmr 1y agoSome stuff like this also exists: https://www.dell.com/en-in/shop/storage-servers-and-networking-for-business/sf/objectscale https://www.dell.com/en-in/shop/storage-servers-and-networki... We could just use something like that Or there is that other Object storage solution called R1 from Cloudflare.
- tauntz 1y ago* R2
- freak42 1y agoThey saved $500k on what total sum? $500'001 or 55'000'000? Without this info the post is moot.
- bdcravens 1y agoI suspect it's a massive amount, as S3 is one of the cheaper services. As we evaluate moving all of our compute off of AWS, S3 (and SQS) are probably services we'll retain because they are still amazing values.
- coleca 1y agoThat's a great point. Sometimes we look for architecture or technology solutions for a problem that could be easily solved at the sales level by negotiating a PPA (Private Pricing Addendum) with AWS.
- 0xbadcafebee 1y agoTheir architecture is internet bandwidth heavy and storage heavy; these are some of the most expensive things in AWS. You probably want to use a different provider for those things. > It turns out that when AWS says an instance can do “Up to 12.5 Gbps”, that’s burstable networking backed by credits; when you’re below the baseline, you accrue credits and can burst for short periods. Yes, AWS has a burst rating and a sustained/baseline rating for both EBS types as well as instance types. Use https://instances.vantage.sh/ https://instances.vantage.sh/ (and make sure you choose specific columns) to compare specific criteria and then export as a CSV to find the lowest price that matches your performance/feature/platform criteria. Design to the baseline if you need guaranteed performance. Do sustained performance testing. > When we Terminated connections idle >10 minutes, memory dropped by ~1 GB immediately; confirming the leak was from dangling sockets. Fix: make sockets short-lived and enforce time limits. We used to do that with Apache 20 years ago. Config option forces a forked subchild to exit after N requests to avoid the inevitable memory leaks. AKA the Windows 95 garbage collector (a reboot a day keeps the slowness at bay). FWIW, if the business feasibility of your architecture depends on custom stuff, performance enhancements, etc, you will find that you eventually have harder and harder problems to solve to keep your business functioning. It's more reliable to waste money on a solution that is brainless, than invest human ingenuity/technical mastery in a solution that is frugal.
- dilyevsky 1y agoThey're very ingest heavy compared to how much of it is actually streamed out and to a very small/local audience so probably don't even need a cdn. And ingest on aws is free. On the other hand, S3 is kind of ridiculously expensive compared to even more expensive on-prem options like a PureStorage SSDs array. With spindles on Ceph you can probably get quite a bit lower than AWS's 2c/Gig/mo. Or you can just use R2 with colocated servers for ingest.
- danjc 1y agoI'd it's processed in 2 seconds, why not just process it immediately in memory?
- swiftcoder 1y agoBecause they are serverless, so there's currently no memory for it to be processed in at the point of upload
- liqilin1567 1y agoMaybe there are too many requests, so they have to offload the videos to s3.
- dboreham 1y agoThat's not the reason. And furthermore any buffer/re-try mechanism should be done at the edge (on the camera).
- swiftcoder 1y agoThis feels like they were using the wrong architecture from the start, and are now papering over that problem with additional layers of cache. The only practical reason to put a video in S3 for an average of 2 seconds is to provide additional redundancy, and replacing that with a cache removes most of the redundancy. Feels like if you uploaded these to an actual server, the server could process them on upload, and you could eliminate S3, the queue in SQS, and the lambdas all in one fell swoop...
- _el1s7 1y agoYes, it's simple, S3 is for storing objects, not for processing. Don't know how they came up with such a bad and complicated cloud design for something that is straight forward.
- Brian_K_White 1y agounnecessary cloud subscription service abjures unnecessary cloud subscription service
- TimCock 1y ago[flagged]
- gethly 1y agoI made my own S3 as well. I used two S3-compatible services before but there was always some issue(first one failed to upload certain file, no matter what and support was unhelpful; second one did not migrate with file metadata properly so i knew this would be ongoing problem). In the end, it is just a dumb file store, nothing else. All you need to do is to write a basic HTTPS API layer and some logic to handle database for the file metadata and possibly location. That is about it. Takes a few days with testing. But then you also have to think about file uploads and file downloads. You cannot have a single server fulfilling all the roles, otherwise you have a bottleneck. So this file storage became a private backend service that end-users never access directly. I have added upload services, whose sole purpose is to allow users to upload files and only then upload them to this central file store, essentially creating a distributed file upload queue(there is also a bit more logic regarding file id creation and validation). Secondly, own CDN was needed for downloads. But only because I use custom access handling and could not have used any of the commercial services(though they do support access via tokens, it just was not working for me). This was tricky because I wanted for the nodes to distribute files between themselves and not always fetch them from the origin to avoid network costs on the origin server. So they had to find each other, talk to each other and know who has which file. In short, rolling your own is not as hard as it might seem and should be preferable. Maybe to save time, use cloud at the beginning, but once you are up and running and your business idea is validated by having customer, immediately move to your own infra in order to avoid astronomical costs of cloud services. btw, i also do video processing like mentioned in the blog post :)
- miroljub 1y agoAh, these modern babies. They can't even sleep without being spied and recorded 24/7. I'm glad both me and my kind grew up in different times. Today's kids will never in their lives know what freedom is, and we are guilty we made such dystopian societies a reality.
- hk1337 1y agoI have always understood S3 is just HDFS with some extra features? So, if you were going to roll your own S3, then you’d stand up an HDFS cluster.
- ruperthair 1y agoThis may be an obvious point, but I didn't see it mentioned in the (otherwise excellent) article: I would have been interested in the cost saving in just implementing the 'delete on read' with S3 that they ended up using with the home-made in-memory cache solution. I can't see this on the S3 billing page, but if the usage is billed per-second, as with some other AWS services, then the savings may be significant. The solution they document also matches the S3 'reduced redundancy' storage option, so I hope they had this enabled from day one.
- JCM9 1y agoThe article strikes me as a self congratulatory solution to solving a problem that they could just have avoided entirely by instead selling hardware with local video storage. Lots of options for doing that efficiently and inexpensively in 2025. Hosting everything in the cloud like this is a 2015-era solution.
- ghm2180 1y agoActually why(just) RAM? Why not have an append only storage to the local disk? WALs are quite fast.
- fergie 1y agoI mean fair enough, but I feel like S3 is one of the few AWS products that is actually pretty cheap for what you get.
- elmigranto 1y agoClassic case of "focus on building your app, not infrastructure". Here's another multi-million dollar idea: put this cache directly inside your own video processing server and upload there.
- jrochkind1 1y agoWhat a great and helpful write-up, love when people share things like this so I can learn. It's less about whether I would have a use case for this exact thing (or whether or not it was appropriate for this use case, i dunno, prob don't have enough context to know). More just seeing what is possible, how they thought about it and analyzed it, what they found unexpected and how, etc. I learned a lot!
- kingnothing 1y agoNanit is horrible spyware. Do not buy their products. If you have a router that lets you inspect data flowing out, you'll be astonished at what your little Nanit cam exfiltrates from your home network. Even if you don't pay for their subscription service, they still attempt to exfil all of the video footage caught on your camera to their servers. You can block it and it will still work, but you shouldn't have to do that in the first place if you don't pay for their cloud service. Stay away if you value your privacy.