14 ms·
How much I’ve spent so far running my own Mastodon server on AWS
- hiddencost 4y agoDigital ocean has a premade droplet that is costing me $20 / mo.
- hectormalot 4y agoLike OP, I keep being surprised by how easy it is to drive up bills with Cloudwatch. It’s small numbers here, but I’ve seen this happen at 4-5 digits as well.
- acjohnson55 4y agoThis is great to know. I've been interested in self hosting my own single user instance and it's been hard to understand the all-in cost.
- genewitch 4y agoI run two small fediverse instances, a misskey, and a pixelfed. both have fewer than 10 users, and the federated timeline only "clicks" once every few minutes. I am on several "medium" instances of ~10,000 users, and the admins that actually do the work on the infra are so tight-lipped - this bugs me. You come to me and ask "how much does it cost to do X", and i know? I'll tell you. I'm a CC0 / public domain kind of person. I don't get the secrecy. Look, how many full cores/threads and how much ram/cache, and how much storage/io, and how much ingress and egress. Sure, those you can obfuscate a bit, but can i run a 10k user instance on 2 dell rackmount machines from 2014? on a single Ryzen? A single Epyc? a half dozen raspberry pi? the chances of being able to parry "i run a php webservice" into retirement levels of money is nil, so i don't even understand the premise! To answer your question more directly: I run misskey on a 4 thread, 8GB mem, 80GB storage VM, where i interact directly with the hypervisor (proxmox). my cost is nothing, but i guess this would be maybe $10/month? I'm not including the monthly cost to host the hyper, the /24, etc; we used to own the metal, but my partner decided there are things more important than owning the metal, right now. The upside is, when i take a snapshot, that isn't managed by disks that i can physically touch. Makes them slightly more reliable to me. Other people may have weirdness about this. of the 4 "cores" the daily average is about 3% CPU usage, with monthly spikes up to 12% average usage. as far as bandwidth, the daily average is under 9kb/s and the monthly averages are under 22kb/s. a small fediverse instance with a single user is likely completely doable with a co-lo raspberry pi!
- acjohnson55 4y ago> a small fediverse instance with a single user is likely completely doable with a co-lo raspberry pi! You buried the lede:) That's great to know. Maybe not worth it to cloud host. Are there any issues to worry about with having downtime of my own node, other than my own ability to post and read? I haven't read enough about ActivityPub to understand the impacts of a server being down on the protocol.
- genewitch 4y agoyes, i realized after i typed up my complaints about fedi admins that i did the same thing. Mastodon is php, and as such it requires php-level stuff. Misskey is nodejs (i think) and python, so it depends on if you're used to wordpress-y stuff or not. I prefer misskey because it has a "drive" - you can upload files and hotlink, or make "tweets" with them. thankfully, activitypub and Ostatus "protocols" are open source, so one could write a server implementation in whatever language du jour makes them happy, but only a few people have. I see a lot of the same issues with realtime applications like matrix (for chat) and audio/video streaming. Some php implementation (or python), and then no one else can make something that adheres to the standard. Whether that's the standard's fault or not isn't for me to say. if i get a jpeg image file, and the joint picture expert group's spec for jpeg format, i can write something to view that in a day or so. same with whatever RFC IRC was. or http. or ftp. try writing a toy matrix or fediverse implementation? meh.
- notacoward 4y ago> the admins that actually do the work on the infra are so tight-lipped Not always. @kris-nova here is the public face of hachyderm.io and has done an excellent job communicating around architecture, incidents, etc. Everyone on that team deserves kudos.
- rsynnott 4y agoMasto.host will charge you $6/month for that, though I think they’ve paused signups for now. If you want to do it yourself you can probably manage that sort of pricing using ovh or similar.
- invalidname 4y agoI would bet the cost would be lower if you use Digital Ocean/Linode and CloudFlare. Both have 5USD instances with no egress costs at these scales. No need to commit for three years or anything. If you already own a domain you can use a subdomain and cut even that cost although it might make up for a long URL. I think the main cost here is time. Probably far more expensive than all of those costs. Even on AWS.
- x86hacker1010 4y agoCame to say this. I think I see a lot of people default to AWS, maybe because their employer uses them and it's what they know. The downside is that AWS has all this expense for fancy stuff that can be done usually with a cheap instance, docker and nginx. - Cloudflare for caching, DNS, domains. - Hetzner, Linode or DO for a cheap instance See what does/doesn't work. Storagebox on Hetzner is cheaper storage hosting and you can have redundant backups for cheap on backblaze as well or offshore cheap VPS doing encrypted backups with rclone/rsync.
- f6v 4y ago> I think I see a lot of people default to AWS, maybe because their employer uses them and it's what they know I don’t get why people assume others are just mindless fruit flies in these threads. Maybe OP just doesn’t want to fiddle with technology and pays Amazon to simply get it done. That’s what I’d do, there’s no need to turn every project into excercise in orchestrating open-source software.
- wkat4242 4y agoAWS is wayyy too complicated to "simply get it done" for a single user. It makes no sense to learn all that if you just do it for yourself. In fact I have to do a bit of AWS for work and I hate it with a passion. It's spread over a ton of services each with their own console (that look & feel like they were designed by completely different people), lots of overlap between services, complex billing. Configuration often needs to be done with complex json files. Ugh. Perhaps it makes sense for a major enterprise where you have orchestration for everything. But I would never pick this total mess for a personal project. Azure has a bad name but at least everything is integrated in the same console pretty nicely. But really if you just need an instance of something for personal use, DigitalOcean, Scaleway etc are so much simpler. And cheaper, and most stuff is just included in the monthly price so you don't get billing surprises.
- panarky 4y agoWith a single-user instance and a relay, traffic is 99% inbound, and the biggest cost will be storage. But once there are a few hundred users, Sidekiq jobs will use a lot of compute and egress data will add up. Might be better in this case to use a provider with generous free bandwidth.
- birdymcbird 4y agoAWS costs can destroy you, unless you’re using managed AWS services to replace bloated IT org full of support personnel not doing much productive work in first place. If you really need scale, AWS can be cheaper than rolling your own infra and abstractions, granted you make it a mission to keep laser focus on AWS costs. Primary ideal use case for AWS is internal Amazon usage. Charge internally but write off as expenses, for infra you had sitting idle anyway. Pretty sure AWS and rest of Amazon are separate legal entities to take advantage of tax laws.
- ReflectedImage 4y agoI'm not so sure about the cheaper at scale. Renting is usually more expensive than owning. The larger you get, the more expensive renting is compared to owning.
- lokedhs 4y agoThe best use cause is a system where you have vast spikes in compute requirements on rare occasions. You can then use AWS to get those resources for limited period of time. For this user case, it actually works pretty well. For anything else, there are always going to be cheaper and better solutions.
- CuriousCosmic 4y agoAWS fits the niche of "holy fuck my site is blowing up and I don't want it to crumble while I onboard new users or some event is going on". You probably shouldn't use it for day to day ops but it's a really good spillway for extremely temporary & particularly for unexpected spikes of traffic. Of course you could argue that AWS is particularly bad but I think this applies to most non-bare-metal instance cloud services.
- Marsymars 4y ago> AWS costs can destroy you, unless you’re using managed AWS services to replace bloated IT org full of support personnel not doing much productive work in first place. OTOH, if you have an understaffed infra team that gets by by accumulating technical debt on infrastructure, AWS takes away that option at several levels of abstraction.
- robga 4y agoThis is a well written post that explains how to tune an AWS single user instance deployment, and the costs that may be expected. It makes me feel that I’m getting rather good value at masto.host for my personal instance (I was on a waitlist for a few weeks). What mastodon is missing is a good catalogue of reference deployments for 1, 10, 100, 1000, users on the major platforms (AWS, GCP, Azure, DO, OVH, Hetzner). Of course, “users” is a rather poor proxy for “activity” but it’s a start. There are “scaling” articles that say “the first thing you’ll need to address is…” and breathless blogs of “what we did when we doubled registered users overnight”. But not so much if you’re starting an instance for a community that you know will be a certain size quickly. How many cores should I expect sidekiq to need to service each thousand typical users? Sure, it’s a piece of cake for sysadmins or an AWS solutions architect, but it makes me wonder how many fragile instances there are out there that have grown fast but could also fail fast.
- user5678 4y ago[dead]
- windexh8er 4y ago> Sure, it’s a piece of cake for sysadmins or an AWS solutions architect, but it makes me wonder how many fragile instances there are out there that have grown fast but could also fail fast. The way I read the post showcased how easy it was for the AWS SA to make insane billing mistakes with unneeded things like CloudWatch. They also hide the cost of EC2 because there's no charge for that instance type until 2023. This isn't a positive advertisement for AWS in my mind. This is in line with experience I've had with former AWS consumers I've converted over to DigitalOcean in 2022. I saved those conversions, on average, 70% of their prior AWS bill - with no loss of functionality. AWS has spun out of control with the nickel and dime aspect to billing. I can't say I'd recommend it to small or very large clients unless you have money to burn. At the end of the day "traditional" architectures on AWS are not cost effective without excessive oversight of choices.
- solatic 4y agoHe talks about using an RI to reduce the monthly bill to under $5/month for compute. But in general, I think you're right. The value AWS offers for traditional architectures is for orgs running the numbers on lift-and-shift, compared to running on-prem infrastructure; greenfield projects should take advantage of cloud-native patterns.
- jiggawatts 4y agoIt cracks me up that both AWS and especially Azure overcharge for logging to the point that it’s easy for logs to cost more than the underlying system they’re monitoring! If you sit down and work out their cost of data processing, it costs them cents to provide a service for which they charge tens to hundreds of dollars. I figure the reasoning is that large enterprise requires logging for compliance, and it’s a hidden charge that’s easy to overlook in a bill.
- somehnacct3757 4y agoI also got a chuckle because the author is a professional AWS architect and still couldn't avoid accidental charges. Well, architect isn't salesman! Thankfully the mistake was corrected by paying the bill in full.
- yegle 4y agoAFAIKT the AWS SA takes commission, and should indeed be considered a sales role.
- Moissanite 4y agoIt is a sales role (part of the sales org), but not paid comission - that is only account managers.
- berkle4455 4y agoThere really needs to be a better layer than vanilla S3. PUT costs are absurdly high and it’s a terrible filesystem.
- dragonwriter 4y agoEFS, while much more expensive for storage, might win out if request costs are the problem.
- 15155 4y agoCloudflare R2 is at least cheaper.
- DoctorOW 4y agoR2 is incomplete S3 API at the moment. It is incompatible with Mastodon.
- hkt 4y agoFor the avid self hoster, SeaweedFS: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- pyuser583 4y agoThe instance I’m on is spending over $1000/month.
- rsynnott 4y agoA lot of instances have had to scale up rapidly lately, which doesn’t help. Steady state cost will likely be lower.
- ihateolives 4y agoHow many active users do you have?
- unshavedyak 4y agoMine (Mastodon.art) published their financials on Patreon. Iirc a couple months ago we were in the 15k user range, for ~$700/m.
- hbogert 4y agoGet fiber and host this from your own home. When i see these absurd costs I'm happy with the k8s I built, with ceph for storage.
- ryanisnan 4y agoLearn from the mistakes of others, do not do this. Remember Yevgeniy Nikulin?
- KomoD 4y agoWhat relevance does Yevgeniy Nikulin have with this, whatsoever?
- ryanisnan 4y agoA lot, actually. He compromised a home-ran webserver on a VM, and compromised his way to a networked work machine. Running exposed ports from your house, no matter how smart you think you are, is not a good idea.
- KomoD 4y agoThanks for the explainer.
- hbogert 4y agoWhat's fundamentally different if you host this in the cloud but still have a public port? What was the exact attack vector in the case of nyevgeny?
- ryanisnan 4y agoThe difference is it would be likely inconsequential if a the cloud resource running this was compromised. There’s no direct link (at least there shouldn’t be) to a work machine, which might have credentials or itself VPN information for an employer. Yevgeny did exactly this. He compromised a web server running Apache or something, used that to brute force his way into a networked work machine, which took weeks, but went undetected. That machine turned out to be the physical host the VM running the web server was on. Then he used the work machine to break into LinkedIn using credentials and VPN profiles found on the iMac. This vector would have been impossible had that open server not been running so close to the work machine.
- abdullahkhalids 4y agoI host a Canvas LMS on Hetzner to run online courses. We sometimes have hundreds of students simultaneously giving quizzes and exams on the LMS. I think we pay 16.81 Euros/month. I am fairly sure a mastodon server should run on a server with half the stats and price.
- georgyo 4y agoThis is well written, but is only 30 days worth of data. $20/month, not including compute, is their starting point on AWS. As they get more followers and follow more people that will continue to grow. My biggest suggestion is to disable the relay. I took experimented with using a public relay, but the stream of content it adds is very low value. Better to create a second account on the instance that is more aggressive about following people your only tangentially interested in. This will make your federated feed interesting, but keep your main feed more focused. To the people suggesting DO/Linode. I'm a big supporter of Linode, however article has no egress or compute costs. Likely because it's a single user instance running on the free tier. Their costs are almost all storage related.
- layoric 4y agoA good compromise I’ve found is using something like “FakeRelay”[0] with “GetMoarFediverse”[1] to pull in specific tags from specific larger servers. This acts as a way to get more interesting content in your search and other mastodon tabs while benefiting from the moderation efforts of larger servers. Always good to ask the admins of those servers before you do incase they are already at capacity and don’t want the extra load, and contribute to their patreon if they have one. [0] https://github.com/g3rv4/FakeRelay https://github.com/g3rv4/FakeRelay [1] https://github.com/g3rv4/GetMoarFediverse https://github.com/g3rv4/GetMoarFediverse
- bayesian_horse 4y agoIs it true that most posts on self-hosted Mastodon are actually about self-hosting Mastodon?
- Legogris 4y agoNo, not by a stretch.
- pimeys 4y agoI run an Akkoma server for a few friends in Hetzner: - Cloudflare DNS - 2x CPU 2GB RAM application server with snapshots - 2x CPU 2GB RAM postgres server with automated backups - 2x CPU 8GB RAM elastic search server to provide a full-text search for the known Fediverse The search is not mandatory, but super nice to have. It is about 20€ a month, not counting the domain costs due to having it already for other purposes. Akkoma is written in elixir and is much better in handling async messaging. No need for Sidekiq or Redis. My CPU usage hovers around 6-12% and memory around 55%. Postgres uses about the same. I've never tried Mastodon, but having been a Ruby developer in my past life, I needed to find something else to install when I joined the Fediverse. Couldn't be happier with my setup. It just works, I never really need to do any maintenance. All the normal clients work with it. https://akkoma.social/ https://akkoma.social/
- aprdm 4y agoCan Akkoma talk with Mastodon? Otherwise I think I miss the point of your post
- bongobingo1 4y ago> Communicate with all your friends on the fediverse, no matter if they're on mastodon, pleroma, or any other fediverse software. Probably? (Maybe it explodes if you try.)
- resonious 4y agoIt says it implements ActivityPub on the site so yes it should talk to Mastodon.
- pimeys 4y agoTalks with Mastodon, Pleroma, PeerHub, Pixelfed, Misskey, Friendica and every other software using the ActivityPub protocol.
- zimpenfish 4y ago> every other software using the ActivityPub protocol. Even GotoSocial now that they've put special handling in to work around their bizarre key-handling design decision.
- rsolva 4y agoMastodon offers a lot of great features but at a high computational cost. I settled for GoToSocial for my personal instance, and although it is still in alpha, everything I need is there and it is stable and barely uses any resources. The setup and maintenance/upgrades is simple since there is very few moving parts, which makes it perfect to host on my home server. I tried, and gave up, setting up Mastodon several times in the past as I got choice paralysis from all the different ways to set it up. That was a few years ago, but from what I hear, it is still a somewhat daunting task.
- cuu508 4y ago>The hidden cost of storing data for Mastodon on S3 is the Requests. There are so many requests! I’m seeing about 40-50K requests per day! There are S3-compatible object storage providers that have no per-request fees, and significantly lower bandwidth and storage fees. Examples – OVH, Scaleway. I had an use case for object storage where my service would be uploading tens to hundreds of small objects per second, 24-7. Running some napkin math, it looked like AWS S3 costs for me would have been in the hundreds. I went with OVH, and have a €1 monthly bill for the object storage :-)
- aprdm 4y agoThat's really cool! Seems like as the comment on your blog said, you could try to template it and perhaps offer Mastodon as a service on AWS? :) Thanks for sharing.
- baeaz 4y agoYou have to have some big balls to run a Mastodon server on AWS. You are just one pissed off guy away from a bill that will cost you dearly.
- nirvgorilla 4y ago[flagged]
- xboxnolifes 4y agoMy upload speed hasn't increased since the 2000s.
- nirvgorilla 4y agobucks out teeth I bet it's still fast enough for whatever you want to do. Don't even bullcrap me because I WILL get you in voice and have you share your screen and prove you need to pay for stupid hosting.
- xboxnolifes 4y ago5mbps is not fast enough past a trivial usage without degraded experience for the server or personal use cases. That's not even taking into account DDoS risk for my personal IP. Been there, done that.
- nirvgorilla 4y agoThere's the ddos bullcrap excuse. These last two generations are SO eager to spend money for stupid crap. Pay your ISP and put your wallet away. That's the call to action on the internet.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- raffraffraff 4y agoOracle Cloud "always free tier" has 24G of memory, 4 ARM CPUs, 200gb block storage, 200g object storage, a load balancer and some database stuff (may or may not be useful). I've been running a 4x node Kubernetes cluster for over a year, mostly for fooling around with.
- death916 4y agoI was able to get a free x86 box but it's been impossible for me to get an arm instance on the west coast always out of space. Otherwise oracle free tier is pretty ncie
- raffraffraff 4y agoI had the same issue after shutting down my instances. I wrote a simple bash loop to recreate them and left it running overnight. All instances were up by the morning. I haven't shit any down since then. Reboots are fine.
- unity1001 4y agoHost at Hetzner or DO. AWS costs are outrageous.
- notacoward 4y agoI love Mastodon, or more accurately the fediverse in general. However, running your own instance seems like a loss in every possible way. It costs money, even if it's not much. It's work, even if it's not much. It incurs risk and liability. And it's functionally worse than being on a shared instance. Larger instances will take forever to federate with you, if they do at all. Your local timeline will be empty. You'll even have to jump through hoops to see all of the replies to posts, since the default is to show only replies from instances you've already federated with. If you often feel like you're only seeing half the conversation it's because you really are. Lastly, the more people run their own instances, the worse everyone's experience will get. The fediverse almost entirely relies on direct communication between instances with no DHT or other mesh routing overlay, so messages are O(n^2) relative to number of instances. Caching works OK for assets but not for ActivityPub RPC, so many instances don't even implement it at all. One of the main reasons the large instances struggle so much is that these communication patterns drastically increase traffic volume and queue length, which is further exacerbated by the fact that Sidekiq (the part of Mastodon that handles those queues) is a poorly implemented tuning nightmare. The best bet IMO is to join one of the medium-sized instances aligned around a community or vibe that you like (I'm on hachyderm BTW and it's awesome for me). Smaller ones have all of the problems I've mentioned above, while the very largest are sub-optimal in terms of both technical UX and social atmosphere. Goldilocks wins again.
- unshavedyak 4y agoYea this has been bothering me. I have to fight to ignore the desire to run my own and just enjoy my medium-sized instance.. because any interest i get in Mastodon itself makes me want to rewrite it to run more lean and allow users to contribute costs (P2P data). I quite enjoy Mastodon, but the whole thing feels.. incorrectly aligned with technical needs. I hope the attention it's getting leads to core implementations more aligned with the community.
- barbariangrunge 4y agoI find my aws billing statements to be so confusing that I lean towards other cloud platforms for my personal projects. Whenever I can at least. It is easier to create a set of load balanced servers involving a dozen aws services than to understand a bill on there
- jherskovic 4y agoI run a self-hosted instance for a bunch of friends. It's on a VM on a 10-year old homebuilt TrueNAS server. For added entertainment, I installed it manually on FreeBSD, because I like playing old-school 'I do things by hand' sysadmin at home. It's connected to the world through a free Cloudflare tunnel. I have a residential fiber connection. It doesn't use S3, it just dumps files to the filesystem. I did clamp down on federation settings, and it purges old content aggressively so federated content doesn't make the hard drives metaphorically explode. One user following someone on mastodon.social and you can see storage usage climb by gigs a day with defaults. After the initial setup it's been honestly quite painless. It just works. If people start relying on it, I'll probably move it to masto.host.
- skwheel 4y ago> For compute - a single Elastic Compute Cloud (EC2) t4g.small instance running Mastodon, it’s PostgresSQL database, and Redis. its
- insane_dreamer 4y agoA premade Mastodon CloudFormation template would be cool for those who want to set up their own.
- jancsika 4y agoLet's say I just want a single-user instance that's in the style of Hacker News: text only. 1. I want to write text-only posts and have them distributed to whoever is following me on Mastodon. 2. I want to receive all the posts of people I follow, and then just display/store the text and nothing else. Does something like this exist? Seems like it should be super slim and easy on resources.
- zimpenfish 4y agoHonk - https://humungus.tedunangst.com/r/honk https://humungus.tedunangst.com/r/honk Minimalist and opinionated but is a single binary with an SQLite db. Source is, uh, esoteric but is definitely hackable (there's a whole bunch of forks.) Seems to work ok with a few minor weirdnesses[1]. Should clarify that it doesn't have an API - you can't use Mastodon clients against it (until/unless someone writes on) - it's basically web only. [1] I can't follow someone on my GotoSocial instance from honk but I'm pretty sure I've debugged that as being GTS's fault[2] rather than honk. [2] They made many odd design decisions which cause(d) issues with federating to other systems. [edit: clarify that honk as no API]
- boyter 4y agoThe source is indeed esoteric, and sometimes hilarious. I have been using it to learn how the auth works against mastodon and some other parts of Activitypub. I do recommend it as a decent starting point to learn from.
- Ambroisie 4y agoCan you elaborate on some of gotosocial odd choices?
- zimpenfish 4y agoThe main one is how they handle keys[1] which broke federation with Pleroma for a while. There's also some weirdness with their HTTP signing which breaks Honk follow requests but I'm still tracking that down. [1] https://github.com/superseriousbusiness/gotosocial/issues/1181#issuecomment-1331877047 https://github.com/superseriousbusiness/gotosocial/issues/11...
- 0x420 4y agoIt’s comforting to know that even people who work at AWS will sometimes forget about runaway cloudwatch charges. it's not just me!
- wkat4242 4y agoGoToSocial seems to be a much better fit for a single user instance and it should be able to run on a scaleway stardust instance for €1.80 a month. I was considering running mastodon too but someone mentioned this here and it does indeed look great
- SergeAx 4y agoThis is so cute! Cartel employee talking about using their drugs) AWS is the most vendor-lock dependent company, IMO. Every single "architect solution" of their is a recipe of how to never get your data back for free.
- woopwoop24 4y agoi don't get why people would choose AWS in the first place ever for any small/amateur project. It is ridiculously expensive, not transparent about the costs which could be coming your way and you need a shitload of components to get by. just get some fixed price machine at some shitty provider and see how it goes, if you need a bigger one, just by a bigger machine. And if you outgrow that it might be a reasonable idea to just rent some colo space and calculate the upfront costs against a small server you build yourself. an intel nuc with 16gb memory and 4 cores is about 200$ and draws nothing in power. buy 2 connect them direct and hook them up to the colo provider. unlimited fiber and there you go. AWS is such a joke