27 ms·

We handle 80TB and 5M page views a month for under $400

766 points by darknoon 5y ago

tomtylor 5y ago
AtlasBarfed 5y ago
Uh, so how does cloudflare do it, since the answer is basically "cloudflare". They have sweet deals with the cloud providers?
- m_ke 5y ago
  They don't have a cloud computing business to put a walled garden around (yet). Also see their post about it https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
- tomnipotent 5y ago
  Bandwidth Alliance, a consortium of companies that offer free (or steeply discounted) bandwidth between participants. https://www.cloudflare.com/bandwidth-alliance/ https://www.cloudflare.com/bandwidth-alliance/
- deleted 5y ago
  [deleted]
- Nextgrid 5y ago
  Cloudflare has their own hardware and so pays a "market price" for bandwidth (which is much lower than what clouds (over)charge) and their business model relies on other value-add products rather than just bandwidth. I wouldn't be surprised if they actually lose money on this deal, but I guess it sometimes pays off to provide loss leaders.
- dreyfan 5y ago
  Cloudflare subsidizes the small guys in hopes they'll turn into big guys. Bandwidth isn't that expensive in the first place though, that's just a myth perpetuated by AWS/Azure/Google.
- tyingq 5y ago
  Cloudflare is currently operating at a close to break-even loss.
  - eastdakota 5y ago
    Sure, because we're hiring like crazy to build new things. As we said during our last earnings call, we intend to stay at break even as long as we can achieve extraordinary revenue growth and continue to deliver innovative new products. The more relevant number for this comparison would be our gross margin — much we have to spend on things like bandwidth and servers to service our customers divided by the revenue those customers generate — which in Q3 2021 (our last reported quarter) was 78%. Which is… pretty good for a services business like ours. I don't know the specifics of this customer, but I don't see anything that leads me to believe our margins would be out of the ordinary with them. There are a lot of scale economics in our business. In other words, we can definitely do things for less money because we service a lot of customers than any one customer could hope to do it on their own.
    - launchiterate 5y ago
      I love HN and it’s always amazing to see the founders answer the questions themselves and most importantly care a lot about the product.
    - jimsimmons 5y ago
      Do you see you offering more dynamic hosting in the future? Things like elastic databases, GPUs etc
    - moralestapia 5y ago
      It's super nice that you're here and your transparency is appreciated, but don't waste time in people like GP. Just walk your way and don't mind the naysayers. My two cents from a regular guy to a billionaire, :).
      - tyingq 5y ago
        >don't mind the naysayers Noting they are operating at a slight loss, during a period where they want to drive growth, isn't "naysaying". The question is whether the current pricing is tied at all to the "operating at a slight loss". He replied to say it mostly wasn't. Remember, for example, when Uber rides were really cheap?
- NorwegianDude 5y ago
  Bandwidth is cheap. Cloud providers are insanely expensive for bandwidth. Cloudflare is peering in a lot of locations, making it even cheaper for them.
- ummonk 5y ago
  The biggest costs of server bandwidth are due to transmitting data over large distances (since the internet backbone operates on a sender-pays model when unbalanced traffic is involved in peering). By locating CDN nodes globally, Cloudflare can ensure data is sent near the last mile and thus inexpensive to send. Unless ISPs decide to shake it down the way they do with Netflix of course.
jaytaylor 5y ago
How do 5M page views consume 80TB of traffic? That sounded like a lot, but calculates out to 1,600KB per page view. That's a lot of JS and images on every page :) A few years ago when I hosted what felt to me like a popular site, I was serving serving > 150K page views a month from a 1.5Mbps adsl uplink. With that said, back then you could gzip everything and there was no letsencrypt or CloudFlare walls across the board (didn't exist yet). Good thing bandwidth is easier to come by these days.
- smadge 5y ago
  The website is for downloading 3D graphics assets, which are typically fairly large files.
  - jaytaylor 5y ago
    I see.. was thrown off by: > Running a massively popular website and asset resource while being funded primarily by donations has always been a core challenge of Poly Haven. My brain went: poly.. fill.io or something Thank you, sorry, am embarrassed.
- schleck8 5y ago
  > asset resource HDRIs, high res textures & 3D models
- ghostly_s 5y ago
  Did you even look at the article? They operate a digital asset store.
  - jrockway 5y ago
    I don't think they're a store, everything is free and Patreon supporters get early access. I do occasional 3D rendering and the HDRIs are great. The downloads are fast enough to make you ask "whoa, how is this free".
  - itsdrewmiller 5y ago
    https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html 'Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."'
latchkey 5y ago
I've deployed similar stacks at scale multiple times and it is amazing how well it all works given how little it costs.
tiffanyh 5y ago
Seems like they could pratically host everything on Cloudflare and save even more money. Asset storage: Backblaze B2 – $11 (replace with Cloudflare R2) Web hosting: Vercel – $20 (replace with Cloudflare Pages, $0 cost) Database: Firestore – $100 (replace with Cloudflare) API: Vultr – $5 Image hosting & optimization: Bunny.net – $27 (replace with Cloudflare) Domains: Cloudflare – $4 Email fees: MXroute – $3
- mrlinx 5y ago
  How so? R2 is 0.015$ per GB while backblaze B2 is just $0.005.
  - deleted 5y ago
    [deleted]
- shmoogy 5y ago
  I don't know if cloudflare image will be cheaper than bunnycdn. We migrated from pro to enterprise cloudflare because the imaging api costs basically put us at the price where upgrading gave us a lot of added benefits and cheaper additional costs during our peak seasons.
- lkbm 5y ago
  > Database: Firestore – $100 (replace with Cloudflare) Does Cloudflare have a database offering? Do you just mean the Worker KVs, or is there a full relational database?
  - dragonwriter 5y ago
    Firestore isn't a full relational DB, so why would a replacement need to be?
    - lkbm 5y ago
      Good point. I'd assumed it was a full RMDB, but hadn't ever really spent much time with it.
mike_d 5y ago
It is annoying that the answer is basically "we paid someone [Cloudflare] to do it." In light of that I decided to run some of my own numbers and figure it out. They are pushing about 245 Mbps out of Cloudflare (averaged over the month). Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume. Cloudflare dumps 40% of their traffic over peering, putting their price at $14.70/mo. Ignoring fixed capex of servers, Cloudflare is making about $25/mo on this customer. Given the $11 Backblaze bill, I estimate about 2 TB of data. A capable dedicated server with 2 TB of disk and 1 Gbps unlimited bandwidth will run about $30 in a major European metro, maybe double that for the US. With a grand total of $370 vs. $60 worst case, they are spending 516% more to be "serverless." Edit: Yes Cloudflare has more than one server. Double the price and put one in the US and you still come out ahead. Edit 2: I'm not saying one way or the other is better. Just that the title is very clickbaity for a "put a credit card into a website" payoff.
- schleck8 5y ago
  That would be a single node though, in contrast to cloudflare's edge, and would probably slow down download speeds for everyone outside europe
- walrus01 5y ago
  An actually capable dedicated bare metal server with a good amount of RAM, CPU and disk, and 1Gbps will be more like $250-300/mo USD. This would be approximately equivalent in spec to what you could build for $2500 purchase cost if buying a 1U machine and colocating itself. For 30 bucks a month you'll get something very old and weak.
  - mike_d 5y ago
    The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s. Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59 https://oneprovider.com/order/item/dediconf/59
    - 5y ago
deleted 5y ago
[deleted]
VWWHFSfQ 5y ago
Question: I want to make a live video streaming website a la Twitch.tv. How much would CloudFlare charge me to stream 8 Mbps to ~80,000 viewers for 4 hours?
- jrockway 5y ago
  I think this is the sort of thing you want to ask their sales team. I thought about doing this a while ago, after I left my job with an ISP. I would just have bought transit from them (and maybe made them regret their 10Gbps for $1000/month plan ;)
- mike_d 5y ago
  Almost $20,000. 19,200,000 minutes at $1 per 1,000 minutes. https://support.cloudflare.com/hc/en-us/articles/360016450871-Billing-for-Cloudflare-Stream https://support.cloudflare.com/hc/en-us/articles/36001645087...
  - busymom0 5y ago
    There's probably some bulk deals if you get in touch with their sales team?
    - ignoramous 5y ago
      Yes, you could. The last time someone I know spoke to them, the standard enterprise plans started at $5000/mo and got you some very nice freebies. Cloudflare's also announced managed WebRTC APIs (in closed beta) which look promising for creating meeting rooms ala Zoom.us. I imagine, these are already available to tech-shops on enterprise plans. Though, mux.com is decent too, from what I've heard; for certain workloads, LiveStream and Vimeo might be cheaper. There's Peer5, PeerTube, and BitTorrent in the P2P space (among many such solutions).
      - busymom0 5y ago
        > mux.com First time hearing of them (due to my own ignorance probably). Any details on them?
qeternity 5y ago
This is undoubtedly better than most chuck-it-on-AWS types would achieve, but it's still quite a bit more expensive than could achieved with marginally more effort. They are already on Digital Ocean - if they used their managed database offering, they could slash their Firestore bill and get rid of their Argo bill, bringing the cost sub $200 (but - massively reduced returns on investment of effort).
willcipriano 5y ago
If you are worried about getting your blog post to the top of Reddit or Hacker News (I've never been there myself); you can have a very modest web server or even a pay per request serverless sort of thing and pay $20 real quick to Cloudflare if you happen to get popular. It's the Bart Simpson method of highly scalability[0], for static content you can have global datacenter coverage in a couple minutes or so if you use them for DNS to start with. It even works if the origin server goes down. [0]https://www.youtube.com/watch?v=aKU3hMvD31w https://www.youtube.com/watch?v=aKU3hMvD31w
- PragmaticPulp 5y ago
  > you can have a very modest web server or even a pay per request serverless sort of thing and pay $20 real quick to Cloudflare if you happen to get popular. I get the impression that a lot of the critics in this thread don't really understand Cloudflare, how cheap it is, or even the concept of CDNs in general. $20/month for Cloudflare Pro is a steal for what you get. Spinning up a dedicated server in a single datacenter somewhere isn't going to give the same results, especially if your users are geographically distributed like in this case.
  - deleted 5y ago
    [deleted]
  - encryptluks2 5y ago
    Don't you get tht same for free with GitLab Paged and last I checked Cloudflare was free too for caching static HTML assets?
    - CRConrad 5y ago
      > last I checked Cloudflare was free too for caching static HTML assets? If not free then very cheap, as I understood TFA: Wasn't that why they have two separate domains and serve static assets from one of them, to be able to use the cheapest Cloudflare tier for that domain = those assets?
  - superkuh 5y ago
throwoutway 5y ago
Ironically, it's struggling to load the images for me.
- busymom0 5y ago
  Worked pretty smoothly for me when I browsed around assets.
- soheilpro 5y ago
  Same here in Indonesia.
- eliseumds 5y ago
  Loading instantly here (Sao Paulo).
- Fornax96 5y ago
  The page took 40 seconds to load for me in the Netherlands
technotarek 5y ago
We have an incredibly similar stack for attic.city, aside from the application layer. We lean heavily on CF (edge caching, Argo and page rules). But we’ve recently changed our image assets from S3 to CF R2. Anyone else put that side by side with Bunny?
- Jamie9912 5y ago
  What? how do you have access to R2?
  - the_duke 5y ago
    It's in private beta.
IvanK_net 5y ago
I was running www.Photopea.com with 10M page views a month for $40 a year. Now, I upgraded to $60 a month. I never used any CDN.
- IvanK_net 5y ago
  I would love to use a CDN. But I am afraid of things I do not fully understand, and I am afraid of making things too complex. If I update a single file, how long it takes until nobody in the world can access the old version anymore? Does it take seconds / minutes / hours? Also, some files should not be cached at all (like PHP requests). I am afraid it would take me days or weeks to learn everything and to cofigure the CDN properly, and I am risking being offline for a part of the world during that time. Also, if there is a problem at the CDN, my website would be broken, too. If someone could help me, you can write me at support@photopea.com
- TheRealSteel 5y ago
  Ever since MacOS broke Gimp I use Photopea regularly! Fantastic site!
  - aasasd 5y ago
    Please tell more about this ‘MacOS broke Gimp’ thing, because it's the first time I hear of that and I'd like to dodge a firsthand experience.
    - metadat 5y ago
      I used it successfully on my Mac last week. It's still GIMP (bleh!) but it works.
      - undefuser 5y ago
        You can consider Affinity Photo
        5y ago
sgtfrankieboy 5y ago
I don't believe Argo is still needed? Tiered Caching is now a free feature.
- DenseComet 5y ago
  Yep was also going to mention this. Obviously, Argo still helps with latency and some other things probably, but if they're paying the additional $160 just for tiered caching, they should be able to switch off Argo.
jason0597 5y ago
I often wonder why so many companies seem to spend tens of thousands of dollars per month on AWS services for their hosting. If most web apps nowadays are just CRUD apps, then why would you need to spend more than €300/month on hosting? All you'd need is data and the website itself, right? I'm currently clicking through various European hosting services and they seem to offer great dedicated servers at good prices. I cannot wrap my head around these ridiculous costs I keep hearing about. A guy I know was telling me how they were spending tens of thousands of dollars per month on AWS at his company. Is it because everyone is writing their stuff on node.js, putting 500kb of JS in every web page they serve, putting Docker everywhere when a chroot would suffice, microservices, Kubernetes, or hell even writing SAPs when we don't need them? I am genuinely confused
- smarx007 5y ago
  I don't think that most workloads are webapps and many webapps are not just CRUD. HN is not the web either, in case HN formed such a viewpoint for you.
- ummonk 5y ago
  Until you're spending the salary of a large fraction of your engineers on hosting costs, there are much bigger things to worry about than hosting costs. AWS is user friendly and a lot of engineers are familiar with it, so developer productivity is strongly in its favor. Also, you know what you're getting security-wise with AWS, and no one will blame you when your website / service goes down because AWS is down.
  - oogetyboogety 5y ago
    Wasn't that like IBM's business motto at some point? No one ever got fired for going with IBM. Then IBM definitely lost a lot of business to cloud modernization. Maybe the next logical step is the cheaper CloudFlare, which another comment mentioned is actually pretty profitable and then I'm sure the next cheaper incumbent will come in with just slightly less trust and reliability.
- jeppesen-io 5y ago
r1ch 5y ago
It's annoying how you have to work around Cloudflare's weird Argo pricing like this. Paying per GB for already-cached data that takes the same route as non-Argo traffic makes no sense. I believe it's also against Cloudflare's TOS to use it as an asset-hosting platform.
celdon25 5y ago
Awesome! I hope this means more competitors for YouTube soon, featuring videos that aren't just related to fringe politics.
- cblconfederate 5y ago
  You think cloudflare will be more tolerant?
- jjcm 5y ago
  Self shill: I'm working on https://non.io https://non.io, a reddit-meets-patreon platform that supports up to 8k video. Planning on launching in the next couple of months (just about got the stripe integration working for auto-payouts). One of my main reasons for making it was I was annoyed at YouTube lately. Would love feedback as I'm aiming to be a competitor. You can sign in with Username: hn@hn.com Password: hackernews - The site is currently in demo mode, and the db will be wiped before launch - feel free to sign in and poke around. Also it's hosted in Australia currently, so site may be a little slow for those in the US. Source is here: https://github.com/jjcm/soci https://github.com/jjcm/soci
deleted 5y ago
[deleted]
bcjordan 5y ago
Fun read, love finding those combination perf/cost wins! Have you considered using Firestore in Datastore mode[0]? It might make all of your reads free[1], though migrating could be a project. [0] https://cloud.google.com/datastore/docs/firestore-or-datastore https://cloud.google.com/datastore/docs/firestore-or-datasto... [1] https://stackoverflow.com/a/53313540 https://stackoverflow.com/a/53313540
ttty 5y ago
ttty 5y ago
jjcm 5y ago
On a similar note - does anyone have a recommendation for a budget cdn for video? I don’t necessarily need encoding capabilities (though it’s a plus), but rather high storage/bandwidth options.
- Ameo 5y ago
  BunnyCDN (mentioned in the blog post) has this functionality. It is mentioned in the blog post and they're using it for image processing, but it works for videos too. I use it myself, but just as a normal static file CDN, but they have dedicated tabs in their UI for video stuff. Their bandwidth pricing is also very reasonable ($5/TB with their "bulk" option and $10/TB for standard)
gardnr 5y ago
Do the Bunny images go through Argo? If you were OK with your content only updating periodically you might be able to do away with Firestore, Argo, and Vultr. You definitely do a great job of mitigating db hits. Consider publishing all the data to a CDN using something like Gatsby. In this scenario your CI/CD would build a static website from all the assets and publish it to a static server. You mentioned that only your view counts are the only really dynamic part. You could just estimate and emulate those or design them a different way. Martin Fowler has an article on this: https://martinfowler.com/bliki/EditingPublishingSeparation.html https://martinfowler.com/bliki/EditingPublishingSeparation.h...
soheil 5y ago
> But, we can get around that because of a partnership between Backblaze and Cloudflare they call the Bandwidth Alliance. This reads like a paid marketing post for Cloudflare. It's astonishing how many times the author conflates browser cache-able assets with cached assets on a CDN. When a browser downloads a static asset if the web server is configured properly those files will be cached and there won't be a need to re-download them for a very long time. There is of course the issue with modern web frameworks like React generating a single massive js/css file that bundles everything all over again in a unique file busting all previous cached versions across all users' browsers just because a comma was added to a sentence. Keep your js/css files small, serve them from your own web server and set a reasonable expire header, no need to pay Cloudflare $40/mo to continue gatekeeping the internet.
- systemvoltage 5y ago
  I didn’t see it as an ad for Cloudflare unless I also then consider every single post about AWS in last 10 years on HN as ads. If anything, I’m fine with a new competitor to BigCloud that’s been unchecked and increasingly hostile (cost-wise).
- DenseComet 5y ago
  They provide 3D asset downloads, most folks will only be downloading an asset a single time. Therefore, browser side caching doesn't really help, they need to cache the files on a CDN.
alphachloride 5y ago
I tried looking, and I find it very irritating that in that whole web page, there is not one single link in the menu/footer/sidebar to polyhaven.com home page so I can click and discover what it actually is. Not one. This occurs on many company blogs as well operating under a subdomain like blog.whatever.com To be clear, this is a very tangential and irrelevant nitpick and I understand it does not contribute to the content of the website itself.
- cerved 5y ago
  I had no problems navigating there. Every single link in the hamburger lead me to polyhaven.com
  - alphachloride 5y ago
    To the polyhaven.com home page? I don't think it did. They are all sub pages which adds another navigation step to see the root page, which is what I find irritating.
- chillfox 5y ago
  Not having a link forces people to navigate to the page on their own. For a lot of people that takes the form of a Google search. Lots of people googling specifically for your brand increases your search rankings. Or so goes the theory anyway.
- helsinkiandrew 5y ago
  Yes! - the same goes with support sites on different domains (support.whatever.com or whatever.zendesk.com etc) that appear in google results and don't have a link to the company.
  - CrimsonRain 5y ago
    I think this is a result of app stores enforcing links to external digital products thing. That's why all these support pages which need to be linked from app, don't have a single link to main site.
helsontaveras18 5y ago
This website is one of the fastest web pages I’ve ever used. Although it’s simple, it immediately loads pages and pages of images. Like in a flash. I’m on mobile web and it’s such a snappy experience.
- lelandfe 5y ago
  If Poly Haven is reading this: you nearly have an amazing FCP on the models browser but the external normalize.css file is killing you.[0] Self-hosting would drastically improve your 75p paint time. I'd also encourage you load your fonts late via JS. Your main JS package competes right now with WOFF files from Google Fonts for priority and there's no need for that. [0] https://www.webpagetest.org/result/220106_BiDc42_428a3caec56095643c1f6786a854cc6a/1/details/#waterfall_view_step1 https://www.webpagetest.org/result/220106_BiDc42_428a3caec56...
  - hansel_der 5y ago
    > I'd also encourage you load your fonts late via JS can't tell if joke, so: there are already enough sites that display content for a split second and then some script runs (or fails?) and there is either nothing on screen or an error message. this is ridiculous - please stop!
    - lelandfe 5y ago
      Loading experience for this would be that a fallback font loads first and is later replaced. If something errors, it’s not replaced. What you’re describing has been thankfully avoidable for many years.
      - hansel_der 5y ago
        this failure mode has been avoidable like forever and i have only seen it become more widespred in the last decade. be sure i blame ad-tech and analytics
  - austinpena 5y ago
otar 5y ago
First of all, kudos, I’m a junkie for such optimizations myself. It’s a pleasure that your systems are optimized well, you sleep well and you’re proud :) Two thoughts: 1) CloudFlare offers incredible service. What would need the team of netadmins/sysadmins can be handled via their UI easily. I use Argo in one project and yes, it is a very efficient tool. CloudFlare is becoming like AWS, knowing its tools is a skill. Maybe one day well see an official CloudFlare Solution Architect certification?.. :)) 2) For such traffic heavy websites I don’t think there’s a better combination than Static Websites hosted over CDN, which is cached aggressively and for the dynamic part serverless as a backend. If you configure it properly (cache optimization, rate limiting to not get a huge paycheck if DDoSed, etc…) this setup is like set-up-and-forget-it, without the need to invest loads of resources into it.
- xyexs 5y ago
  I'm new to this stuff, but what the source article describes is essentially just that it is cheaper to run a static website with a CDN than without - right? I'm mostly curious because they described the cloudflare service as a cache layer rather than a CDN.
  - collinmanderson 5y ago
    > They described the cloudflare service as a cache layer rather than a CDN. Not sure how most CDNs work, but for Cloudflare, content is not actively pushed to edge nodes, it’s only pulled to an edge when needed and then cached there usually for an hour or two. The caching is still distributed, and all of the tcp+https roundtrips are done to a local data center which makes things faster. > what the source article describes is essentially just that it is cheaper to run a static website with a CDN than without - right? Right. The cost savings comes from caching done by the CDN, and public static content is super easy to cache. The distributed part makes things faster, but doesn’t necessarily save money.
goldbattle 5y ago
My thought is why don't the authors use github pages to host their static site with their links to a dedicated server with an unmetered link since their whole project is CC0. Downloading assets doesn't need to be low latency, thus I don't think the caching spatially near the user is important. Having a user login for patreon is something that would be missing and require some more thought I admit. I think quite a lot of other people have mentioned in the thread that they are getting a lot of other "benefits" from using multiple services, but I don't see how these help solve the problem of data delivery besides taking advantage of the Cloudflare + Backblaze alliance which is $31 if their main website is a static one.
- aardvarkr 5y ago
  80TB/mo is quite a bit beyond the generosity of GitHub pages
  - lorey 5y ago
    any source on that?
    - klohto 5y ago
      I mean, kinda obvious since everything is metered on GH. Soft limit is around 100GB.
otabdeveloper4 5y ago
Isn't this like 2 requests per second? That's massively overpriced, with current prices you should fit into a 5 dollar budget.
jijji 5y ago
I like the idea of cloudflare as a OSPF style file cache, but what about putting 80TB in a colo-facility on a 1U box and paying $80/month to host that yourself? is it cheaper to use Amazon S3?
- indigodaddy 5y ago
  I think you’ve got the 80TB item confused with storage. They are talking about 80 TB of transfer over the internet in a month. I doubt they have 80TB of assets laying around on Vercel. That would cost them an arm and a leg with Vercel I’d expect.
throw14082020 5y ago
> To avoid this problem in the future, I decided to splurge a bit and go for a cloud solution where I wouldn’t have to worry about reliability, performance, scaling or integrity ever again: Google Firestore. > Google Firebase is nice and convenient, but it is quite expensive. We could investigate some other managed database options in future. I've only seen people get annoyed with Firestore over time, and migrating out of it. People do end up worrying. At first, They seem to choose Firestore because it's strongly marketed and seems suitable for a new project. And then data modeling, high prices or scalability becomes a problem.
- MichaelRazum 5y ago
  What is a good solution compared to firebase ? I mean for a new project, alone something like user handling, registration, email sending, recovering passwords seem to cost some effort. Also be able to react to events, is kind of trivial in firebase. So is it really a bad choise (for the beginning)? Later you can always try to optimize.
  - simonbarker87 5y ago
    SupaBase is a good alternative in my opinion but they are not as feature rich yet - but you do get SQL again which is delightful
  - janto 5y ago
    Many use Firebase for auth and user management only.
  - mellavora 5y ago
    so the problem here is if you have all of those needs, then really soon in the life of the project you are going to need good tooling to handle them. So choosing something which makes it "one-click" to set up but total madness to manage is a really short-term optimization, only worth it for a pure prototype which you will throw away no matter how successful. If you know you need those things to reach success, then it is better to make the up-front investment to get good tools for those. If you still want to go with a cloud provider, AWS Amplify has some interesting tooling. I've build products both against Amplify and Firestore (and Firebase). Yes, Firebase is a few days to a week faster to set up (integrated user management, as you say), but AWS gives more sophisticated control and is built around scripted deployments. You pay for it, of course, and I'm not arguing AWS vs running your own server. I am saying if the choice is AWS or Firebase, that a few days researching the choice would give you knowledge you could use for launching the next 10 prototypes you have in mind.
techie128 5y ago
80TB for 5M page views works out to 16.7MiB per page view. I assume you're serving images and other assets on the page otherwise you have room to optimize on the page weight.
- semi-extrinsic 5y ago
  Even if you are serving images that's very heavy for a single page. A 4K resolution jpeg sits at around 6-7 MiB.
martindbp 5y ago
As someone building on a mostly free service with lots of content this is very helpful, thank you! Will probably migrate from S3 to Blackblaze, seems like a great deal.
- austinpena 5y ago
  Depending on how much you access per month, Storj DCS[0] and Wasabi[1] can also be interesting choices. - Storj has the lowest cost per TB, but charges for egress - Wasabi doesn't charge for egress, but has a fair use policy where if you egress more than you store they can kick you off their platform[2] - Wasabi is also a better fit only if you plan to keep your files around for 90+ days (they have a minimum object retention period) - The bandwidth alliance is only available for HTML related content unless you're on a specific paid plan[3] [0] https://www.storj.io/ https://www.storj.io/ [1] https://wasabi.com/ https://wasabi.com/ [2] https://wasabi.com/paygo-pricing-faq/#free-egress-policy https://wasabi.com/paygo-pricing-faq/#free-egress-policy [3] https://www.cloudflare.com/terms/#28-limitation-on-serving-non-html-content https://www.cloudflare.com/terms/#28-limitation-on-serving-n... *opinion is mine, not my employer's
cblconfederate 5y ago
how is 5M page views a month a lot? I just looked and my tiny old hetzner server serves at least 30M per month. Presumably hetzner gives 1Gbit for free so that should be enough for 80T (0.2Gbit/month, though i dont have that much traffic). It cost $60
hffft 5y ago
mad182 5y ago
I handle ~150TB and 26M page views for ~$500 by simply renting a few dedicated servers at hetzner. And if I didn't need quite a lot of processing power (more than average website), it would be much lower. I only need so many servers for the CPU power, not traffic.
- folli 5y ago
  Any experience with other providers of dedicated servers, e.g. Digitalocean?
  - mad182 5y ago
    Does DO even offer dedicated servers? I have used their vps in the past. It was ok, no issues, but more expensive than hetzners vps or dedicated for large traffic/cpu needs.
    - reducesuffering 5y ago
      Yes DO does. Generally slightly more expensive than Hetzner, with a few more AWS-like features.
  - GOATS- 5y ago
    You cannot go wrong with Leaseweb or Nforce if you're looking for something in EU.
- razemio 5y ago
  This! Hetzner root servers, there is nothing else I know on the market that comes close in price/power ratio when you use the "Serverbörse". I run multiple low traffic websites, databases and a heavy traffic elk instance + my complete homelab in proxmox / docker for 48€ a month and there is even room for more. Highly recommended!
  - SergeAx 5y ago
    What is your strategy for dealing with server failure?
kkjjkgjjgg 5y ago
When I tried CloudFlare, they injected a JavaScript into my site that was bigger than my entire site. That was a couple of years ago, but I can't bring myself to trust them anymore. I also don't like their growing control over the web.
BiteCode_dev 5y ago
Honestly you can go very far with unmettered data plans. We consume 2GB/S, maxed out, which ends up being 5184TB a month. We have millions on users a day, all on a streaming video platform. It cost less than $2K/Month. The cloud is crazy expensive. Private servers are beasts, and they are cheap. Of course, for this price, you don't have redundancy and horizontal scaling. You also don't have to maintain and debug a system with redundancy and horizontal scaling.
- nix23 5y ago
  Thanks! Exactly that! I always tell that customers and their response is: -But google/facebook/amazon... -But uptime needs to be 99.999 -But everyone uses cloud Most businesses are not a trading-market, have less then 100 peoples (aka you are probably not another amazon), and no bonus using a cloud/kubernetes etc. But it's the same old story, in the 00's i used the ~same arguments against buying OracleDB ;)
  - CRConrad 5y ago
    > But it's the same old story, in the 00's i used the ~same arguments against buying OracleDB ;) But no matter how logically convincing your arguments were, most of the time upper manglement just went on buying Oracle, right...?
  - XCSme 5y ago
    > uptime needs to be 99.999 Isn't AWS down like every two months for a few hours? That's far off the 99.999% mark. No one can guarantee 100% uptime and sometimes it's even better to have that under your control (eg. have a dedicated server and a backup one from different providers). My point is that, if you want the highest possible uptime, you shouldn't rely on a single (cloud) provider.
  - Cthulhu_ 5y ago
    And you can tell them all of those are possible, but they need to have a massive budget, not just for the monthly bills but also to hire experts to set things up with those constraints. I worked at a company once that, from higher up, said that they had to have five nines of uptime. We had some really good cloud engineers there (one guy set up a server / internet container for the military in Afghanistan; in hindsight he said they should've just sent a container of porn dvd's), and they really went to town. For that five nines uptime, you're already pretty much required to set up your infrastructure to use multiple availability zones, everything redundant multiple times, etc. Of course, the actual software we wrote was just a bunch of CRUD services written in nodejs (later scala because IDK), on top of a pile of shit java that abstracted away decades of legacy mainframes.
dmitrygr 5y ago
5M views per month is under 2 per second. In 2005, that was servable from a 16-bit pic microcontroller with a 10-base-t phy from microchip inc
sgt 5y ago
How are Polyhaven funding their staff? Their Patreon income won't even cover a tenth. Do they have corporate sponsors or special agreements?
twblalock 5y ago
When I see per month or per day stats I get suspicious. Let's say a month has 30 days -- 5 million views a months is 166,666k per day, 6944 per hour, 116 per minute, and ~1.92 per second. 1.92 qps. Of course it's not expensive, it's a tiny amount of traffic!
- PUSH_AX 5y ago
  To look at a different angle, I would imagine the traffic isn't evenly distributed, and part of being happy with your offering will be if things don't grind to a halt during peaks.
- Karishma1234fff 5y ago
  I know, but it is impressive that they are doing it for $400.
  - richardwhiuk 5y ago
    It's impressive it takes that much money to do almost no traffic. A raspberry PI could handle the compute.
    - andyp-kw 5y ago
      Only if the traffic is evenly distributed during the day. Which it's almost certainly not.
      - richardwhiuk 5y ago
        The traffic graphs in their blog look pretty stable. Maybe 1.5x, but not hugely uneven.
      - netr0ute 5y ago
        You can get a $15 2.5 gigabit ethernet adapter for that Raspberry and now you're set for peak loads.
  - deleted 5y ago
    [deleted]
bvm 5y ago
> This website blog.polyhaven.com/how-we-handle-80tb-and-5m-page-views-a-month-for-under-400/ is currently offline. Cloudflare's Always Online™ shows a snapshot of this web page from the Internet Archive's Wayback Machine. To check for the live version, click Refresh. hm
- leke 5y ago
  Yep, I got that too. Absolutely beautiful.
- iso1631 5y ago
  The blog's down, but polyhaven.com isn't. Presumably the blog isn't as important as the core service and thus isn't hosted on the same infrastructure
nuker 5y ago
They dead now? Btw 80TB/mo = 30 MB/s, 5M/mo = 2 views per second, not mind blowing
darkwater 5y ago
The origin website is currently down according to Cloudflare...
johnyreader 5y ago
[flagged]
AtNightWeCode 5y ago
Worked on several sites with similar setup. Works great and is affordable. One thing to be aware of is that sometimes you can get at a lot of cache misses. Therefor the back-end services must not stop working completely if flooded with requests.
deleted 5y ago
[deleted]
imdsm 5y ago
s/handle/handled
HeavyStorm 5y ago
Well, it's down now. Foul play?
reph2097 5y ago
Your blog won't load after waiting for like 10 seconds. Funny.
xurukefi 5y ago
> This website blog.polyhaven.com/how-we-handle-80tb-and-5m-page-views-a-month-for-under-400/ is currently offline. Cloudflare's Always Online™ shows a snapshot of this web page from the Internet Archive's Wayback Machine. To check for the live version, click Refresh. oh the irony
- mrintellectual 5y ago
  Indeed not the greatest advertisement for their infrastructure, but regardless you have to admit that <$400 is fairly impressive.
- anatolinicolae 5y ago
  Aged well I see...
- gionn 5y ago
  I bet it's powered by wordpress.
Gepsens 5y ago
Hey look here guys it's cheap to do html !
ThinkBeat 5y ago
I dont understand this entirely . A lot of the article is spent talking about how great and fast Cloudflare is. Then at the end he says he uses Bunny as the CDN for images. Why use Bunny instead of Cloudflare? One more moving part. I presume it is cost related?
- spiffytech 5y ago
  I too wish they'd spelled out that choice. I speculate it's because Cloudflare's TOS prohibits you from using their CDN to serve "video or a disproportionate percentage of pictures, audio files, or other non-HTML content" on typical plans. See section 2.8 on https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ . Since PolyHaven seems to be a purveyor purveyor of 3D assets and they use Bunny to host "all of our images shown on the website (thumbnails, renders, previews, etc.)", I'm guessing a lot of their assets exceed Cloudflare's TOS. PolyHaven mentions using Bunny's image optimization service, so that'd factor into the decision too.
maxclark 5y ago
The important thing here is caching/static delivery I've had this argument too many times to count. Every page view does not need to be dynamic per user request. Creating a sane cache policy reduces origin resources, servers, cost, etc... and surprise gives a better user experience at the same time.

766 points by darknoon 5y ago

Uh, so how does cloudflare do it, since the answer is basically "cloudflare". They have sweet deals with the cloud providers?

m_ke 5y ago
They don't have a cloud computing business to put a walled garden around (yet). Also see their post about it https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
tomnipotent 5y ago
Bandwidth Alliance, a consortium of companies that offer free (or steeply discounted) bandwidth between participants. https://www.cloudflare.com/bandwidth-alliance/ https://www.cloudflare.com/bandwidth-alliance/
deleted 5y ago
[deleted]
Nextgrid 5y ago
Cloudflare has their own hardware and so pays a "market price" for bandwidth (which is much lower than what clouds (over)charge) and their business model relies on other value-add products rather than just bandwidth. I wouldn't be surprised if they actually lose money on this deal, but I guess it sometimes pays off to provide loss leaders.
dreyfan 5y ago
Cloudflare subsidizes the small guys in hopes they'll turn into big guys. Bandwidth isn't that expensive in the first place though, that's just a myth perpetuated by AWS/Azure/Google.
tyingq 5y ago
Cloudflare is currently operating at a close to break-even loss.
- eastdakota 5y ago
  Sure, because we're hiring like crazy to build new things. As we said during our last earnings call, we intend to stay at break even as long as we can achieve extraordinary revenue growth and continue to deliver innovative new products. The more relevant number for this comparison would be our gross margin — much we have to spend on things like bandwidth and servers to service our customers divided by the revenue those customers generate — which in Q3 2021 (our last reported quarter) was 78%. Which is… pretty good for a services business like ours. I don't know the specifics of this customer, but I don't see anything that leads me to believe our margins would be out of the ordinary with them. There are a lot of scale economics in our business. In other words, we can definitely do things for less money because we service a lot of customers than any one customer could hope to do it on their own.
  - launchiterate 5y ago
    I love HN and it’s always amazing to see the founders answer the questions themselves and most importantly care a lot about the product.
  - jimsimmons 5y ago
    Do you see you offering more dynamic hosting in the future? Things like elastic databases, GPUs etc
  - moralestapia 5y ago
    It's super nice that you're here and your transparency is appreciated, but don't waste time in people like GP. Just walk your way and don't mind the naysayers. My two cents from a regular guy to a billionaire, :).
    - tyingq 5y ago
      >don't mind the naysayers Noting they are operating at a slight loss, during a period where they want to drive growth, isn't "naysaying". The question is whether the current pricing is tied at all to the "operating at a slight loss". He replied to say it mostly wasn't. Remember, for example, when Uber rides were really cheap?
NorwegianDude 5y ago
Bandwidth is cheap. Cloud providers are insanely expensive for bandwidth. Cloudflare is peering in a lot of locations, making it even cheaper for them.
ummonk 5y ago
The biggest costs of server bandwidth are due to transmitting data over large distances (since the internet backbone operates on a sender-pays model when unbalanced traffic is involved in peering). By locating CDN nodes globally, Cloudflare can ensure data is sent near the last mile and thus inexpensive to send. Unless ISPs decide to shake it down the way they do with Netflix of course.

jaytaylor 5y ago

How do 5M page views consume 80TB of traffic? That sounded like a lot, but calculates out to 1,600KB per page view. That's a lot of JS and images on every page :) A few years ago when I hosted what felt to me like a popular site, I was serving serving > 150K page views a month from a 1.5Mbps adsl uplink. With that said, back then you could gzip everything and there was no letsencrypt or CloudFlare walls across the board (didn't exist yet). Good thing bandwidth is easier to come by these days.

smadge 5y ago
The website is for downloading 3D graphics assets, which are typically fairly large files.
- jaytaylor 5y ago
  I see.. was thrown off by: > Running a massively popular website and asset resource while being funded primarily by donations has always been a core challenge of Poly Haven. My brain went: poly.. fill.io or something Thank you, sorry, am embarrassed.
schleck8 5y ago
> asset resource HDRIs, high res textures & 3D models
ghostly_s 5y ago
Did you even look at the article? They operate a digital asset store.
- jrockway 5y ago
  I don't think they're a store, everything is free and Patreon supporters get early access. I do occasional 3D rendering and the HDRIs are great. The downloads are fast enough to make you ask "whoa, how is this free".
- itsdrewmiller 5y ago
  https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html 'Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."'

latchkey 5y ago

I've deployed similar stacks at scale multiple times and it is amazing how well it all works given how little it costs.

tiffanyh 5y ago

Seems like they could pratically host everything on Cloudflare and save even more money. Asset storage: Backblaze B2 – $11 (replace with Cloudflare R2) Web hosting: Vercel – $20 (replace with Cloudflare Pages, $0 cost) Database: Firestore – $100 (replace with Cloudflare) API: Vultr – $5 Image hosting & optimization: Bunny.net – $27 (replace with Cloudflare) Domains: Cloudflare – $4 Email fees: MXroute – $3

mrlinx 5y ago
How so? R2 is 0.015$ per GB while backblaze B2 is just $0.005.
- deleted 5y ago
  [deleted]
shmoogy 5y ago
I don't know if cloudflare image will be cheaper than bunnycdn. We migrated from pro to enterprise cloudflare because the imaging api costs basically put us at the price where upgrading gave us a lot of added benefits and cheaper additional costs during our peak seasons.
lkbm 5y ago
> Database: Firestore – $100 (replace with Cloudflare) Does Cloudflare have a database offering? Do you just mean the Worker KVs, or is there a full relational database?
- dragonwriter 5y ago
  Firestore isn't a full relational DB, so why would a replacement need to be?
  - lkbm 5y ago
    Good point. I'd assumed it was a full RMDB, but hadn't ever really spent much time with it.

mike_d 5y ago

It is annoying that the answer is basically "we paid someone [Cloudflare] to do it." In light of that I decided to run some of my own numbers and figure it out. They are pushing about 245 Mbps out of Cloudflare (averaged over the month). Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume. Cloudflare dumps 40% of their traffic over peering, putting their price at $14.70/mo. Ignoring fixed capex of servers, Cloudflare is making about $25/mo on this customer. Given the $11 Backblaze bill, I estimate about 2 TB of data. A capable dedicated server with 2 TB of disk and 1 Gbps unlimited bandwidth will run about $30 in a major European metro, maybe double that for the US. With a grand total of $370 vs. $60 worst case, they are spending 516% more to be "serverless." Edit: Yes Cloudflare has more than one server. Double the price and put one in the US and you still come out ahead. Edit 2: I'm not saying one way or the other is better. Just that the title is very clickbaity for a "put a credit card into a website" payoff.

schleck8 5y ago
That would be a single node though, in contrast to cloudflare's edge, and would probably slow down download speeds for everyone outside europe
walrus01 5y ago
An actually capable dedicated bare metal server with a good amount of RAM, CPU and disk, and 1Gbps will be more like $250-300/mo USD. This would be approximately equivalent in spec to what you could build for $2500 purchase cost if buying a 1U machine and colocating itself. For 30 bucks a month you'll get something very old and weak.
- mike_d 5y ago
  The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s. Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59 https://oneprovider.com/order/item/dediconf/59
  - 5y ago

deleted 5y ago

[deleted]

VWWHFSfQ 5y ago

Question: I want to make a live video streaming website a la Twitch.tv. How much would CloudFlare charge me to stream 8 Mbps to ~80,000 viewers for 4 hours?

jrockway 5y ago
I think this is the sort of thing you want to ask their sales team. I thought about doing this a while ago, after I left my job with an ISP. I would just have bought transit from them (and maybe made them regret their 10Gbps for $1000/month plan ;)
mike_d 5y ago
Almost $20,000. 19,200,000 minutes at $1 per 1,000 minutes. https://support.cloudflare.com/hc/en-us/articles/360016450871-Billing-for-Cloudflare-Stream https://support.cloudflare.com/hc/en-us/articles/36001645087...
- busymom0 5y ago
  There's probably some bulk deals if you get in touch with their sales team?
  - ignoramous 5y ago
    Yes, you could. The last time someone I know spoke to them, the standard enterprise plans started at $5000/mo and got you some very nice freebies. Cloudflare's also announced managed WebRTC APIs (in closed beta) which look promising for creating meeting rooms ala Zoom.us. I imagine, these are already available to tech-shops on enterprise plans. Though, mux.com is decent too, from what I've heard; for certain workloads, LiveStream and Vimeo might be cheaper. There's Peer5, PeerTube, and BitTorrent in the P2P space (among many such solutions).
    - busymom0 5y ago
      > mux.com First time hearing of them (due to my own ignorance probably). Any details on them?

qeternity 5y ago

This is undoubtedly better than most chuck-it-on-AWS types would achieve, but it's still quite a bit more expensive than could achieved with marginally more effort. They are already on Digital Ocean - if they used their managed database offering, they could slash their Firestore bill and get rid of their Argo bill, bringing the cost sub $200 (but - massively reduced returns on investment of effort).

willcipriano 5y ago

If you are worried about getting your blog post to the top of Reddit or Hacker News (I've never been there myself); you can have a very modest web server or even a pay per request serverless sort of thing and pay $20 real quick to Cloudflare if you happen to get popular. It's the Bart Simpson method of highly scalability[0], for static content you can have global datacenter coverage in a couple minutes or so if you use them for DNS to start with. It even works if the origin server goes down. [0]https://www.youtube.com/watch?v=aKU3hMvD31w https://www.youtube.com/watch?v=aKU3hMvD31w

PragmaticPulp 5y ago
> you can have a very modest web server or even a pay per request serverless sort of thing and pay $20 real quick to Cloudflare if you happen to get popular. I get the impression that a lot of the critics in this thread don't really understand Cloudflare, how cheap it is, or even the concept of CDNs in general. $20/month for Cloudflare Pro is a steal for what you get. Spinning up a dedicated server in a single datacenter somewhere isn't going to give the same results, especially if your users are geographically distributed like in this case.
- deleted 5y ago
  [deleted]
- encryptluks2 5y ago
  Don't you get tht same for free with GitLab Paged and last I checked Cloudflare was free too for caching static HTML assets?
  - CRConrad 5y ago
    > last I checked Cloudflare was free too for caching static HTML assets? If not free then very cheap, as I understood TFA: Wasn't that why they have two separate domains and serve static assets from one of them, to be able to use the cheapest Cloudflare tier for that domain = those assets?
- superkuh 5y ago

throwoutway 5y ago

Ironically, it's struggling to load the images for me.

busymom0 5y ago
Worked pretty smoothly for me when I browsed around assets.
soheilpro 5y ago
Same here in Indonesia.
eliseumds 5y ago
Loading instantly here (Sao Paulo).
Fornax96 5y ago
The page took 40 seconds to load for me in the Netherlands

technotarek 5y ago

We have an incredibly similar stack for attic.city, aside from the application layer. We lean heavily on CF (edge caching, Argo and page rules). But we’ve recently changed our image assets from S3 to CF R2. Anyone else put that side by side with Bunny?

Jamie9912 5y ago
What? how do you have access to R2?
- the_duke 5y ago
  It's in private beta.

IvanK_net 5y ago

I was running www.Photopea.com with 10M page views a month for $40 a year. Now, I upgraded to $60 a month. I never used any CDN.

IvanK_net 5y ago
I would love to use a CDN. But I am afraid of things I do not fully understand, and I am afraid of making things too complex. If I update a single file, how long it takes until nobody in the world can access the old version anymore? Does it take seconds / minutes / hours? Also, some files should not be cached at all (like PHP requests). I am afraid it would take me days or weeks to learn everything and to cofigure the CDN properly, and I am risking being offline for a part of the world during that time. Also, if there is a problem at the CDN, my website would be broken, too. If someone could help me, you can write me at support@photopea.com
TheRealSteel 5y ago
Ever since MacOS broke Gimp I use Photopea regularly! Fantastic site!
- aasasd 5y ago
  Please tell more about this ‘MacOS broke Gimp’ thing, because it's the first time I hear of that and I'd like to dodge a firsthand experience.
  - metadat 5y ago
    I used it successfully on my Mac last week. It's still GIMP (bleh!) but it works.
    - undefuser 5y ago
      You can consider Affinity Photo
      - 5y ago

sgtfrankieboy 5y ago

I don't believe Argo is still needed? Tiered Caching is now a free feature.

DenseComet 5y ago
Yep was also going to mention this. Obviously, Argo still helps with latency and some other things probably, but if they're paying the additional $160 just for tiered caching, they should be able to switch off Argo.

jason0597 5y ago

I often wonder why so many companies seem to spend tens of thousands of dollars per month on AWS services for their hosting. If most web apps nowadays are just CRUD apps, then why would you need to spend more than €300/month on hosting? All you'd need is data and the website itself, right? I'm currently clicking through various European hosting services and they seem to offer great dedicated servers at good prices. I cannot wrap my head around these ridiculous costs I keep hearing about. A guy I know was telling me how they were spending tens of thousands of dollars per month on AWS at his company. Is it because everyone is writing their stuff on node.js, putting 500kb of JS in every web page they serve, putting Docker everywhere when a chroot would suffice, microservices, Kubernetes, or hell even writing SAPs when we don't need them? I am genuinely confused

smarx007 5y ago
I don't think that most workloads are webapps and many webapps are not just CRUD. HN is not the web either, in case HN formed such a viewpoint for you.
ummonk 5y ago
Until you're spending the salary of a large fraction of your engineers on hosting costs, there are much bigger things to worry about than hosting costs. AWS is user friendly and a lot of engineers are familiar with it, so developer productivity is strongly in its favor. Also, you know what you're getting security-wise with AWS, and no one will blame you when your website / service goes down because AWS is down.
- oogetyboogety 5y ago
  Wasn't that like IBM's business motto at some point? No one ever got fired for going with IBM. Then IBM definitely lost a lot of business to cloud modernization. Maybe the next logical step is the cheaper CloudFlare, which another comment mentioned is actually pretty profitable and then I'm sure the next cheaper incumbent will come in with just slightly less trust and reliability.
jeppesen-io 5y ago

r1ch 5y ago

It's annoying how you have to work around Cloudflare's weird Argo pricing like this. Paying per GB for already-cached data that takes the same route as non-Argo traffic makes no sense. I believe it's also against Cloudflare's TOS to use it as an asset-hosting platform.

celdon25 5y ago

Awesome! I hope this means more competitors for YouTube soon, featuring videos that aren't just related to fringe politics.

cblconfederate 5y ago
You think cloudflare will be more tolerant?
jjcm 5y ago
Self shill: I'm working on https://non.io https://non.io, a reddit-meets-patreon platform that supports up to 8k video. Planning on launching in the next couple of months (just about got the stripe integration working for auto-payouts). One of my main reasons for making it was I was annoyed at YouTube lately. Would love feedback as I'm aiming to be a competitor. You can sign in with Username: hn@hn.com Password: hackernews - The site is currently in demo mode, and the db will be wiped before launch - feel free to sign in and poke around. Also it's hosted in Australia currently, so site may be a little slow for those in the US. Source is here: https://github.com/jjcm/soci https://github.com/jjcm/soci

deleted 5y ago

[deleted]

bcjordan 5y ago

Fun read, love finding those combination perf/cost wins! Have you considered using Firestore in Datastore mode[0]? It might make all of your reads free[1], though migrating could be a project. [0] https://cloud.google.com/datastore/docs/firestore-or-datastore https://cloud.google.com/datastore/docs/firestore-or-datasto... [1] https://stackoverflow.com/a/53313540 https://stackoverflow.com/a/53313540

ttty 5y ago

jjcm 5y ago

On a similar note - does anyone have a recommendation for a budget cdn for video? I don’t necessarily need encoding capabilities (though it’s a plus), but rather high storage/bandwidth options.

Ameo 5y ago
BunnyCDN (mentioned in the blog post) has this functionality. It is mentioned in the blog post and they're using it for image processing, but it works for videos too. I use it myself, but just as a normal static file CDN, but they have dedicated tabs in their UI for video stuff. Their bandwidth pricing is also very reasonable ($5/TB with their "bulk" option and $10/TB for standard)

gardnr 5y ago

Do the Bunny images go through Argo? If you were OK with your content only updating periodically you might be able to do away with Firestore, Argo, and Vultr. You definitely do a great job of mitigating db hits. Consider publishing all the data to a CDN using something like Gatsby. In this scenario your CI/CD would build a static website from all the assets and publish it to a static server. You mentioned that only your view counts are the only really dynamic part. You could just estimate and emulate those or design them a different way. Martin Fowler has an article on this: https://martinfowler.com/bliki/EditingPublishingSeparation.html https://martinfowler.com/bliki/EditingPublishingSeparation.h...

soheil 5y ago

> But, we can get around that because of a partnership between Backblaze and Cloudflare they call the Bandwidth Alliance. This reads like a paid marketing post for Cloudflare. It's astonishing how many times the author conflates browser cache-able assets with cached assets on a CDN. When a browser downloads a static asset if the web server is configured properly those files will be cached and there won't be a need to re-download them for a very long time. There is of course the issue with modern web frameworks like React generating a single massive js/css file that bundles everything all over again in a unique file busting all previous cached versions across all users' browsers just because a comma was added to a sentence. Keep your js/css files small, serve them from your own web server and set a reasonable expire header, no need to pay Cloudflare $40/mo to continue gatekeeping the internet.

systemvoltage 5y ago
I didn’t see it as an ad for Cloudflare unless I also then consider every single post about AWS in last 10 years on HN as ads. If anything, I’m fine with a new competitor to BigCloud that’s been unchecked and increasingly hostile (cost-wise).
DenseComet 5y ago
They provide 3D asset downloads, most folks will only be downloading an asset a single time. Therefore, browser side caching doesn't really help, they need to cache the files on a CDN.

alphachloride 5y ago

I tried looking, and I find it very irritating that in that whole web page, there is not one single link in the menu/footer/sidebar to polyhaven.com home page so I can click and discover what it actually is. Not one. This occurs on many company blogs as well operating under a subdomain like blog.whatever.com To be clear, this is a very tangential and irrelevant nitpick and I understand it does not contribute to the content of the website itself.

cerved 5y ago
I had no problems navigating there. Every single link in the hamburger lead me to polyhaven.com
- alphachloride 5y ago
  To the polyhaven.com home page? I don't think it did. They are all sub pages which adds another navigation step to see the root page, which is what I find irritating.
chillfox 5y ago
Not having a link forces people to navigate to the page on their own. For a lot of people that takes the form of a Google search. Lots of people googling specifically for your brand increases your search rankings. Or so goes the theory anyway.
helsinkiandrew 5y ago
Yes! - the same goes with support sites on different domains (support.whatever.com or whatever.zendesk.com etc) that appear in google results and don't have a link to the company.
- CrimsonRain 5y ago
  I think this is a result of app stores enforcing links to external digital products thing. That's why all these support pages which need to be linked from app, don't have a single link to main site.

helsontaveras18 5y ago

This website is one of the fastest web pages I’ve ever used. Although it’s simple, it immediately loads pages and pages of images. Like in a flash. I’m on mobile web and it’s such a snappy experience.

lelandfe 5y ago
If Poly Haven is reading this: you nearly have an amazing FCP on the models browser but the external normalize.css file is killing you.[0] Self-hosting would drastically improve your 75p paint time. I'd also encourage you load your fonts late via JS. Your main JS package competes right now with WOFF files from Google Fonts for priority and there's no need for that. [0] https://www.webpagetest.org/result/220106_BiDc42_428a3caec56095643c1f6786a854cc6a/1/details/#waterfall_view_step1 https://www.webpagetest.org/result/220106_BiDc42_428a3caec56...
- hansel_der 5y ago
  > I'd also encourage you load your fonts late via JS can't tell if joke, so: there are already enough sites that display content for a split second and then some script runs (or fails?) and there is either nothing on screen or an error message. this is ridiculous - please stop!
  - lelandfe 5y ago
    Loading experience for this would be that a fallback font loads first and is later replaced. If something errors, it’s not replaced. What you’re describing has been thankfully avoidable for many years.
    - hansel_der 5y ago
      this failure mode has been avoidable like forever and i have only seen it become more widespred in the last decade. be sure i blame ad-tech and analytics
- austinpena 5y ago

otar 5y ago

First of all, kudos, I’m a junkie for such optimizations myself. It’s a pleasure that your systems are optimized well, you sleep well and you’re proud :) Two thoughts: 1) CloudFlare offers incredible service. What would need the team of netadmins/sysadmins can be handled via their UI easily. I use Argo in one project and yes, it is a very efficient tool. CloudFlare is becoming like AWS, knowing its tools is a skill. Maybe one day well see an official CloudFlare Solution Architect certification?.. :)) 2) For such traffic heavy websites I don’t think there’s a better combination than Static Websites hosted over CDN, which is cached aggressively and for the dynamic part serverless as a backend. If you configure it properly (cache optimization, rate limiting to not get a huge paycheck if DDoSed, etc…) this setup is like set-up-and-forget-it, without the need to invest loads of resources into it.

xyexs 5y ago
I'm new to this stuff, but what the source article describes is essentially just that it is cheaper to run a static website with a CDN than without - right? I'm mostly curious because they described the cloudflare service as a cache layer rather than a CDN.
- collinmanderson 5y ago
  > They described the cloudflare service as a cache layer rather than a CDN. Not sure how most CDNs work, but for Cloudflare, content is not actively pushed to edge nodes, it’s only pulled to an edge when needed and then cached there usually for an hour or two. The caching is still distributed, and all of the tcp+https roundtrips are done to a local data center which makes things faster. > what the source article describes is essentially just that it is cheaper to run a static website with a CDN than without - right? Right. The cost savings comes from caching done by the CDN, and public static content is super easy to cache. The distributed part makes things faster, but doesn’t necessarily save money.

goldbattle 5y ago

My thought is why don't the authors use github pages to host their static site with their links to a dedicated server with an unmetered link since their whole project is CC0. Downloading assets doesn't need to be low latency, thus I don't think the caching spatially near the user is important. Having a user login for patreon is something that would be missing and require some more thought I admit. I think quite a lot of other people have mentioned in the thread that they are getting a lot of other "benefits" from using multiple services, but I don't see how these help solve the problem of data delivery besides taking advantage of the Cloudflare + Backblaze alliance which is $31 if their main website is a static one.

aardvarkr 5y ago
80TB/mo is quite a bit beyond the generosity of GitHub pages
- lorey 5y ago
  any source on that?
  - klohto 5y ago
    I mean, kinda obvious since everything is metered on GH. Soft limit is around 100GB.

otabdeveloper4 5y ago

Isn't this like 2 requests per second? That's massively overpriced, with current prices you should fit into a 5 dollar budget.

jijji 5y ago

I like the idea of cloudflare as a OSPF style file cache, but what about putting 80TB in a colo-facility on a 1U box and paying $80/month to host that yourself? is it cheaper to use Amazon S3?

indigodaddy 5y ago
I think you’ve got the 80TB item confused with storage. They are talking about 80 TB of transfer over the internet in a month. I doubt they have 80TB of assets laying around on Vercel. That would cost them an arm and a leg with Vercel I’d expect.

throw14082020 5y ago

> To avoid this problem in the future, I decided to splurge a bit and go for a cloud solution where I wouldn’t have to worry about reliability, performance, scaling or integrity ever again: Google Firestore. > Google Firebase is nice and convenient, but it is quite expensive. We could investigate some other managed database options in future. I've only seen people get annoyed with Firestore over time, and migrating out of it. People do end up worrying. At first, They seem to choose Firestore because it's strongly marketed and seems suitable for a new project. And then data modeling, high prices or scalability becomes a problem.

MichaelRazum 5y ago
What is a good solution compared to firebase ? I mean for a new project, alone something like user handling, registration, email sending, recovering passwords seem to cost some effort. Also be able to react to events, is kind of trivial in firebase. So is it really a bad choise (for the beginning)? Later you can always try to optimize.
- simonbarker87 5y ago
  SupaBase is a good alternative in my opinion but they are not as feature rich yet - but you do get SQL again which is delightful
- janto 5y ago
  Many use Firebase for auth and user management only.
- mellavora 5y ago
  so the problem here is if you have all of those needs, then really soon in the life of the project you are going to need good tooling to handle them. So choosing something which makes it "one-click" to set up but total madness to manage is a really short-term optimization, only worth it for a pure prototype which you will throw away no matter how successful. If you know you need those things to reach success, then it is better to make the up-front investment to get good tools for those. If you still want to go with a cloud provider, AWS Amplify has some interesting tooling. I've build products both against Amplify and Firestore (and Firebase). Yes, Firebase is a few days to a week faster to set up (integrated user management, as you say), but AWS gives more sophisticated control and is built around scripted deployments. You pay for it, of course, and I'm not arguing AWS vs running your own server. I am saying if the choice is AWS or Firebase, that a few days researching the choice would give you knowledge you could use for launching the next 10 prototypes you have in mind.

techie128 5y ago

80TB for 5M page views works out to 16.7MiB per page view. I assume you're serving images and other assets on the page otherwise you have room to optimize on the page weight.

semi-extrinsic 5y ago
Even if you are serving images that's very heavy for a single page. A 4K resolution jpeg sits at around 6-7 MiB.

martindbp 5y ago

As someone building on a mostly free service with lots of content this is very helpful, thank you! Will probably migrate from S3 to Blackblaze, seems like a great deal.

austinpena 5y ago
Depending on how much you access per month, Storj DCS[0] and Wasabi[1] can also be interesting choices. - Storj has the lowest cost per TB, but charges for egress - Wasabi doesn't charge for egress, but has a fair use policy where if you egress more than you store they can kick you off their platform[2] - Wasabi is also a better fit only if you plan to keep your files around for 90+ days (they have a minimum object retention period) - The bandwidth alliance is only available for HTML related content unless you're on a specific paid plan[3] [0] https://www.storj.io/ https://www.storj.io/ [1] https://wasabi.com/ https://wasabi.com/ [2] https://wasabi.com/paygo-pricing-faq/#free-egress-policy https://wasabi.com/paygo-pricing-faq/#free-egress-policy [3] https://www.cloudflare.com/terms/#28-limitation-on-serving-non-html-content https://www.cloudflare.com/terms/#28-limitation-on-serving-n... *opinion is mine, not my employer's

cblconfederate 5y ago

how is 5M page views a month a lot? I just looked and my tiny old hetzner server serves at least 30M per month. Presumably hetzner gives 1Gbit for free so that should be enough for 80T (0.2Gbit/month, though i dont have that much traffic). It cost $60

hffft 5y ago

mad182 5y ago

I handle ~150TB and 26M page views for ~$500 by simply renting a few dedicated servers at hetzner. And if I didn't need quite a lot of processing power (more than average website), it would be much lower. I only need so many servers for the CPU power, not traffic.

folli 5y ago
Any experience with other providers of dedicated servers, e.g. Digitalocean?
- mad182 5y ago
  Does DO even offer dedicated servers? I have used their vps in the past. It was ok, no issues, but more expensive than hetzners vps or dedicated for large traffic/cpu needs.
  - reducesuffering 5y ago
    Yes DO does. Generally slightly more expensive than Hetzner, with a few more AWS-like features.
- GOATS- 5y ago
  You cannot go wrong with Leaseweb or Nforce if you're looking for something in EU.
razemio 5y ago
This! Hetzner root servers, there is nothing else I know on the market that comes close in price/power ratio when you use the "Serverbörse". I run multiple low traffic websites, databases and a heavy traffic elk instance + my complete homelab in proxmox / docker for 48€ a month and there is even room for more. Highly recommended!
- SergeAx 5y ago
  What is your strategy for dealing with server failure?

kkjjkgjjgg 5y ago

When I tried CloudFlare, they injected a JavaScript into my site that was bigger than my entire site. That was a couple of years ago, but I can't bring myself to trust them anymore. I also don't like their growing control over the web.

BiteCode_dev 5y ago

Honestly you can go very far with unmettered data plans. We consume 2GB/S, maxed out, which ends up being 5184TB a month. We have millions on users a day, all on a streaming video platform. It cost less than $2K/Month. The cloud is crazy expensive. Private servers are beasts, and they are cheap. Of course, for this price, you don't have redundancy and horizontal scaling. You also don't have to maintain and debug a system with redundancy and horizontal scaling.

nix23 5y ago
Thanks! Exactly that! I always tell that customers and their response is: -But google/facebook/amazon... -But uptime needs to be 99.999 -But everyone uses cloud Most businesses are not a trading-market, have less then 100 peoples (aka you are probably not another amazon), and no bonus using a cloud/kubernetes etc. But it's the same old story, in the 00's i used the ~same arguments against buying OracleDB ;)
- CRConrad 5y ago
  > But it's the same old story, in the 00's i used the ~same arguments against buying OracleDB ;) But no matter how logically convincing your arguments were, most of the time upper manglement just went on buying Oracle, right...?
- XCSme 5y ago
  > uptime needs to be 99.999 Isn't AWS down like every two months for a few hours? That's far off the 99.999% mark. No one can guarantee 100% uptime and sometimes it's even better to have that under your control (eg. have a dedicated server and a backup one from different providers). My point is that, if you want the highest possible uptime, you shouldn't rely on a single (cloud) provider.
- Cthulhu_ 5y ago
  And you can tell them all of those are possible, but they need to have a massive budget, not just for the monthly bills but also to hire experts to set things up with those constraints. I worked at a company once that, from higher up, said that they had to have five nines of uptime. We had some really good cloud engineers there (one guy set up a server / internet container for the military in Afghanistan; in hindsight he said they should've just sent a container of porn dvd's), and they really went to town. For that five nines uptime, you're already pretty much required to set up your infrastructure to use multiple availability zones, everything redundant multiple times, etc. Of course, the actual software we wrote was just a bunch of CRUD services written in nodejs (later scala because IDK), on top of a pile of shit java that abstracted away decades of legacy mainframes.

dmitrygr 5y ago

5M views per month is under 2 per second. In 2005, that was servable from a 16-bit pic microcontroller with a 10-base-t phy from microchip inc

sgt 5y ago

How are Polyhaven funding their staff? Their Patreon income won't even cover a tenth. Do they have corporate sponsors or special agreements?

twblalock 5y ago

When I see per month or per day stats I get suspicious. Let's say a month has 30 days -- 5 million views a months is 166,666k per day, 6944 per hour, 116 per minute, and ~1.92 per second. 1.92 qps. Of course it's not expensive, it's a tiny amount of traffic!

PUSH_AX 5y ago
To look at a different angle, I would imagine the traffic isn't evenly distributed, and part of being happy with your offering will be if things don't grind to a halt during peaks.
Karishma1234fff 5y ago
I know, but it is impressive that they are doing it for $400.
- richardwhiuk 5y ago
  It's impressive it takes that much money to do almost no traffic. A raspberry PI could handle the compute.
  - andyp-kw 5y ago
    Only if the traffic is evenly distributed during the day. Which it's almost certainly not.
    - richardwhiuk 5y ago
      The traffic graphs in their blog look pretty stable. Maybe 1.5x, but not hugely uneven.
    - netr0ute 5y ago
      You can get a $15 2.5 gigabit ethernet adapter for that Raspberry and now you're set for peak loads.
- deleted 5y ago
  [deleted]

bvm 5y ago

> This website blog.polyhaven.com/how-we-handle-80tb-and-5m-page-views-a-month-for-under-400/ is currently offline. Cloudflare's Always Online™ shows a snapshot of this web page from the Internet Archive's Wayback Machine. To check for the live version, click Refresh. hm

leke 5y ago
Yep, I got that too. Absolutely beautiful.
iso1631 5y ago
The blog's down, but polyhaven.com isn't. Presumably the blog isn't as important as the core service and thus isn't hosted on the same infrastructure

nuker 5y ago

They dead now? Btw 80TB/mo = 30 MB/s, 5M/mo = 2 views per second, not mind blowing

darkwater 5y ago

The origin website is currently down according to Cloudflare...

johnyreader 5y ago

[flagged]

AtNightWeCode 5y ago

Worked on several sites with similar setup. Works great and is affordable. One thing to be aware of is that sometimes you can get at a lot of cache misses. Therefor the back-end services must not stop working completely if flooded with requests.

deleted 5y ago

[deleted]

imdsm 5y ago

s/handle/handled

HeavyStorm 5y ago

Well, it's down now. Foul play?

reph2097 5y ago

Your blog won't load after waiting for like 10 seconds. Funny.

xurukefi 5y ago

mrintellectual 5y ago
Indeed not the greatest advertisement for their infrastructure, but regardless you have to admit that <$400 is fairly impressive.
anatolinicolae 5y ago
Aged well I see...
gionn 5y ago
I bet it's powered by wordpress.

Gepsens 5y ago

Hey look here guys it's cheap to do html !

ThinkBeat 5y ago

I dont understand this entirely . A lot of the article is spent talking about how great and fast Cloudflare is. Then at the end he says he uses Bunny as the CDN for images. Why use Bunny instead of Cloudflare? One more moving part. I presume it is cost related?

spiffytech 5y ago
I too wish they'd spelled out that choice. I speculate it's because Cloudflare's TOS prohibits you from using their CDN to serve "video or a disproportionate percentage of pictures, audio files, or other non-HTML content" on typical plans. See section 2.8 on https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ . Since PolyHaven seems to be a purveyor purveyor of 3D assets and they use Bunny to host "all of our images shown on the website (thumbnails, renders, previews, etc.)", I'm guessing a lot of their assets exceed Cloudflare's TOS. PolyHaven mentions using Bunny's image optimization service, so that'd factor into the decision too.

maxclark 5y ago

The important thing here is caching/static delivery I've had this argument too many times to count. Every page view does not need to be dynamic per user request. Creating a sane cache policy reduces origin resources, servers, cost, etc... and surprise gives a better user experience at the same time.