5 ms·

> with the availability of Cloudflare, running high-traffic websites is much more cost effective. sidetrack but how does cloudflare make things cost effective

> with the availability of Cloudflare, running high-traffic websites is much more cost effective. sidetrack but how does cloudflare make things cost effective? wouldn't it be cheaper if i just hosted the wiki on a simple vps?

Ambroos 2y ago
If you can run your application on Cloudflare Pages / Workers with Cloudflare's storage/DB things, it really gets dirt cheap (if not free) and very fast. And even without that, Cloudflare's caching CDN is very good, very cheap and very easy.
bombcar 2y ago
Ten years ago bandwidth was expensive. Still is, even if not as much. A simple VPS gets overwhelmed, but a simple VPS behind cloudflare can do quite well.
- thinkmassive 2y ago
  s/cloudflare/a CDN/
citricsquid 2y ago
More than a decade has passed since then so I am stretching my memory. At peak we were serving in the region of 10 million page views per day which made us one of the most popular websites on the internet (Minecraft was a phenomenon and every Minecraft player needed the wiki). We were probably the highest traffic Wiki after Wikipedia. Nowadays Cloudflare could absorb most traffic because of the highly cacheable nature of it, but at the time, Cloudflare didn't exist, and every request hit our servers.
- owyn 2y ago
  Yeah, Wikia in aggregate was in the top 50, maybe a top 20 site at various points. Wikia was built on caching. From my memory, about 99% of page views hit some kind of cache. If that dropped down to 97%, servers started to suffer. It's good to remember that the Fastly CDN company is a spinoff of Wikia, it was developed internally there first. Without that (varnish cache plus lots of memcache) Wikia would not have been able to handle the traffic. Mediawiki is horribly inefficient and one reason why Wikia was attractive as a host was that we had figured out a bunch of tricks to run it efficiently. The default configuration of mediawiki/wikipedia is real bad. Bigger independent wikis just couldn't handle the scale and many of the best independent wikis moved there for that reason. Just as one example, every link/url on a page hits a hook/callback that can call into an extension literally anywhere in the code base, which was several million lines of PHP code. I remember the "Batman" page on the DC wiki used to take several minutes to render a new copy if it fell out of the cache. That was one page I used for performance optimization tests. The muppet wiki and the lyrics wiki also had huge performance issues and fixing them was some of the most fun engineering work I've done. Every useful feature had some kind of horrible performance side effect, so it was always a fun puzzle. I also hate landing on a Fandom wiki now but thanks to the actual editors, it's still got some good content.
  - immibis 2y ago
    How much of your server load was Grand Exchange Market Watch?
- BlueTemplar 2y ago
  I bet, being by some counts the most popular video game ever - but which also makes it kind of a bad example to use when talking about wikis. By definition, very few wikis will have to deal with becoming one of the most popular websites. (And as you say, at that point one should be able to figure out funding.)
pjc50 2y ago
Cloudflare get the best deals on bandwidth. It will usually be cheaper to serve a terabyte from Cloudflare than to do it yourself: you could probably run the wiki on the free plan!
- diggan 2y ago
  > Cloudflare get the best deals on bandwidth. If you want to pay for bandwidth then yeah, CloudFlare is a great option. Otherwise, if you like the experience of not paying per GB/TB, go for a dedicated server with unmetered connection that has the same price every month, regardless.
  - KomoD 2y ago
    You don't need to pay anything to run TBs through Cloudflare, you could use the free plan. Rent VPS or managed hosting or host wherever you want, proxy it with Cloudflare on the free plan, Cloudflare caches it.
  - robertlagrant 2y ago
    It's more like: if you have a website that (sometimes) gets a lot of traffic, do you want Cloudflare to cache it and serve it with very few hits to your cheap server, or do you want your compute costs to expand to cope with the requests?
    - diggan 2y ago
      > do you want Cloudflare to cache it and serve it with very few hits to your cheap server, or do you want your compute costs to expand to cope with the requests? Usually you have something like a platform/tool/service that is mostly static requests that could be cached, with some dynamic requests that couldn't, as they're CRUD requests or similar. If your struggling to serve static content, then do go ahead and slap Cloudflare on top of that bad boy and probably your visitors will be a bit happier, instead of upgrading from a cheap VPS. If you're struggling to serve the dynamic requests, Cloudflare/CDN won't matter because these things actually need to be processed by your backend. So instead of trying to shave 50ms off from my simple static requests with a CDN, I'd much happier to optimize for all the requests, including the "dynamic requests" that need to hit the backend anyway. I'll still go for a dedicated server with proper connection and performance rather than a shitty cheap VPS with a CDN in front off it.
pornel 2y ago
Cloudflare caches pages at many many datacenters, often colocated with large ISPs. This lets Cloudflare deliver pages from their local cache over local links (which is fast and cheap), instead of fetching the data every time across the world from wherever the VPS is located.

Ambroos 2y ago

If you can run your application on Cloudflare Pages / Workers with Cloudflare's storage/DB things, it really gets dirt cheap (if not free) and very fast. And even without that, Cloudflare's caching CDN is very good, very cheap and very easy.

bombcar 2y ago

Ten years ago bandwidth was expensive. Still is, even if not as much. A simple VPS gets overwhelmed, but a simple VPS behind cloudflare can do quite well.

thinkmassive 2y ago
s/cloudflare/a CDN/

citricsquid 2y ago

More than a decade has passed since then so I am stretching my memory. At peak we were serving in the region of 10 million page views per day which made us one of the most popular websites on the internet (Minecraft was a phenomenon and every Minecraft player needed the wiki). We were probably the highest traffic Wiki after Wikipedia. Nowadays Cloudflare could absorb most traffic because of the highly cacheable nature of it, but at the time, Cloudflare didn't exist, and every request hit our servers.

owyn 2y ago
Yeah, Wikia in aggregate was in the top 50, maybe a top 20 site at various points. Wikia was built on caching. From my memory, about 99% of page views hit some kind of cache. If that dropped down to 97%, servers started to suffer. It's good to remember that the Fastly CDN company is a spinoff of Wikia, it was developed internally there first. Without that (varnish cache plus lots of memcache) Wikia would not have been able to handle the traffic. Mediawiki is horribly inefficient and one reason why Wikia was attractive as a host was that we had figured out a bunch of tricks to run it efficiently. The default configuration of mediawiki/wikipedia is real bad. Bigger independent wikis just couldn't handle the scale and many of the best independent wikis moved there for that reason. Just as one example, every link/url on a page hits a hook/callback that can call into an extension literally anywhere in the code base, which was several million lines of PHP code. I remember the "Batman" page on the DC wiki used to take several minutes to render a new copy if it fell out of the cache. That was one page I used for performance optimization tests. The muppet wiki and the lyrics wiki also had huge performance issues and fixing them was some of the most fun engineering work I've done. Every useful feature had some kind of horrible performance side effect, so it was always a fun puzzle. I also hate landing on a Fandom wiki now but thanks to the actual editors, it's still got some good content.
- immibis 2y ago
  How much of your server load was Grand Exchange Market Watch?
BlueTemplar 2y ago
I bet, being by some counts the most popular video game ever - but which also makes it kind of a bad example to use when talking about wikis. By definition, very few wikis will have to deal with becoming one of the most popular websites. (And as you say, at that point one should be able to figure out funding.)

pjc50 2y ago

Cloudflare get the best deals on bandwidth. It will usually be cheaper to serve a terabyte from Cloudflare than to do it yourself: you could probably run the wiki on the free plan!

diggan 2y ago
> Cloudflare get the best deals on bandwidth. If you want to pay for bandwidth then yeah, CloudFlare is a great option. Otherwise, if you like the experience of not paying per GB/TB, go for a dedicated server with unmetered connection that has the same price every month, regardless.
- KomoD 2y ago
  You don't need to pay anything to run TBs through Cloudflare, you could use the free plan. Rent VPS or managed hosting or host wherever you want, proxy it with Cloudflare on the free plan, Cloudflare caches it.
- robertlagrant 2y ago
  It's more like: if you have a website that (sometimes) gets a lot of traffic, do you want Cloudflare to cache it and serve it with very few hits to your cheap server, or do you want your compute costs to expand to cope with the requests?
  - diggan 2y ago
    > do you want Cloudflare to cache it and serve it with very few hits to your cheap server, or do you want your compute costs to expand to cope with the requests? Usually you have something like a platform/tool/service that is mostly static requests that could be cached, with some dynamic requests that couldn't, as they're CRUD requests or similar. If your struggling to serve static content, then do go ahead and slap Cloudflare on top of that bad boy and probably your visitors will be a bit happier, instead of upgrading from a cheap VPS. If you're struggling to serve the dynamic requests, Cloudflare/CDN won't matter because these things actually need to be processed by your backend. So instead of trying to shave 50ms off from my simple static requests with a CDN, I'd much happier to optimize for all the requests, including the "dynamic requests" that need to hit the backend anyway. I'll still go for a dedicated server with proper connection and performance rather than a shitty cheap VPS with a CDN in front off it.

pornel 2y ago

Cloudflare caches pages at many many datacenters, often colocated with large ISPs. This lets Cloudflare deliver pages from their local cache over local links (which is fast and cheap), instead of fetching the data every time across the world from wherever the VPS is located.