8 ms·
Writing a Mini-CDN to Learn Nginx/Prometheus/Grafana/Lua
- mnutt 4y agoThis is nicely written, and a lot of it mirrors my experience using nginx as a pseudo-cdn. Another area worth exploring might be http3, ssl session caching, and general latency/ttfb optimizations.
- zspace2 4y agoVery good project. thanks for sharing
- chrsig 4y agoI'm curious if any HNers have opinions on prometheus vs other time series databases like influxdb? I periodically consider a grafana & backend setup for when datadog becomes cost prohibitive for metrics with several tags.
- firstSpeaker 4y agoGo with Mimir. It is Prometheus compatible and horizontally scalable for read/write path separately. Mimir: https://github.com/grafana/mimir https://github.com/grafana/mimir
- deleted 4y ago[deleted]
- flyingsky 4y agoYou did not answer OPs question tho'. prometheus vs influxDB.
- beardedetim 4y agoAt $dayjob we're considering replacing DataDog with Grafana and friends, already using it elsewhere to great affect. Haven't used influxdb yet so can't speak as a comparison but from my usage, I'm sold on Grafana, Loki, Prometheus, and friends over DataDog. It mixed with OTel have been a real pleasure to use.
- therealdrag0 4y agoWe’ve done that migration at $dayjob. Has taken a lot of getting used to. The data model is different (perhaps poor setting choices) causing some wacky query requirements for our charts which devs often don’t understand to misleading charts. We’re slowly switching to OTEL which should solve that problem. Otherwise there’s also much to be desired in how our grafana behaves interims of saving preferences (ours doesn’t; stateless head? I forget why) which is annoying. Also the tracing/APM is missing out of the box though some teams are working on getting that going too. So a classic build vs buy story I guess. Probably will pay less in the long run but for a worse and rocky road experience so far.
- xiwenc 4y agoWe have been using prometheus at a client for little over a year now. Since we need to keep metrics for years, prometheus cannot seem to be able to deal with it well. One behavior we observed is it crashes consistently in k8s. We couldn’t pin down the root cause but suspect it’s the amount of metrics we collect continuously and keep (archive). Now we are considering to switch to thanos or mimir.
- jeacken 4y agoAnother example of a project duped into thinking Lua is “powerful”. It is small. That is it. Lua has near zero useful functionality and makes the developer repeatedly reinvent functionality over and over and over again. https://media1.giphy.com/media/TFO2mwVPIFoOJcuTSC/giphy.gif https://media1.giphy.com/media/TFO2mwVPIFoOJcuTSC/giphy.gif
- klelatti 4y agoWould you like to expand on why you think Lua is a bad choice for this particular project and what you would have used instead. That would be much more helpful than a generic attack on the language itself.
- giraffe_lady 4y agoI think they were as specific as they need to be: "Lua has near zero useful functionality and makes the developer repeatedly reinvent functionality over and over and over again." That may feel uselessly broad if you're not familiar with the language or ecosystem but I've used lua professionally for years on projects of all different sizes and it's a truth I recognize.
- deleted 4y ago[deleted]
- berkut 4y agoIt's small, fast, and doesn't have a GIL lock, so concurrent executions are trivial.
- giraffe_lady 4y agoYes it's good if you're writing a C app and need to embed a concurrent scripting language with long-running tasks, and for some reason you can't write that part in C and just expose it to the script env, and also you can't use one of the three or four alternatives that are still a better choice for this specific scenario.
- daniels1006 4y agoGreat content, helpful and inspiring. Thanks!
- asjkaehauisa 4y agoWhy didn't you use varnish for that?
- tecleandor 4y agoI guess it's "...to Learn Nginx/Prometheus/Grafana/Lua". Per the first line of the link: "The objective of this repo is to build a body of knowledge on how CDNs work by coding one from "scratch". "
- sodez117 4y agoGood read. Is there something similar for building a DDoS protection feature? Like Cloudflare?
- friendlyHornet 4y agoThanks for this
- dreampeppers99 4y agomy pleasure
- sandGorgon 4y agothis is very very cool! One thing i would definitely like to see is domain name resolution. Shopify, Dukaan, Vercel all make a big deal out of it ...going all the way to BGP. https://twitter.com/subhashchy/status/1536769406801309696 https://twitter.com/subhashchy/status/1536769406801309696
- vitorbaptistaa 4y agoBeautifully written! Thanks for sharing, Leandro.
- dreampeppers99 4y ago<3
- hardwaresofton 4y agoIt would be nice to discuss the common approaches to global name resolution —- anycast vs geo-routing.
- wrigby 4y agoIIRC the industry standard is to serve your authoritative DNS with anycast, and have those servers do geo-based dns resolution to shift HTTP traffic to a nearby edge POP.
- xmorse 4y agoThe hard part of building a CDN is scaling it. The best approach imo is to use fly.io to host an anycast IP (with horizontal scaling) and store cache files on disk Fly.io also has a Grafana dashboard built in for your machines
- jay6282 4y agoThe hard part of building a CDN is to know when you need it. 99.9% of all websites with CDN do not need it. Serving static files consumes so little resources that a single server can serve billions of users as long as you dont use script for serving the file. The most cost-effective with also the lowest latency solution is to never use CDN. If your webserver provider charge you a lot for traffic you are better off using another provider.
- latchkey 4y ago> The most cost-effective with also the lowest latency solution is to never use CDN. CloudFlare is free at my tier and gives me the ability to have the lowest latency.
- youngtaff 4y ago> The most cost-effective with also the lowest latency solution is to never use CDN The lowest latency solution is to put the content near the user and a CDN is probably the easiest way of doing that if someone needs to server a geographically dispersed audience
- nine_k 4y agoStatic files can be relatively large, like pictures, or sounds, or even videos. At busier moments, the p99 latency can become pretty bad if you only have one virtual NIC serving them. Also, geography: ping time from Singapore to the US can never be negligibly short, which is pretty noticeable during TLS handshakes. But yes, likely these considerations do not matter for 90% of websites (not 99.9% though).
- 411111111111111 4y agoIt's definitely 99.9% if the websites aren't filtered at all. A lot lower if you're only taking about the top 100 visited websites according to Alexa or apple however...
- sokoloff 4y agoIn OP’s premise, they were filtered by “websites who chose to be served by a CDN”.
- nnadams 4y agoIs it possible for CDNs to cache per URL per user? I'm thinking of something like /favorites where one URL would list something different for everyone. When I've setup caching on backend it was keyed off the user. This was a very informative read!
- jay6282 4y agoUser-aware CDN would require scripting of some kind to handle sessions. However, if the data is not sensitive you could use random string uris to publicly available files. That way it is difficult to guess/brute force the url to the files. (sensitive=person identifiable data)
- mnutt 4y agoMany CDNs support caching based on a particular cookie value, incorporating it into the cache key. I’d just be extra careful, the worst case for many server settings is an inoperable service but choosing the wrong cache key can easily result in a data leak. (serving one user’s response to another user)
- Matthias247 4y agoYou can configure whether the cache key includes a particular header or query parameter in a lot of CDNs. So as long as your user identify is transmitted in one of those, it would work.
- rmetzler 4y agoI don't know why you want to hurt yourself. If these are public, put them on /favorites/$USERNAME or something similar. If they are private, don't cache them. You can cache with specific headers as cache keys, but I would advise against doing this too much / abusing it. It really makes caching complicated. And from a data privacy standpoint it's better to opt-in into caching. I've witnessed incidents where visitors saw the private profile page of another user, because it was cached in the CDN.
- nesarkvechnep 4y agoYou can use the `Vary` header.