7 ms·

OpenFreeMap survived 100k requests per second

609 points by hyperknot 1y ago

colinbartlett 1y ago
Thank you for this breakdown and for this level of transparency. We have been thinking of moving from MapTiler to OpenFreeMap for StatusGator's outage maps.
- hyperknot 1y ago
  Feel free to migrate. If you ever worry about High Availability, self-hosting is always an option. But I'm working hard on making the public instance as reliable as possible.
v5v3 1y ago
The article mentions Cloudflare, so how much of this was cached by them?
- do_anh_tu 1y ago
  Do you even read the article?
  - keketi 1y ago
    Are you new? Nobody actually reads the articles.
    - LorenDB 1y ago
      False. I almost never upvote an article without reading it, and half of those upvotes are because I already read something similar recently that gave me the same information.
      - eszed 1y ago
        I'll submit in the second case (already read something similar) that, properly speaking, we should read both, and upvote (or submit, if not already here) the better of the articles. Not that, you know, I often take the time to do that, either - but it would improve the site and the discussions if we all did.
  - jwilk 1y ago
    From the HN Guidelines <https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html>: > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
    - RandomBacon 1y ago
      That guideline is decent I guess. I am disappointed that they edited another guideline for the worse: > Please don't comment about the voting on comments. It never does any good, and it makes boring reading. It used to just say, don't complain about voting. If the number of votes are so taboo, why do they even show us the number or user karma (and have a top list)?
      - RandomBacon 1y ago
        We can't even talk about the guidelines?
- alessandroberna 1y ago
  99.38%
fnord77 1y ago
sounds like they survived 1,000 reqs/sec and the cloudflare CDN survived 99,000 reqs/sec
LoganDark 1y ago
> I believe what is happening is that those images are being drawn by some script-kiddies. Oh absolutely not. I've seen so many autistic people literally just nolifing and also collaborating on huge arts on wplace. It is absolutely not just script kiddies. > 3 billion requests / 2 million users is an average of 1,500 req/user. A normal user might make 10-20 requests when loading a map, so these are extremely high, scripted use cases. I don't know about that either. Users don't just load a map, they look all around the place to search for and see a bunch of the art others have made. I don't know how many requests is typical for "exploring a map for hours on end" but I imagine a lot of people are doing just that. I wouldn't completely discount automation but these usage patterns seem by far not impossible. Especially since wplace didn't expect sudden popularity so they may not have optimized their traffic patterns as much as they could have.
- nemomarx 1y ago
  There are some user scripts to overlay templates on the map and coordinate working together, but I can't imagine that increases the load much. What might is that wplace has been struggling under the load and you have to refresh to see your pixels placed or any changes and that could be causing more calls an hour maybe?
- Karliss 1y ago
  Just scrolled around a little bit 2-3minutes with network monitor open. That already resulted in 500requests, 5MB transferred (after filtering by vector tile data). Not sure how many of those got cached by browser with no actual requests, cached by browser exchanging only headers or cached by cloudflare. I am guessing that the typical 10-20 requests/user case is for embedded map fragment like those commonly found in contact page where most users don't scroll at all or at most slightly zoom out to better see rest of city.
charcircuit 1y ago
>Nice idea, interesting project, next time please contact me before. It's impossible to predict that one's project may go viral. >As a single user, you broke the service for everyone. Or you did by not having a high enough fd limit. Blaming sites when using it too much when you advertise there is no limit is not cool. It's not like wplace themselves were maliciously hammering the API.
- columb 1y ago
  You are so entitled... Because of you most nice things have "no limits but...". Not cool stress testing someone's infrastructure. Not cool. The author of this post is more than understanding, tried to fix it and offered a solution even after blocking them. On a free service. Show us what you have done.
  - charcircuit 1y ago
    >You are so entitled That's how agreements work. If someone says they will sell a hamburger for $5, and another person pays $5 for a hamburger, then they are entitled to a hamburger. >On a free service. It's up to the owner to price the service. Being overwhelmed by traffic when there are no limits is not a problem limited only to free services.
    - perching_aix 1y ago
      > Do you offer support and SLA guarantees? > > At the moment, I don’t offer SLA guarantees or personalized support. From the website.
    - eszed 1y ago
      Sure, and if you bulk-order 5k hamburgers the restaurant will honor the price, but they'll also tell you "we're going to need some notice to handle that much product". Perfect analogy, really. This guy handled the situation perfectly, imo.
      - charcircuit 1y ago
feverzsj 1y ago
So, OFM was hit by another Million Dollar Homepage for kids.
eggbrain 1y ago
Limiting by referrer seems strange — if you know a normal user makes 10-20 requests (let’s assume per minute), can’t you just rate limit requests to 100 requests per minute per IP (5x the average load) and still block the majority of these cases? Or, if it’s just a few bad actors, block based on JA4/JA3 fingerprint?
- hyperknot 1y ago
  What if one user really wants to browse around the world and explore the map. I remember spending half an hour in Google Earth desktop, just exploring around interesting places. I think referer based limits are better, this way I can ask high users to please choose self-hosting instead of the public instance.
- toast0 1y ago
  Limiting by referrer is probably the right first step. (And changing the front page text) You want to track usage by the site, not the person, because you can ask a site to change usage patterns in a way you can't really ask a site's users. Maybe a per IP limit makes sense too, but you wouldn't want them low enough that it would be effective for something like this.
jspiner 1y ago
The cache hit rate is amazing. Is there something you implemented specifically for this?
- hyperknot 1y ago
  Yes, I designed the whole path structure / location blocks with caching in mind. Here is the generated nginx.conf, if you are interested: https://github.com/hyperknot/openfreemap/blob/main/docs/assets/nginx.conf https://github.com/hyperknot/openfreemap/blob/main/docs/asse...
rtaylorgarlock 1y ago
Is it always/only 'laziness' (derogatory, i know) when caching isn't implemented by a site like wplace.live ? Why wouldn't they save openfreemap all the traffic when a caching server on their side presumably could serve tiles almost as fast or faster than openfreemap?
- deleted 1y ago
  [deleted]
- deleted 1y ago
  [deleted]
- VladVladikoff 1y ago
  I actually have a direct answer for this: priorities. I run a fairly popular auction website and we have map tiles via stadia maps. We spend about $80/month on this service for our volume. We definitely could get this cost down to a lower tier by caching the tiles and serving them from our proxy. However we simply haven’t yet had the time to work on this, as there is always some other task which is higher priority.
  - latchkey 1y ago
    Like reading and commenting on HN articles! ;-)
- markerz 1y ago
  It looks like a fun website, not a for-profit website. The expectations and focus of fun websites is more to just get it working than to handle the scale. It sounds like their user base exploded overnight, doubling every 14 hours or so. It also sounds like it’s other a solo dev or a small group based on the maintainers wording.
  - thunderfork 1y ago
    [dead]
- hyperknot 1y ago
willsmith72 1y ago
so 96% availability = "survived" now? but interesting write-up. If I were a consumer of OpenFreeMap, I would be concerned that such an availability drop was only detected by user reports
- ndriscoll 1y ago
  If I were a consumer of a free service from someone who will not take your money to offer support or an SLA (i.e. is not trying to run a business), I would assume there's little to no monitoring at all.
- timmg 1y ago
  96% during a unique event. I think you would typically consider long term in a stat like that. Assuming it was close to 100% the rest of the year, that works out to 99.97% over 12 months.
sour-taste 1y ago
Since the limit you ran into was number of open files could you just raise that limit? I get blocking the spammy traffic but theoretically could you have handled more if that limit was upped?
- hyperknot 1y ago
  I've just written my question to the nginx community forum, after a lengthy debugging session with multiple LLMs. Right now, I believe it was the combination of multi_accept + open_file_cache > worker_rlimit_nofile. https://community.nginx.org/t/too-many-open-files-at-1000-req-sec/5796?u=hyperknot https://community.nginx.org/t/too-many-open-files-at-1000-re... Also, the servers were doing 200 Mbps, so I couldn't have kept up _much_ longer, no matter the limits.
  - ndriscoll 1y ago
    One thing that might work for you is to actually make the empty tile file, and hard link it everywhere it needs to be. Then you don't need to special case it at runtime, but instead at generation time. NVMe disks are incredibly fast and 1k rps is not a lot (IIRC my n100 seems to be capable of ~40k if not for the 1 Gbit NIC bottlenecking). I'd try benchmarking without the tuning options you've got. Like do you actually get 40k concurrent connections from cloudflare? If you have connections to your upstream kept alive (so no constant slow starts), ideally you have numCores workers and they each do one thing at a time, and that's enough to max out your NIC. You only add concurrency if latency prevents you from maxing bandwidth.
    - hyperknot 1y ago
      Yes, that's a good idea. But we are talking about 90+% of the titles being empty (I might be wrong on that), that's a lot of hard links. I think the nginx config just need to be fixed, I hope I'll receive some help on their forum.
      - ndriscoll 1y ago
        You could also try turning off the file descriptor cache. Keep in mind that nvme ssds can do ~30-50k random reads/second with no concurrency, or at least hundreds of thousands with concurrency, so even if every request hit disk 10 times it should be fine. There's also kernel caching which I think includes some of what you'd get from nginx's metadata cache?
perching_aix 1y ago
Haven't worked with Cloudflare yet first hand, and I'm not familiar with web map tech. But if the site really is pretty much just serving lots of static files, why is Hetzner in the loop? Wouldn't fully migrating to Cloudflare Pages be possible?
- internetter 1y ago
  The tiles need to be rendered. Yes frequent tiles can be cached but you already have a cache… it’s Cloudflare. Theoretically you could port the tileserver to Cloudflare pages but then you’d need to… port it… and it probably wouldn’t be cheaper
  - perching_aix 1y ago
    Oh interesting, okay. For some reason I had the impression that the tiles were static and rendered offline.
  - hyperknot 1y ago
    They are actually static files. There is just too many of them, about 300 million. You cannot put that in Pages.
    - jonathanlydall 1y ago
      Is CloudFlare’s R2 an option for you?
      - piperswe 1y ago
        That, or enabling Cache Reserve to automatically have a global cache in R2 instead of only caching on the PoP-level (disclaimer/source: I am a CF employee)
  - hoppp 1y ago
    It would cost a lot. Hetzner is hardware and them you can hammer it, free bandwidth. You get a very good server for cheap. cloudflare would be pay per request, a hefty sum if ddos happens
andai 1y ago
From the screenshot I wanted to say, couldn't this be done on a single VPS? Seemed over engineered to me. Then I realized the silly pixels are on top of a map of the entire earth. Dang! I'm curious what the peak req/s is like. I think it might be just barely within the range supported by benchmark-friendly web servers. Unless there's some kind of order of magnitude slowdowns due to the nature of the application. Edit: Looks like about 64 pixels per km (4096 per km^2). At full color uncompressed that's about 8TB to cover the entire earth (thinking long-term!). 10TB box is €20/month from Hetzner. You'd definitely want some caching though ;) Edit 2: wplace uses 1000x1000 px pngs for the drawing layer. The drawings load instantly, while the map itself is currently very laggy, and some chunks permanently missing.
- TylerE 1y ago
  "€20/month from Hetzner" is great until you actually need it to be up and working when you need it.
  - immibis 1y ago
    IME Hetzner's not unreliable. I don't think you could serve 100k requests per second on a single VPS though. (And with dedicated, you're on the hook for redundancy yourself, same as any dedicated.)
    - cyberpunk 1y ago
      They’re unreliable as soon as you have to deal with their support who have the technical knowledge of a brick. And as soon as you have to do ant business / deal with the german side of the business expect everything to slow down to 2 weeks for response which will still be incorrect. They are simply not worth the hassle. Go with a competent host.
      - Aeolun 1y ago
        > They’re unreliable as soon as you have to deal with their support who have the technical knowledge of a brick. Since I never have to, that’s perfect isn’t it? If you need support from Hetzner you are using the wrong host.
Starlevel004 1y ago
> I believe what is happening is that those images are being drawn by some script-kiddies. If I understand correctly, the website limited everyone to 1 pixel per 30 seconds, so I guess everyone was just scripting Puppeteer/Chromium to start a new browser, click a pixel, and close the browser, possibly with IP address rotation, but maybe that wasn't even needed. I think you perhaps underestimate just how big of a thing this became basically overnight. I mentioned a drawing over my house to a few people and literally everyone instantly knew what I meant without even saying the website. People love /r/place style things every few years, and this having such a big canvas and being on a world map means that there is a lot of space for everyone to draw literally where they live.
- Aurornis 1y ago
  > I think you perhaps underestimate just how big of a thing this became basically overnight. They don’t need to estimate because in the article they talked to the site and got their traffic numbers: An estimated 2 million users. That’s 1500 requests per user, which implies a lot of scripting is going on.
- zahlman 1y ago
  > I think you perhaps underestimate just how big of a thing this became basically overnight. I mentioned a drawing over my house to a few people and literally everyone instantly knew what I meant without even saying the website. On the other hand, this is the first I've heard of this thing.
  - johnisgood 1y ago
    I have known about this kind of pixel drawing but it was on empty canvas.
- yifanl 1y ago
  They have the user count from the dev, 2 million daily users shouldn't be generating billions of requests unless a good portion of them are botting.
hoppp 1y ago
Cool... You did well to ban them. Its a ddos attack, lucky you dont have to pay for the brandwidth, then its a denial of wallet
proshno 1y ago
[dead]
biker142541 1y ago
Curious how this would have compared to a static pmtiles file being read directly by maplibre. I’ve had good luck with virtually equal latency to served tiles when consuming pmtiles via range requests on Bunnycdn.
- hyperknot 1y ago
  Yes, wplace could solve their whole need by a single, custom-built static pmtile. No need to serve 150 GB of OSM data for their use-case.
- nielsole 1y ago
  https://github.com/hyperknot/openfreemap?tab=readme-ov-file#what-about-pmtiles-and-using-the-cloud https://github.com/hyperknot/openfreemap?tab=readme-ov-file#...
  - biker142541 1y ago
    Interesting, I should benchmark this. I have only used Bunnycdn so far and latency seemed similar to most tile providers like Maptiler and others (but a very limited test). This was using the full planet pmtiles file. Bunnycdn also makes it easy to prevent downloading the entire file, either in case you care about anyone using it or just want to prevent surprise downloads for those exploring network tab.
    - biker142541 1y ago
      Quick benchmark of pmtiles directly in maplibre vs served tiles, both via Bunnycdn and 5 areas sampled using same style. Total impact on page end to end load time: 39ms longer with cached range requests from pmtiles than cached tiles. Individual requests are comparable in the 20-35ms range, so the slight extra time seems to be from the additional round trip for range headers (makes sense).
      - nielsole 1y ago
        What I haven't understood is why not write a program that serves the pmtiles individually on the server? Might not be as performant (but easily hit 1k RPS) but wouldn't require mounting another filesystem which practically rules out a containerised environment.
Ericson2314 1y ago
Oh wow, TIL there is finally a simple way to actually view OpenStreetMap! Gosh, that's overdue. Glad it's done though!
- bravesoul2 1y ago
  ... And then it became 1M rps!
- bspammer 1y ago
  What was wrong with the main site? Genuine question https://www.openstreetmap.org https://www.openstreetmap.org
  - Ericson2314 1y ago
    Oh.... last I checked they didn't have that?
- drewda 1y ago
  The OSM Foundation has been serving raster tiles for years and years (that's what's visible by default on the slippy map at www.openstreetmap.org): https://wiki.openstreetmap.org/wiki/OpenStreetMap_Carto https://wiki.openstreetmap.org/wiki/OpenStreetMap_Carto After on and off experimentation by various contributors, OSMF just released vector tiles as well: https://operations.osmfoundation.org/policies/vector/ https://operations.osmfoundation.org/policies/vector/
  - Ericson2314 1y ago
    Thanks, I'm just out of date
bravesoul2 1y ago
429 is your friend... but well done for handling the load!
wiradikusuma 1y ago
"Wplace.live happened. Out of the blue, a new collaborative drawing website appeared, built from scratch using OpenFreeMap." -- as a founder, you know you're working on the wrong thing when there's a "fun project" getting daily traffic more than what you'd get in a lifetime :)
parhamn 1y ago
Since cloudflare is already sponsoring it, I do wonder how much of this type of service can be implemented all on cloudflare. Their stack could be great for tile serving.
- leobuskin 1y ago
  I’m also surprised to see nginx and hetzner in this project. Why not entirely Cloudflare: workers, R2, and cache
  - conradfr 1y ago
    You can get cheap dedicated server on Hetzner with unlimited bandwidth, would the cost be similar with CF?
- PUSH_AX 1y ago
  You can run containers now on CF, I think this was one of the last barriers that might have prevented a lot of software being migrated without a re-architecting. Most stuff could run there now.
ivanjermakov 1y ago
> Nice idea, interesting project, next time please contact me before. I understand that my popular service might bring your less popular one to the halt, but please configure it on your end so I know _programmatically_ what its capabilities are. I host no API without rate-limiting. Additionally, clearly listing usage limits might be a good idea.
- Aeolun 1y ago
  I think it’s reasonable to assume that a free service is not going to deal gracefully with your 100k rps hug of death. The fact that it actually did is an exception, not the rule. If you are hitting anything free with more than 10rps (temporarily) you are an taking advantage in my opinion.
- Aurornis 1y ago
  > I understand that my popular service might bring your less popular one to the halt, but please configure it on your end so I know _programmatically_ what its capabilities are. Quite entitled expectations for someone using a free and open service to underpin their project. The requests were coming from distributed clients, not a central API gateway that could respond to rate limiting requests > I host no API without rate-limiting. Additionally, clearly listing usage limits might be a good idea. Again, this wasn’t a central, well-behaved client hitting the API from a couple of IPs or with a known API key. They calculate that per every 1 user of the wlive.place website, they were getting 1500 requests. This implies a lot of botting and scripting. This is basically load testing the web site at DDoS scale.
  - ivanjermakov 1y ago
    Ugh, then I agree. This way it's indistinguishable from DDoS attack.
  - zamadatix 1y ago
    > The requests were coming from distributed clients, not a central API gateway that could respond to rate limiting requests The block was done based on URL origin rather than client/token, why wouldn't a rate limiter solution consider the same? For this case (a site which uses the API) it would work perfectly fine. Especially since the bots don't even care about the information from this API so non-site based bots aren't even going to bother to pull the OpenFreeMap tiles.
CommanderData 1y ago
I love sites like wplace can still go viral and blow up, in a age of an increasingly centralised web. Woot
arend321 1y ago
I'm using OpenFreeMap commercially, fantastic and stable service.
- deleted 1y ago
  [deleted]
- cube00 1y ago
  Hopefully you've been able to contribute back financially if you're making money from this service.
  - arend321 1y ago
    For sure, monthly sponsor.
cube00 1y ago
It's really surprising that no CDNs or cloud storage providers offer even the single PMTiles file in some sort of shared library customers can use. I guess they'd all rather their customers each upload the 120GB file and then charge them all individually. If they're crafty they'll have their storage configured so there's only one actual copy on the underlying infra so every other shadow copy is pure profit.

609 points by hyperknot 1y ago

Thank you for this breakdown and for this level of transparency. We have been thinking of moving from MapTiler to OpenFreeMap for StatusGator's outage maps.

hyperknot 1y ago
Feel free to migrate. If you ever worry about High Availability, self-hosting is always an option. But I'm working hard on making the public instance as reliable as possible.

v5v3 1y ago

The article mentions Cloudflare, so how much of this was cached by them?

do_anh_tu 1y ago
Do you even read the article?
- keketi 1y ago
  Are you new? Nobody actually reads the articles.
  - LorenDB 1y ago
    False. I almost never upvote an article without reading it, and half of those upvotes are because I already read something similar recently that gave me the same information.
    - eszed 1y ago
      I'll submit in the second case (already read something similar) that, properly speaking, we should read both, and upvote (or submit, if not already here) the better of the articles. Not that, you know, I often take the time to do that, either - but it would improve the site and the discussions if we all did.
- jwilk 1y ago
  From the HN Guidelines <https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html>: > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
  - RandomBacon 1y ago
    That guideline is decent I guess. I am disappointed that they edited another guideline for the worse: > Please don't comment about the voting on comments. It never does any good, and it makes boring reading. It used to just say, don't complain about voting. If the number of votes are so taboo, why do they even show us the number or user karma (and have a top list)?
    - RandomBacon 1y ago
      We can't even talk about the guidelines?
alessandroberna 1y ago
99.38%

fnord77 1y ago

sounds like they survived 1,000 reqs/sec and the cloudflare CDN survived 99,000 reqs/sec

LoganDark 1y ago

> I believe what is happening is that those images are being drawn by some script-kiddies. Oh absolutely not. I've seen so many autistic people literally just nolifing and also collaborating on huge arts on wplace. It is absolutely not just script kiddies. > 3 billion requests / 2 million users is an average of 1,500 req/user. A normal user might make 10-20 requests when loading a map, so these are extremely high, scripted use cases. I don't know about that either. Users don't just load a map, they look all around the place to search for and see a bunch of the art others have made. I don't know how many requests is typical for "exploring a map for hours on end" but I imagine a lot of people are doing just that. I wouldn't completely discount automation but these usage patterns seem by far not impossible. Especially since wplace didn't expect sudden popularity so they may not have optimized their traffic patterns as much as they could have.

nemomarx 1y ago
There are some user scripts to overlay templates on the map and coordinate working together, but I can't imagine that increases the load much. What might is that wplace has been struggling under the load and you have to refresh to see your pixels placed or any changes and that could be causing more calls an hour maybe?
Karliss 1y ago
Just scrolled around a little bit 2-3minutes with network monitor open. That already resulted in 500requests, 5MB transferred (after filtering by vector tile data). Not sure how many of those got cached by browser with no actual requests, cached by browser exchanging only headers or cached by cloudflare. I am guessing that the typical 10-20 requests/user case is for embedded map fragment like those commonly found in contact page where most users don't scroll at all or at most slightly zoom out to better see rest of city.

charcircuit 1y ago

>Nice idea, interesting project, next time please contact me before. It's impossible to predict that one's project may go viral. >As a single user, you broke the service for everyone. Or you did by not having a high enough fd limit. Blaming sites when using it too much when you advertise there is no limit is not cool. It's not like wplace themselves were maliciously hammering the API.

columb 1y ago
You are so entitled... Because of you most nice things have "no limits but...". Not cool stress testing someone's infrastructure. Not cool. The author of this post is more than understanding, tried to fix it and offered a solution even after blocking them. On a free service. Show us what you have done.
- charcircuit 1y ago
  >You are so entitled That's how agreements work. If someone says they will sell a hamburger for $5, and another person pays $5 for a hamburger, then they are entitled to a hamburger. >On a free service. It's up to the owner to price the service. Being overwhelmed by traffic when there are no limits is not a problem limited only to free services.
  - perching_aix 1y ago
    > Do you offer support and SLA guarantees? > > At the moment, I don’t offer SLA guarantees or personalized support. From the website.
  - eszed 1y ago
    Sure, and if you bulk-order 5k hamburgers the restaurant will honor the price, but they'll also tell you "we're going to need some notice to handle that much product". Perfect analogy, really. This guy handled the situation perfectly, imo.
    - charcircuit 1y ago

feverzsj 1y ago

So, OFM was hit by another Million Dollar Homepage for kids.

eggbrain 1y ago

Limiting by referrer seems strange — if you know a normal user makes 10-20 requests (let’s assume per minute), can’t you just rate limit requests to 100 requests per minute per IP (5x the average load) and still block the majority of these cases? Or, if it’s just a few bad actors, block based on JA4/JA3 fingerprint?

hyperknot 1y ago
What if one user really wants to browse around the world and explore the map. I remember spending half an hour in Google Earth desktop, just exploring around interesting places. I think referer based limits are better, this way I can ask high users to please choose self-hosting instead of the public instance.
toast0 1y ago
Limiting by referrer is probably the right first step. (And changing the front page text) You want to track usage by the site, not the person, because you can ask a site to change usage patterns in a way you can't really ask a site's users. Maybe a per IP limit makes sense too, but you wouldn't want them low enough that it would be effective for something like this.

jspiner 1y ago

The cache hit rate is amazing. Is there something you implemented specifically for this?

hyperknot 1y ago
Yes, I designed the whole path structure / location blocks with caching in mind. Here is the generated nginx.conf, if you are interested: https://github.com/hyperknot/openfreemap/blob/main/docs/assets/nginx.conf https://github.com/hyperknot/openfreemap/blob/main/docs/asse...

rtaylorgarlock 1y ago

Is it always/only 'laziness' (derogatory, i know) when caching isn't implemented by a site like wplace.live ? Why wouldn't they save openfreemap all the traffic when a caching server on their side presumably could serve tiles almost as fast or faster than openfreemap?

deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
VladVladikoff 1y ago
I actually have a direct answer for this: priorities. I run a fairly popular auction website and we have map tiles via stadia maps. We spend about $80/month on this service for our volume. We definitely could get this cost down to a lower tier by caching the tiles and serving them from our proxy. However we simply haven’t yet had the time to work on this, as there is always some other task which is higher priority.
- latchkey 1y ago
  Like reading and commenting on HN articles! ;-)
markerz 1y ago
It looks like a fun website, not a for-profit website. The expectations and focus of fun websites is more to just get it working than to handle the scale. It sounds like their user base exploded overnight, doubling every 14 hours or so. It also sounds like it’s other a solo dev or a small group based on the maintainers wording.
- thunderfork 1y ago
  [dead]
hyperknot 1y ago

willsmith72 1y ago

so 96% availability = "survived" now? but interesting write-up. If I were a consumer of OpenFreeMap, I would be concerned that such an availability drop was only detected by user reports

ndriscoll 1y ago
If I were a consumer of a free service from someone who will not take your money to offer support or an SLA (i.e. is not trying to run a business), I would assume there's little to no monitoring at all.
timmg 1y ago
96% during a unique event. I think you would typically consider long term in a stat like that. Assuming it was close to 100% the rest of the year, that works out to 99.97% over 12 months.

sour-taste 1y ago

Since the limit you ran into was number of open files could you just raise that limit? I get blocking the spammy traffic but theoretically could you have handled more if that limit was upped?

hyperknot 1y ago
I've just written my question to the nginx community forum, after a lengthy debugging session with multiple LLMs. Right now, I believe it was the combination of multi_accept + open_file_cache > worker_rlimit_nofile. https://community.nginx.org/t/too-many-open-files-at-1000-req-sec/5796?u=hyperknot https://community.nginx.org/t/too-many-open-files-at-1000-re... Also, the servers were doing 200 Mbps, so I couldn't have kept up _much_ longer, no matter the limits.
- ndriscoll 1y ago
  One thing that might work for you is to actually make the empty tile file, and hard link it everywhere it needs to be. Then you don't need to special case it at runtime, but instead at generation time. NVMe disks are incredibly fast and 1k rps is not a lot (IIRC my n100 seems to be capable of ~40k if not for the 1 Gbit NIC bottlenecking). I'd try benchmarking without the tuning options you've got. Like do you actually get 40k concurrent connections from cloudflare? If you have connections to your upstream kept alive (so no constant slow starts), ideally you have numCores workers and they each do one thing at a time, and that's enough to max out your NIC. You only add concurrency if latency prevents you from maxing bandwidth.
  - hyperknot 1y ago
    Yes, that's a good idea. But we are talking about 90+% of the titles being empty (I might be wrong on that), that's a lot of hard links. I think the nginx config just need to be fixed, I hope I'll receive some help on their forum.
    - ndriscoll 1y ago
      You could also try turning off the file descriptor cache. Keep in mind that nvme ssds can do ~30-50k random reads/second with no concurrency, or at least hundreds of thousands with concurrency, so even if every request hit disk 10 times it should be fine. There's also kernel caching which I think includes some of what you'd get from nginx's metadata cache?

perching_aix 1y ago

Haven't worked with Cloudflare yet first hand, and I'm not familiar with web map tech. But if the site really is pretty much just serving lots of static files, why is Hetzner in the loop? Wouldn't fully migrating to Cloudflare Pages be possible?

internetter 1y ago
The tiles need to be rendered. Yes frequent tiles can be cached but you already have a cache… it’s Cloudflare. Theoretically you could port the tileserver to Cloudflare pages but then you’d need to… port it… and it probably wouldn’t be cheaper
- perching_aix 1y ago
  Oh interesting, okay. For some reason I had the impression that the tiles were static and rendered offline.
- hyperknot 1y ago
  They are actually static files. There is just too many of them, about 300 million. You cannot put that in Pages.
  - jonathanlydall 1y ago
    Is CloudFlare’s R2 an option for you?
    - piperswe 1y ago
      That, or enabling Cache Reserve to automatically have a global cache in R2 instead of only caching on the PoP-level (disclaimer/source: I am a CF employee)
- hoppp 1y ago
  It would cost a lot. Hetzner is hardware and them you can hammer it, free bandwidth. You get a very good server for cheap. cloudflare would be pay per request, a hefty sum if ddos happens

andai 1y ago

From the screenshot I wanted to say, couldn't this be done on a single VPS? Seemed over engineered to me. Then I realized the silly pixels are on top of a map of the entire earth. Dang! I'm curious what the peak req/s is like. I think it might be just barely within the range supported by benchmark-friendly web servers. Unless there's some kind of order of magnitude slowdowns due to the nature of the application. Edit: Looks like about 64 pixels per km (4096 per km^2). At full color uncompressed that's about 8TB to cover the entire earth (thinking long-term!). 10TB box is €20/month from Hetzner. You'd definitely want some caching though ;) Edit 2: wplace uses 1000x1000 px pngs for the drawing layer. The drawings load instantly, while the map itself is currently very laggy, and some chunks permanently missing.

TylerE 1y ago
"€20/month from Hetzner" is great until you actually need it to be up and working when you need it.
- immibis 1y ago
  IME Hetzner's not unreliable. I don't think you could serve 100k requests per second on a single VPS though. (And with dedicated, you're on the hook for redundancy yourself, same as any dedicated.)
  - cyberpunk 1y ago
    They’re unreliable as soon as you have to deal with their support who have the technical knowledge of a brick. And as soon as you have to do ant business / deal with the german side of the business expect everything to slow down to 2 weeks for response which will still be incorrect. They are simply not worth the hassle. Go with a competent host.
    - Aeolun 1y ago
      > They’re unreliable as soon as you have to deal with their support who have the technical knowledge of a brick. Since I never have to, that’s perfect isn’t it? If you need support from Hetzner you are using the wrong host.

Starlevel004 1y ago

> I believe what is happening is that those images are being drawn by some script-kiddies. If I understand correctly, the website limited everyone to 1 pixel per 30 seconds, so I guess everyone was just scripting Puppeteer/Chromium to start a new browser, click a pixel, and close the browser, possibly with IP address rotation, but maybe that wasn't even needed. I think you perhaps underestimate just how big of a thing this became basically overnight. I mentioned a drawing over my house to a few people and literally everyone instantly knew what I meant without even saying the website. People love /r/place style things every few years, and this having such a big canvas and being on a world map means that there is a lot of space for everyone to draw literally where they live.

Aurornis 1y ago
> I think you perhaps underestimate just how big of a thing this became basically overnight. They don’t need to estimate because in the article they talked to the site and got their traffic numbers: An estimated 2 million users. That’s 1500 requests per user, which implies a lot of scripting is going on.
zahlman 1y ago
> I think you perhaps underestimate just how big of a thing this became basically overnight. I mentioned a drawing over my house to a few people and literally everyone instantly knew what I meant without even saying the website. On the other hand, this is the first I've heard of this thing.
- johnisgood 1y ago
  I have known about this kind of pixel drawing but it was on empty canvas.
yifanl 1y ago
They have the user count from the dev, 2 million daily users shouldn't be generating billions of requests unless a good portion of them are botting.

hoppp 1y ago

Cool... You did well to ban them. Its a ddos attack, lucky you dont have to pay for the brandwidth, then its a denial of wallet

proshno 1y ago

[dead]

biker142541 1y ago

Curious how this would have compared to a static pmtiles file being read directly by maplibre. I’ve had good luck with virtually equal latency to served tiles when consuming pmtiles via range requests on Bunnycdn.

hyperknot 1y ago
Yes, wplace could solve their whole need by a single, custom-built static pmtile. No need to serve 150 GB of OSM data for their use-case.
nielsole 1y ago
https://github.com/hyperknot/openfreemap?tab=readme-ov-file#what-about-pmtiles-and-using-the-cloud https://github.com/hyperknot/openfreemap?tab=readme-ov-file#...
- biker142541 1y ago
  Interesting, I should benchmark this. I have only used Bunnycdn so far and latency seemed similar to most tile providers like Maptiler and others (but a very limited test). This was using the full planet pmtiles file. Bunnycdn also makes it easy to prevent downloading the entire file, either in case you care about anyone using it or just want to prevent surprise downloads for those exploring network tab.
  - biker142541 1y ago
    Quick benchmark of pmtiles directly in maplibre vs served tiles, both via Bunnycdn and 5 areas sampled using same style. Total impact on page end to end load time: 39ms longer with cached range requests from pmtiles than cached tiles. Individual requests are comparable in the 20-35ms range, so the slight extra time seems to be from the additional round trip for range headers (makes sense).
    - nielsole 1y ago
      What I haven't understood is why not write a program that serves the pmtiles individually on the server? Might not be as performant (but easily hit 1k RPS) but wouldn't require mounting another filesystem which practically rules out a containerised environment.

Ericson2314 1y ago

Oh wow, TIL there is finally a simple way to actually view OpenStreetMap! Gosh, that's overdue. Glad it's done though!

bravesoul2 1y ago
... And then it became 1M rps!
bspammer 1y ago
What was wrong with the main site? Genuine question https://www.openstreetmap.org https://www.openstreetmap.org
- Ericson2314 1y ago
  Oh.... last I checked they didn't have that?
drewda 1y ago
The OSM Foundation has been serving raster tiles for years and years (that's what's visible by default on the slippy map at www.openstreetmap.org): https://wiki.openstreetmap.org/wiki/OpenStreetMap_Carto https://wiki.openstreetmap.org/wiki/OpenStreetMap_Carto After on and off experimentation by various contributors, OSMF just released vector tiles as well: https://operations.osmfoundation.org/policies/vector/ https://operations.osmfoundation.org/policies/vector/
- Ericson2314 1y ago
  Thanks, I'm just out of date

bravesoul2 1y ago

429 is your friend... but well done for handling the load!

wiradikusuma 1y ago

"Wplace.live happened. Out of the blue, a new collaborative drawing website appeared, built from scratch using OpenFreeMap." -- as a founder, you know you're working on the wrong thing when there's a "fun project" getting daily traffic more than what you'd get in a lifetime :)

parhamn 1y ago

Since cloudflare is already sponsoring it, I do wonder how much of this type of service can be implemented all on cloudflare. Their stack could be great for tile serving.

leobuskin 1y ago
I’m also surprised to see nginx and hetzner in this project. Why not entirely Cloudflare: workers, R2, and cache
- conradfr 1y ago
  You can get cheap dedicated server on Hetzner with unlimited bandwidth, would the cost be similar with CF?
PUSH_AX 1y ago
You can run containers now on CF, I think this was one of the last barriers that might have prevented a lot of software being migrated without a re-architecting. Most stuff could run there now.

ivanjermakov 1y ago

> Nice idea, interesting project, next time please contact me before. I understand that my popular service might bring your less popular one to the halt, but please configure it on your end so I know _programmatically_ what its capabilities are. I host no API without rate-limiting. Additionally, clearly listing usage limits might be a good idea.

Aeolun 1y ago
I think it’s reasonable to assume that a free service is not going to deal gracefully with your 100k rps hug of death. The fact that it actually did is an exception, not the rule. If you are hitting anything free with more than 10rps (temporarily) you are an taking advantage in my opinion.
Aurornis 1y ago
> I understand that my popular service might bring your less popular one to the halt, but please configure it on your end so I know _programmatically_ what its capabilities are. Quite entitled expectations for someone using a free and open service to underpin their project. The requests were coming from distributed clients, not a central API gateway that could respond to rate limiting requests > I host no API without rate-limiting. Additionally, clearly listing usage limits might be a good idea. Again, this wasn’t a central, well-behaved client hitting the API from a couple of IPs or with a known API key. They calculate that per every 1 user of the wlive.place website, they were getting 1500 requests. This implies a lot of botting and scripting. This is basically load testing the web site at DDoS scale.
- ivanjermakov 1y ago
  Ugh, then I agree. This way it's indistinguishable from DDoS attack.
- zamadatix 1y ago
  > The requests were coming from distributed clients, not a central API gateway that could respond to rate limiting requests The block was done based on URL origin rather than client/token, why wouldn't a rate limiter solution consider the same? For this case (a site which uses the API) it would work perfectly fine. Especially since the bots don't even care about the information from this API so non-site based bots aren't even going to bother to pull the OpenFreeMap tiles.

CommanderData 1y ago

I love sites like wplace can still go viral and blow up, in a age of an increasingly centralised web. Woot

arend321 1y ago

I'm using OpenFreeMap commercially, fantastic and stable service.

deleted 1y ago
[deleted]
cube00 1y ago
Hopefully you've been able to contribute back financially if you're making money from this service.
- arend321 1y ago
  For sure, monthly sponsor.

cube00 1y ago

It's really surprising that no CDNs or cloud storage providers offer even the single PMTiles file in some sort of shared library customers can use. I guess they'd all rather their customers each upload the 120GB file and then charge them all individually. If they're crafty they'll have their storage configured so there's only one actual copy on the underlying infra so every other shadow copy is pure profit.