5 ms·

Couldn't he just pregenerate all 1.048.576 responses, load them somewhere in RAM (or just a bunch of HTML files on an nginx with caching on) and be done with it

Couldn't he just pregenerate all 1.048.576 responses, load them somewhere in RAM (or just a bunch of HTML files on an nginx with caching on) and be done with it? I mean he writes that a single response, gzipped, averages 10kb so that's only 1GB of RAM in total. Even better: host this on a service like Netlify and not even have the 30 day cache timeout Troy has here (which means 30 days old info in case of new breaches). Just regenerate the entire set on the dev box whenever there's a new breach (should be fast enough, it's a linear search & split) and push it to Netlify, it'll invalidate all edge caches automatically.

Bedon292 9y ago
The breach data isn't updated that often. It was 6 months between v1 and v2, so a long cache is totally fine. And then invalidate the cache if there is an update.
- skrebbel 9y ago
  That.. sort of supports my argument :-) He could cache it indefinitely instead of 30 days if he's going to invalidate the cache anyway.
  - Bedon292 9y ago
    I do appear to have misinterpreted what your point was. Sorry about that. Although I assume the idea was limiting how much is cached at any point on their part. Not sure it really matters though its not that much data.
sciurus 9y ago
It sounds like he did pregenerate all the responses, then stored them in Azure Blob Storage. From the article I can't tell why he even needs to use Azure Function. Azure Blob Storage can be configured for public access over HTTP [0], so he could point Cloudflare directly at it. [1] https://docs.microsoft.com/en-us/azure/architecture/patterns/static-content-hosting https://docs.microsoft.com/en-us/azure/architecture/patterns...
- manigandham 9y ago
  The function takes in the hash prefix and sends the correct file in response. That decision has to be made somewhere.
  - sciurus 9y ago
    I'm saying that, at least as described in the blog post, the Azure function is unnecessary because that decision can be made on the frontend (or in the API client). Here is how it sounds like it works now when someone visits https://haveibeenpwned.com/Passwords https://haveibeenpwned.com/Passwords 1) User enters abc123 in the form 2) The frontend translates that to the hash prefix 61ee8 3) The frontend interpolates that into the URL for the function, https://api.pwnedpasswords.com/range/{hashPrefix} https://api.pwnedpasswords.com/range/{hashPrefix} 4) The frontend requests https://api.pwnedpasswords.com/range/61ee8 https://api.pwnedpasswords.com/range/61ee8 5) The Azure function interpolates the hash prefix into the URL for the response in blob storage, something like http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}.txt http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}... 6) The Azure function requests http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt 7) The Azure function returns that to the frontend Here is what I was proposing 1) User enters abc123 in the form 2) The frontend hashes that to 61ee8 3) The frontend interpolates that into the URL for the response in blob storage, something like http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}.txt http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}... 4) The frontend requests http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt
    - manigandham 9y ago
      Sure that works for that site, but it's also a public API that can be used by any client so it's easier to just give out a single URL rather than point to files. The HIBP website just uses the same API instead of doing different logic.
      - deleted 9y ago
        [deleted]

Bedon292 9y ago

The breach data isn't updated that often. It was 6 months between v1 and v2, so a long cache is totally fine. And then invalidate the cache if there is an update.

skrebbel 9y ago
That.. sort of supports my argument :-) He could cache it indefinitely instead of 30 days if he's going to invalidate the cache anyway.
- Bedon292 9y ago
  I do appear to have misinterpreted what your point was. Sorry about that. Although I assume the idea was limiting how much is cached at any point on their part. Not sure it really matters though its not that much data.

sciurus 9y ago

It sounds like he did pregenerate all the responses, then stored them in Azure Blob Storage. From the article I can't tell why he even needs to use Azure Function. Azure Blob Storage can be configured for public access over HTTP [0], so he could point Cloudflare directly at it. [1] https://docs.microsoft.com/en-us/azure/architecture/patterns/static-content-hosting https://docs.microsoft.com/en-us/azure/architecture/patterns...

manigandham 9y ago
The function takes in the hash prefix and sends the correct file in response. That decision has to be made somewhere.
- sciurus 9y ago
  I'm saying that, at least as described in the blog post, the Azure function is unnecessary because that decision can be made on the frontend (or in the API client). Here is how it sounds like it works now when someone visits https://haveibeenpwned.com/Passwords https://haveibeenpwned.com/Passwords 1) User enters abc123 in the form 2) The frontend translates that to the hash prefix 61ee8 3) The frontend interpolates that into the URL for the function, https://api.pwnedpasswords.com/range/{hashPrefix} https://api.pwnedpasswords.com/range/{hashPrefix} 4) The frontend requests https://api.pwnedpasswords.com/range/61ee8 https://api.pwnedpasswords.com/range/61ee8 5) The Azure function interpolates the hash prefix into the URL for the response in blob storage, something like http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}.txt http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}... 6) The Azure function requests http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt 7) The Azure function returns that to the frontend Here is what I was proposing 1) User enters abc123 in the form 2) The frontend hashes that to 61ee8 3) The frontend interpolates that into the URL for the response in blob storage, something like http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}.txt http://pwnedpasswords.blob.core.windows.net/foo/{hashPrefix}... 4) The frontend requests http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt http://pwnedpasswords.blob.core.windows.net/foo/61ee8.txt
  - manigandham 9y ago
    Sure that works for that site, but it's also a public API that can be used by any client so it's easier to just give out a single URL rather than point to files. The HIBP website just uses the same API instead of doing different logic.
    - deleted 9y ago
      [deleted]