4 ms·
> A process at my day job handles millions of URLs per day. As part of handling each URL, it checked whether the URL was present in the database, and then logge
by neverminder 11y ago
> A process at my day job handles millions of URLs per day. As part of handling each URL, it checked whether the URL was present in the database, and then logged something if it was. This was implemented so that we could get interesting stats on how many of the URLs were already present at report time. It turned out that it was spending 30% of its time doing these database queries, there was still no automated process to generate the stats, and humans had long since stopped checking. Deleting that 1 line of code knocked 30% off the execution time of every URL.
Or you can make the logging part asynchronous (provided the server itself is not maxed out) and cut that 30% of request time.
- sethammons 11y agoAnd toss in a bloom filter to test for membership and reduce the likelihood of even needing to call the db.
- GrinningFool 11y agoBut that also loses your backpressure - increasing chances that sooner or later (depending on platform/technology), the whole server will go down in a bad way.