2 ms·

The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to ar

by raybb 18d ago

The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting better, while simultaneously optimizing Solr and removing archive.org from the critical path where possible.

anandology 17d ago
I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path. 2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.
- mekarpeles 16d ago
  Wonderful to see @raybb and @anandology in this thread. Lots of operational challenges come up when running a service for 14M patrons. And Open Library in particular has a handful of challenges. 1. It's database has grown significantly (800+ GB) and Anand is right that IO (even on SSDs) is a challenge. The `thing` (infobase/infogami) triple-store design is well thought out and gets us a lot, and any system has to be tuned as it scales to hundreds of millions of rows. One strategy here is being smarter about cache and also shifting some of the load from psql to solr. Rishabh and others volunteers have been amazing assets as we've moved in this direction. Jim Champ on staff has been helping me tune psql, pgbouncer, and some of our high IO crons to improve raw db performance. 2. Limited hardware resources. We're trying to move some of our services within the Internet Archive's kubernetes cluster and we've done a great job migrating towards a world where everything is dockerized. It used to be a very painful process for our team of 3 to handle server ops, upgrades, and networking for nearly 15 manually orchestrated servers. One of the bare-metal racks running much of Open Library is significanly oversubscribed on vCPUs and so moving services off to free space and eliminate steal is critical for us right now. Our main web server (ol-www0) suffers from up to 20% steal and we're seeing a lot of congestion before requests even get to our web nodes (app servers). We have a plan and it takes time. 3. Open Library is still dependent on Archive.org for many lookups -- like book availability (which Ben Deitch has been helping me and Drini move into solr). When there are network issues and a network requests takes 5+ seconds, every web.py worker on that thread grinds to a halt and Ray's work moving us to FastAPI has made a significant impact 4. Solr. Drini has been heroic at restructuring our setup to use replicated solr in a way that has increased performance and relieved some of the pressure on our main cluster. This was a huge bottleneck for us this time last year and we've taken a lot of steps to ameliorate our situation. See: https://blog.openlibrary.org/2025/09/12/open-library-search-balancing-high-impact-with-high-demand/ https://blog.openlibrary.org/2025/09/12/open-library-search-... 5. Raw spikes in traffic. We are seeing massive amounts of traffic that slams our book pages, increasing the pain of all the above. It saturates our limited resources, puts more strain on our database, ties us web workers... It makes modsecurity even more expensive. Part of the solutions is being more clever about provisioning, part of the solution is using fail2ban to prevent bad traffic from subtracting from the experience of the patrons who depend on us. Part of the solution is caching and optimizing our database to scale with load. There isn't just one solution and the same 3 engineers on staff (and the support of a completely stellar community of dedicated volunteers fellows and leads) are doing our best to balance ops improvements with the necessary "product" and design improvements necessary that ensure we're useful to people to begin with. I hope this gives the world a bit more of a glimpse how we operate and what some of our challenges are. We're an open source project and our goal is to share as many learnings as we can and to build something useful, sustainable, and beneficial for the community at large. Thank you Ray, Anand, Drini, Jim, Lokesh, Lisa, Charles, and so many dozens more for your tremendous work (present and past) and thank you for being in our corner.

by raybb 18d ago

anandology 17d ago

I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path. 2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.

mekarpeles 16d ago
Wonderful to see @raybb and @anandology in this thread. Lots of operational challenges come up when running a service for 14M patrons. And Open Library in particular has a handful of challenges. 1. It's database has grown significantly (800+ GB) and Anand is right that IO (even on SSDs) is a challenge. The `thing` (infobase/infogami) triple-store design is well thought out and gets us a lot, and any system has to be tuned as it scales to hundreds of millions of rows. One strategy here is being smarter about cache and also shifting some of the load from psql to solr. Rishabh and others volunteers have been amazing assets as we've moved in this direction. Jim Champ on staff has been helping me tune psql, pgbouncer, and some of our high IO crons to improve raw db performance. 2. Limited hardware resources. We're trying to move some of our services within the Internet Archive's kubernetes cluster and we've done a great job migrating towards a world where everything is dockerized. It used to be a very painful process for our team of 3 to handle server ops, upgrades, and networking for nearly 15 manually orchestrated servers. One of the bare-metal racks running much of Open Library is significanly oversubscribed on vCPUs and so moving services off to free space and eliminate steal is critical for us right now. Our main web server (ol-www0) suffers from up to 20% steal and we're seeing a lot of congestion before requests even get to our web nodes (app servers). We have a plan and it takes time. 3. Open Library is still dependent on Archive.org for many lookups -- like book availability (which Ben Deitch has been helping me and Drini move into solr). When there are network issues and a network requests takes 5+ seconds, every web.py worker on that thread grinds to a halt and Ray's work moving us to FastAPI has made a significant impact 4. Solr. Drini has been heroic at restructuring our setup to use replicated solr in a way that has increased performance and relieved some of the pressure on our main cluster. This was a huge bottleneck for us this time last year and we've taken a lot of steps to ameliorate our situation. See: https://blog.openlibrary.org/2025/09/12/open-library-search-balancing-high-impact-with-high-demand/ https://blog.openlibrary.org/2025/09/12/open-library-search-... 5. Raw spikes in traffic. We are seeing massive amounts of traffic that slams our book pages, increasing the pain of all the above. It saturates our limited resources, puts more strain on our database, ties us web workers... It makes modsecurity even more expensive. Part of the solutions is being more clever about provisioning, part of the solution is using fail2ban to prevent bad traffic from subtracting from the experience of the patrons who depend on us. Part of the solution is caching and optimizing our database to scale with load. There isn't just one solution and the same 3 engineers on staff (and the support of a completely stellar community of dedicated volunteers fellows and leads) are doing our best to balance ops improvements with the necessary "product" and design improvements necessary that ensure we're useful to people to begin with. I hope this gives the world a bit more of a glimpse how we operate and what some of our challenges are. We're an open source project and our goal is to share as many learnings as we can and to build something useful, sustainable, and beneficial for the community at large. Thank you Ray, Anand, Drini, Jim, Lokesh, Lisa, Charles, and so many dozens more for your tremendous work (present and past) and thank you for being in our corner.