3 ms·

I have been volunteering with the Internet Archive via Open Library for about half a decade now. Sending money is awesome and they're also always looking for vo

by raybb 19d ago

I have been volunteering with the Internet Archive via Open Library for about half a decade now. Sending money is awesome and they're also always looking for volunteers! There are so many projects going on for Open Library and staff has quite a few volunteers but right now especially I think it would be awesome to have a few more experienced folks joining to help (many volunteers are students). Here's just a few things we could use more experienced folks to help on: - Solr performance, it currently gets quite slow under load and there are two read replicas but we think there may be some ways to improve it besides adding more replicas. You can run a full production copy of solr based on the dumps... But the current workflow to do this is kinda broke so I had AI fix it up but we haven't been able to verify if it's a fully working/good solution. - We're doing big redesigns of major parts of the frontend. If your down to help for that Lokesh had many tickets open and guidelines and welcomes PRs. - I'm currently migrating us from web.py to fastapi and could certainly use more experienced help for trickier ones. Anyway, if you're actually willing to help all you gotta do is join our community call tomorrow at 9am PST and share your interests. We'll you on to something you like quickly :) https://docs.openlibrary.org/everyone/community-call.html#community-call https://docs.openlibrary.org/everyone/community-call.html#co...

b40d-48b2-979e 18d ago
I'm curious, why a BDFL framework like FastAPI over Litestar or a different, performant language altogether?
- raybb 18d ago
  Not another language because the performance issues are limited to the code being sync. Anyway, a gradual migration was a must as well. This is the first I hear of Litestar and I think it does seem very values aligned. I fastapi was picked simply because it was already known by the team and had a very thriving ecosystem. That being said, the hard part is getting off the old sync web.py stuff. Once everything is in async land moving to another framework if needed is relatively trivial. Thanks for sharing that though I'll keep an eye on it!
  - zplizzi 18d ago
    I’ve been using litestar for loads of projects recently and have been loving it. The ability to have full request/response typing (and openapi spec autogeneration) with your favorite dataclass framework is quite nice (I use attrs). I didn’t want to be forced to use pydantic like some other frameworks do.
    - what 17d ago
      Glad I’m not the only one that prefers (c)attrs over pydantic.
  - anandology 18d ago
    I was one of the initial developers of Open Library and wrote most of it's backend code. I doubt any of the performance issues could be because of code being sync. Open Library is IO bound and it is most likely spending time waiting for db to respond. Was there any performance analysis done that indicated that the "code being sync" the main bottleneck? Happy to help.
    - raybb 18d ago
      The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting better, while simultaneously optimizing Solr and removing archive.org from the critical path where possible.
      - anandology 17d ago
        I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path. 2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.
T3RMINATED 18d ago
[dead]

by raybb 19d ago

b40d-48b2-979e 18d ago

I'm curious, why a BDFL framework like FastAPI over Litestar or a different, performant language altogether?

raybb 18d ago
Not another language because the performance issues are limited to the code being sync. Anyway, a gradual migration was a must as well. This is the first I hear of Litestar and I think it does seem very values aligned. I fastapi was picked simply because it was already known by the team and had a very thriving ecosystem. That being said, the hard part is getting off the old sync web.py stuff. Once everything is in async land moving to another framework if needed is relatively trivial. Thanks for sharing that though I'll keep an eye on it!
- zplizzi 18d ago
  I’ve been using litestar for loads of projects recently and have been loving it. The ability to have full request/response typing (and openapi spec autogeneration) with your favorite dataclass framework is quite nice (I use attrs). I didn’t want to be forced to use pydantic like some other frameworks do.
  - what 17d ago
    Glad I’m not the only one that prefers (c)attrs over pydantic.
- anandology 18d ago
  I was one of the initial developers of Open Library and wrote most of it's backend code. I doubt any of the performance issues could be because of code being sync. Open Library is IO bound and it is most likely spending time waiting for db to respond. Was there any performance analysis done that indicated that the "code being sync" the main bottleneck? Happy to help.
  - raybb 18d ago
    The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting better, while simultaneously optimizing Solr and removing archive.org from the critical path where possible.
    - anandology 17d ago
      I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path. 2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.

T3RMINATED 18d ago

[dead]