4 ms·

Plugging https://search.marginalia.nu/ https://search.marginalia.nu/

• marginalia_nu 2y ago
  Yeah I ran that on like $5k worth of consumer hardware initially. The only caveat is that you want a lot of RAM, and since consumer machines are mostly limited to 128 GB, that's a bit of a hard limit on how much you can do. In terms of storage space and compute, it's relatively fine. I'm on a single ~$17k server now, probably utilizing about 20% of it's capacity.
  • timeon 2y ago
    Naive question: would lower-level language help here?
    • marginalia_nu 2y ago
      The problem domain is sprawling and varied, so you ideally want a bit of everything. You definitely need to do a bunch of custom low level programming, as a lot of standard tools can't cope with the data volumes involved. Like I'm not even storing known URLs in a DBMS anymore, the table would be too slow to update. High level language with the ability to make downcalls to a low level language is probably a sane compromise.