3 ms·

> search.marginalia.nu consisted of 1 Swedish engineer and 1 server last I heard and still managed to outclass DDG in a number of query types relevant to me. S

> search.marginalia.nu consisted of 1 Swedish engineer and 1 server last I heard and still managed to outclass DDG in a number of query types relevant to me. Still just 1 dinky lil' consumer hardware server in my living room. I think what is limiting the project the most is the hardware. Like I can definitely squeeze more out of it, but I could probably do 100 times more if I had any sort of operational budget. But at least the development is funded for the moment. We'll see where I am when the NLnet money runs out...

kragen 4y ago
Does this mean you have more engineers working on it now? That was my reading of the grandparent comment's implications. BTW, thank you very much for offering this service, even though I don't use it often.
- marginalia_nu 4y ago
  No I'm still solo developing. Would having a few more people working on it make stuff move faster? Undeniably. But I don't think lack of manpower is an obstacle at the moment in the same lack of hardware. It's reasonable to expect there's an upper limit to how big a project you can manage as a single developer, but I don't think I'm there yet.
  - reitanqild 4y ago
    I really wish there was someone reading this who could donate some hardware. I would love to but I am not in a position to do it, at least not right now.
    - marginalia_nu 4y ago
      I view it as a long march. Right now I've got about two years runway, which isn't a lot but it also isn't nothing. Considering the project had barely started two years ago, and considering the opportunities it's already created, hopefully more options will make themselves available along the way somehow.
      - kragen 4y ago
        hopefully not the kind where you lose 90% of your forces, many to desertion and others to fighting with ethnic warlords, before finally defeating the encirclement campaign
  - cvwright 4y ago
    What hardware would you want, and how much would it cost?
    - marginalia_nu 4y ago
      Interesting question. If I wanted to say grow the index 10-fold and aim for a billion documents indexed (~2.5bn crawled), I'd need: * More crawlers. This is mostly a network limitation. Disk requirement would be about 50 Tb in total of literally any hard drive, split along the crawlers. * A dedicated number crunching machine for index construction and processing. Basically needs to be all cores and all RAM. * A dedicated SQL server. Doesn't have to be super big or fast, but right now it's contending with the index for RAM. * Either a bigger index server, or a few machines for a sharded index. I'd basically need 40 Tb worth of Enterpise SSD and 1 Tb total of RAM. The problem space is very nice in that almost everything is either embarrassingly parallel along some axis or another or O(n)-O(n log n) so adding more hardware pays off well. This is clearly too much hardware to run out of my apartment, even if the fuses held up the noise and heat would be intolerable, so likely some form of colocation hosting as well. I dunno what it would land on. Probably a one-off cost of about $50k-$100k, depending the degree of redundancy. There's definitely stuff that could be done for less as well. Like right now the entire operation runs out of a single machine, and it's causing me to have to bring the service down for days at a time when I'm switching to a new index.
      - cvwright 4y ago
        Cool, thanks. I will pass this along to some people who might be able to help.

kragen 4y ago

Does this mean you have more engineers working on it now? That was my reading of the grandparent comment's implications. BTW, thank you very much for offering this service, even though I don't use it often.

marginalia_nu 4y ago
No I'm still solo developing. Would having a few more people working on it make stuff move faster? Undeniably. But I don't think lack of manpower is an obstacle at the moment in the same lack of hardware. It's reasonable to expect there's an upper limit to how big a project you can manage as a single developer, but I don't think I'm there yet.
- reitanqild 4y ago
  I really wish there was someone reading this who could donate some hardware. I would love to but I am not in a position to do it, at least not right now.
  - marginalia_nu 4y ago
    I view it as a long march. Right now I've got about two years runway, which isn't a lot but it also isn't nothing. Considering the project had barely started two years ago, and considering the opportunities it's already created, hopefully more options will make themselves available along the way somehow.
    - kragen 4y ago
      hopefully not the kind where you lose 90% of your forces, many to desertion and others to fighting with ethnic warlords, before finally defeating the encirclement campaign
- cvwright 4y ago
  What hardware would you want, and how much would it cost?
  - marginalia_nu 4y ago
    Interesting question. If I wanted to say grow the index 10-fold and aim for a billion documents indexed (~2.5bn crawled), I'd need: * More crawlers. This is mostly a network limitation. Disk requirement would be about 50 Tb in total of literally any hard drive, split along the crawlers. * A dedicated number crunching machine for index construction and processing. Basically needs to be all cores and all RAM. * A dedicated SQL server. Doesn't have to be super big or fast, but right now it's contending with the index for RAM. * Either a bigger index server, or a few machines for a sharded index. I'd basically need 40 Tb worth of Enterpise SSD and 1 Tb total of RAM. The problem space is very nice in that almost everything is either embarrassingly parallel along some axis or another or O(n)-O(n log n) so adding more hardware pays off well. This is clearly too much hardware to run out of my apartment, even if the fuses held up the noise and heat would be intolerable, so likely some form of colocation hosting as well. I dunno what it would land on. Probably a one-off cost of about $50k-$100k, depending the degree of redundancy. There's definitely stuff that could be done for less as well. Like right now the entire operation runs out of a single machine, and it's causing me to have to bring the service down for days at a time when I'm switching to a new index.
    - cvwright 4y ago
      Cool, thanks. I will pass this along to some people who might be able to help.