5 ms·

No sure if relevant but I've hated google's search for a unique reason: I find it horribly slow, especially compared to what it used to be. It's so slow, I've d

No sure if relevant but I've hated google's search for a unique reason: I find it horribly slow, especially compared to what it used to be. It's so slow, I've designed + partially implemented an alternative for my own use: https://github.com/Jeffrey-P-McAteer/dindex https://github.com/Jeffrey-P-McAteer/dindex I've only tested with 1000 records, but the query times are all <200ms.

tomsmeding 7y ago
How does your performance scale? If it scales linearly in records, you're going to have trouble if you're only below 200ms on 1k records. If it scales logarithmically, there are still a couple orders of magnitude between a comprehensive index like Google's and your 1k record index, which is still going to give you trouble. In short, 200ms is slow, not fast. EDIT: Just to clarify, I'm not saying it's necessarily easy to be faster than 200ms (I'd need to look closer into fuzzy text searching algorithms to be able to make an educated guess here); I'm just saying it's not fast enough.
- arch-ninja 7y ago
  Thanks for the insight; indexing the entire web is not a goal of dindex. Instead, it's designed to work like DNS where you can host your own server + have it federate queries to other servers if it doesn't have records which match a query. Basically it moves control to the client; if the client wants to query servers across the globe there's nothing stopping them, but by default you only query servers in your config file. Organizations like the new york times, universities, and governments would host servers for the same reason they host their own email infrastructure. It also moves a ton of control to publishers; dindex doesn't crawl the web unless a client explicitly wants to.
  - arch-ninja 7y ago
    But yea while performance is a huge goal of mine, at the moment I don't even cache compiled regular expressions. It's definitely an alpha-stage project.
judge2020 7y ago
Maybe it's your part of the world, but my TTFB to google search pages is less than 150ms and the "generated in x seconds" is usually less than 1 second. That's pretty good for an index searching effectively every public internet page.
- arch-ninja 7y ago
  I measure from when I hit enter to when my screen is full of results, and just _rendering_ google.com takes a full second on my macbook with 8gb ram and an i5 processor. It's so bad I have a shell script which forces my processor into the 3200mhz range when I'm on my browser workspace. When I'm not looking at my browser (+no downloading files, no audio) the same script sends a SIGSTOP to it so it isn't eating CPU cycles while I'm writing code.
estebarb 7y ago
It's easy to achieve good performance on small datasets. But we are speaking of Internet scale!!! And not only that, Google is capable of serving responses to millions of persons!!! The index take a lot of disk, memory, computing power, bandwidth... And then you have to handle attacks, spam, dead websites and not going bankrupt in the process. I don't know you, but my best toy website is only capable of serving 100k reqs/second in localhost if serving pong...

tomsmeding 7y ago

How does your performance scale? If it scales linearly in records, you're going to have trouble if you're only below 200ms on 1k records. If it scales logarithmically, there are still a couple orders of magnitude between a comprehensive index like Google's and your 1k record index, which is still going to give you trouble. In short, 200ms is slow, not fast. EDIT: Just to clarify, I'm not saying it's necessarily easy to be faster than 200ms (I'd need to look closer into fuzzy text searching algorithms to be able to make an educated guess here); I'm just saying it's not fast enough.

arch-ninja 7y ago
Thanks for the insight; indexing the entire web is not a goal of dindex. Instead, it's designed to work like DNS where you can host your own server + have it federate queries to other servers if it doesn't have records which match a query. Basically it moves control to the client; if the client wants to query servers across the globe there's nothing stopping them, but by default you only query servers in your config file. Organizations like the new york times, universities, and governments would host servers for the same reason they host their own email infrastructure. It also moves a ton of control to publishers; dindex doesn't crawl the web unless a client explicitly wants to.
- arch-ninja 7y ago
  But yea while performance is a huge goal of mine, at the moment I don't even cache compiled regular expressions. It's definitely an alpha-stage project.

judge2020 7y ago

Maybe it's your part of the world, but my TTFB to google search pages is less than 150ms and the "generated in x seconds" is usually less than 1 second. That's pretty good for an index searching effectively every public internet page.

arch-ninja 7y ago
I measure from when I hit enter to when my screen is full of results, and just _rendering_ google.com takes a full second on my macbook with 8gb ram and an i5 processor. It's so bad I have a shell script which forces my processor into the 3200mhz range when I'm on my browser workspace. When I'm not looking at my browser (+no downloading files, no audio) the same script sends a SIGSTOP to it so it isn't eating CPU cycles while I'm writing code.

estebarb 7y ago

It's easy to achieve good performance on small datasets. But we are speaking of Internet scale!!! And not only that, Google is capable of serving responses to millions of persons!!! The index take a lot of disk, memory, computing power, bandwidth... And then you have to handle attacks, spam, dead websites and not going bankrupt in the process. I don't know you, but my best toy website is only capable of serving 100k reqs/second in localhost if serving pong...