4 ms·

But many people including the OP think it is. It’s like me running a web crawler on my phone and saying I can replace Google.

Dylan16807 4y ago
If you were able to index the same amount of content you'd have a damn good alternative. And that's the level of this experiment.
- threeseed 4y ago
  It is trivial to index the same amount of content as Google as the web is mostly static. The hard part is in being able to translate a search query into a list of pages. And that requires a level of sophistication that far exceeds a laptop.
  - Dylan16807 4y ago
    It is not trivial at all. That is a huge amount of scraping. Any machine that can do that will be a similar spec to what you need for serving queries. Not as fast as google does it, but a good amount of them.
    - threeseed 4y ago
      It is trivial compared to the task of translating a user query into search results. It's akin to saying the magic behind OpenGPT is the dataset.
      - Dylan16807 4y ago
        Keyword search is not that hard. Pagerank is not that hard. The comparable goal to the article is to be a search engine, not to fight google for best results.
        threeseed 4y ago
        > Keyword search is not that hard. Pagerank is not that hard. George Hotz, is that you ?
        Dylan16807 4y ago
        Keyword search and pagerank had working solutions decades ago. Hotz was trying to make a car controller that had never been done before, by himself, and then he wanted to """improve""" search with no explanation of what that meant that I saw. I think if he was tasked with taking twitter from no search to "has a search" he probably could have managed it. A team of five people definitely could have managed it.
        ilyt 4y ago
        I think you severely underestimate the effort search engines have to put to filter out all the SEO garbage and come up with some sensible results
        Dylan16807 4y ago
        I don't underestimate the effort, I claim that it's not strictly necessary to have more than a couple people working on it for a minimum viable product, and less for a beta.
seritools 4y ago
What? The OP covers much more than "just shuffling tweet sized buffers to network offload cards"

Dylan16807 4y ago

If you were able to index the same amount of content you'd have a damn good alternative. And that's the level of this experiment.

threeseed 4y ago
It is trivial to index the same amount of content as Google as the web is mostly static. The hard part is in being able to translate a search query into a list of pages. And that requires a level of sophistication that far exceeds a laptop.
- Dylan16807 4y ago
  It is not trivial at all. That is a huge amount of scraping. Any machine that can do that will be a similar spec to what you need for serving queries. Not as fast as google does it, but a good amount of them.
  - threeseed 4y ago
    It is trivial compared to the task of translating a user query into search results. It's akin to saying the magic behind OpenGPT is the dataset.
    - Dylan16807 4y ago
      Keyword search is not that hard. Pagerank is not that hard. The comparable goal to the article is to be a search engine, not to fight google for best results.
      - threeseed 4y ago
        > Keyword search is not that hard. Pagerank is not that hard. George Hotz, is that you ?
        Dylan16807 4y ago
        Keyword search and pagerank had working solutions decades ago. Hotz was trying to make a car controller that had never been done before, by himself, and then he wanted to """improve""" search with no explanation of what that meant that I saw. I think if he was tasked with taking twitter from no search to "has a search" he probably could have managed it. A team of five people definitely could have managed it.
      - ilyt 4y ago
        I think you severely underestimate the effort search engines have to put to filter out all the SEO garbage and come up with some sensible results
        Dylan16807 4y ago
        I don't underestimate the effort, I claim that it's not strictly necessary to have more than a couple people working on it for a minimum viable product, and less for a beta.

seritools 4y ago

What? The OP covers much more than "just shuffling tweet sized buffers to network offload cards"