5 ms·

As a systems enthusiast I enjoy articles like this. It is really easy to get into the mindset of "this must be perfect". In the Blekko search engine back end w

As a systems enthusiast I enjoy articles like this. It is really easy to get into the mindset of "this must be perfect". In the Blekko search engine back end we built an index that was 'eventually consistent' which allowed updates to the index to be propagated to the user facing index more quickly, at the expense that two users doing the exact same query would get slightly different results. If they kept doing those same queries they would eventually get the exact same results. Systems like this bring in a lot of control systems theory because they have the potential to oscillate if there is positive feedback (and in search engines that positive feedback comes from the ranker which is looking at which link you clicked and giving it a higher weight) and it is important that they not go crazy. Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly. Reading this description of how user's timelines are sharded and the same sorts of feedback loops (in this case 'likes' or 'reposts') sounds like a pretty interesting problem space to explore.

culi 2y ago
What became of Blekko?
- an_ko 2y ago
  > It was acquired by IBM in March 2015, and the service was discontinued. — https://en.wikipedia.org/wiki/Blekko https://en.wikipedia.org/wiki/Blekko Perhaps GP has a more interesting answer though.
  - ChuckMcM 2y ago
    That's the correct answer, IBM wanted the crawler mostly to feed Watson. Building a full search engine (crawler, indexer, ranker, API, web application) for the English language was a hell of an accomplishment but by the time Blekko was acquired Google was paying out tens of billions of dollars to people to send them and only them their search queries. For a service that nominally has to live on advertising revenue getting humans to use it was the only way to be net profitable, and you can't spend billions buying traffic and hope to make it back on advertising as the #3 search engine in the English speaking markets. There are other ways to monetize search (look at Kagi for example) than advertising. Blekko missed that window though. (too early, Google needed to get a crappy as it is today to make the value of a spam free search engine desirable)
    - chrisweekly 2y ago
      Not my Q but thanks for the interesting history. Also, (for other readers), I'm a huge fan of Kagi. Highly recommended.
      - NetOpWibby 2y ago
        I really thought Neeva was gonna make it. I'm glad Kagi swooped in when they exited.
    - NetOpWibby 2y ago
      Blekko was gone by the time I learned about it. Recently (past few years) I emailed someone who worked on Blekko to get his opinion on a search engine concept I still have yet to start. His advice was to not bother competing with Google (obviously) LOL! I don’t know if anyone’s embarked on a P2P search engine but that’s essentially my concept. Anyhoo, thanks for the inspiration!
      - ChuckMcM 2y ago
        Peer to peer would be tough, you really need a 10G network connection to some tier 1 provider, and about 2500 machines to distribute the crawling/serving load. (that is if you want to do a full stack search engine). And while you can run that infrastructure for on the order of $100K/month (not counting depreciation) that means you need roughly $5K/day in revenue from that cluster. At $10 RPM ($10 revenue per thousand queries) you're looking at a minimum of 500,000 'real' search queries during 'English time' (roughly 7AM to 11PM GMT). That's 31,250 queries per hour or ~9 queries per second (average). And that just pays to keep the lights on at the colocation center. If you're paying off the development costs (30 - 50 developers over 2 - 3 years) and the cost of an office somewhere. You'll want at least double that revenue or you'll go broke before you break even. Ideally you are the 'go to' place for people looking to buy something as those queries make money. People researching Douglas Fairbanks for a high school essay consume queries but don't generate ad revenue. It isn't for the faint of heart.
        2y ago
gregw134 2y ago
Would you be willing to share more about how you guys did click ranking at Blekko? It's an interesting problem.
snailmailman 2y ago
I guess I hadn’t considered that search engines could be reranking pages on the fly as I click them. I’ve been seeing my DuckDuckGo results shuffle around for a while now thinking it’s an awful bug. Like I click one page, don’t find what I want, and go back thinking “no, I want that other result that was below” and it’s an entirely different page with shuffled results, missing the one that I think might have been good.
- PaulHoule 2y ago
  That's connected with a basic usability complaint about current web interfaces, that ads and recommended content aren't stable. You very well might want to engage with an ad after you are done engaging what you wanted to engage with but you might never see it again. Similarly, you might see two or three videos that you want to click on on the side of a YouTube video you're watching but you can only click on one (though if you are thinking ahead you can open these in another tab.) On top of that immediate frustration, the YouTube style interface here https://marvelpresentssalo.com/wp-content/uploads/2015/09/idiocracy-ow-my-balls.jpg https://marvelpresentssalo.com/wp-content/uploads/2015/09/id... collects terrible data for recommendations because, even though it gives them information that you liked the thumbnail for a video, they can't come to any conclusion about whether or not you liked any of the other videos. TikTok, by focusing on one video at a time, collects much better information.
  - 4ggr0 2y ago
    > though if you are thinking ahead you can open these in another tab or add it to the "Watch Later" playlist :) so you can watch it...later.
- cgriswald 2y ago
  I don't use DDG, but in my (very limited, just now) testing it doesn't seem to shuffle results unless you reload the page in some way. Is it possible you're browser is reloading the page when you go back? If so, setting DDG to open links in new tabs might fix this problem.
  - 2y ago
dwedge 2y ago
Similar to how Google images loads lower quality blurred thumbnails towards the bottom of the window at first so that the user thinks they loaded faster
aqueueaqueue 2y ago
This is less a question of perfection and one of trade off's. Laws of physics put a limit on how efficiently you can keep data in NYC and London in perfect sync, so you choose CAP-style trade-offs. There are also $/SLO trade-offs. Each 9 costs more money. I like your example it is very interesting. If I get to work on (or even hear someone in my team is working on) such interesting problems and I can hear about it, I get happy. Interesting problems are rare because like a house you might talk about brick vs. Timber frame once, but you'll talk about cleaning the house every week!
genewitch 2y ago
PID techniques useful?
gopher_space 2y ago
> Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly. Looking back at my early work with microservices I'm wondering how much time I would have saved by just manually setting a tongue weight.

culi 2y ago

What became of Blekko?

an_ko 2y ago
> It was acquired by IBM in March 2015, and the service was discontinued. — https://en.wikipedia.org/wiki/Blekko https://en.wikipedia.org/wiki/Blekko Perhaps GP has a more interesting answer though.
- ChuckMcM 2y ago
  That's the correct answer, IBM wanted the crawler mostly to feed Watson. Building a full search engine (crawler, indexer, ranker, API, web application) for the English language was a hell of an accomplishment but by the time Blekko was acquired Google was paying out tens of billions of dollars to people to send them and only them their search queries. For a service that nominally has to live on advertising revenue getting humans to use it was the only way to be net profitable, and you can't spend billions buying traffic and hope to make it back on advertising as the #3 search engine in the English speaking markets. There are other ways to monetize search (look at Kagi for example) than advertising. Blekko missed that window though. (too early, Google needed to get a crappy as it is today to make the value of a spam free search engine desirable)
  - chrisweekly 2y ago
    Not my Q but thanks for the interesting history. Also, (for other readers), I'm a huge fan of Kagi. Highly recommended.
    - NetOpWibby 2y ago
      I really thought Neeva was gonna make it. I'm glad Kagi swooped in when they exited.
  - NetOpWibby 2y ago
    Blekko was gone by the time I learned about it. Recently (past few years) I emailed someone who worked on Blekko to get his opinion on a search engine concept I still have yet to start. His advice was to not bother competing with Google (obviously) LOL! I don’t know if anyone’s embarked on a P2P search engine but that’s essentially my concept. Anyhoo, thanks for the inspiration!
    - ChuckMcM 2y ago
      Peer to peer would be tough, you really need a 10G network connection to some tier 1 provider, and about 2500 machines to distribute the crawling/serving load. (that is if you want to do a full stack search engine). And while you can run that infrastructure for on the order of $100K/month (not counting depreciation) that means you need roughly $5K/day in revenue from that cluster. At $10 RPM ($10 revenue per thousand queries) you're looking at a minimum of 500,000 'real' search queries during 'English time' (roughly 7AM to 11PM GMT). That's 31,250 queries per hour or ~9 queries per second (average). And that just pays to keep the lights on at the colocation center. If you're paying off the development costs (30 - 50 developers over 2 - 3 years) and the cost of an office somewhere. You'll want at least double that revenue or you'll go broke before you break even. Ideally you are the 'go to' place for people looking to buy something as those queries make money. People researching Douglas Fairbanks for a high school essay consume queries but don't generate ad revenue. It isn't for the faint of heart.
      - 2y ago

gregw134 2y ago

Would you be willing to share more about how you guys did click ranking at Blekko? It's an interesting problem.

snailmailman 2y ago

I guess I hadn’t considered that search engines could be reranking pages on the fly as I click them. I’ve been seeing my DuckDuckGo results shuffle around for a while now thinking it’s an awful bug. Like I click one page, don’t find what I want, and go back thinking “no, I want that other result that was below” and it’s an entirely different page with shuffled results, missing the one that I think might have been good.

PaulHoule 2y ago
That's connected with a basic usability complaint about current web interfaces, that ads and recommended content aren't stable. You very well might want to engage with an ad after you are done engaging what you wanted to engage with but you might never see it again. Similarly, you might see two or three videos that you want to click on on the side of a YouTube video you're watching but you can only click on one (though if you are thinking ahead you can open these in another tab.) On top of that immediate frustration, the YouTube style interface here https://marvelpresentssalo.com/wp-content/uploads/2015/09/idiocracy-ow-my-balls.jpg https://marvelpresentssalo.com/wp-content/uploads/2015/09/id... collects terrible data for recommendations because, even though it gives them information that you liked the thumbnail for a video, they can't come to any conclusion about whether or not you liked any of the other videos. TikTok, by focusing on one video at a time, collects much better information.
- 4ggr0 2y ago
  > though if you are thinking ahead you can open these in another tab or add it to the "Watch Later" playlist :) so you can watch it...later.
cgriswald 2y ago
I don't use DDG, but in my (very limited, just now) testing it doesn't seem to shuffle results unless you reload the page in some way. Is it possible you're browser is reloading the page when you go back? If so, setting DDG to open links in new tabs might fix this problem.
- 2y ago

dwedge 2y ago

Similar to how Google images loads lower quality blurred thumbnails towards the bottom of the window at first so that the user thinks they loaded faster

aqueueaqueue 2y ago

This is less a question of perfection and one of trade off's. Laws of physics put a limit on how efficiently you can keep data in NYC and London in perfect sync, so you choose CAP-style trade-offs. There are also $/SLO trade-offs. Each 9 costs more money. I like your example it is very interesting. If I get to work on (or even hear someone in my team is working on) such interesting problems and I can hear about it, I get happy. Interesting problems are rare because like a house you might talk about brick vs. Timber frame once, but you'll talk about cleaning the house every week!

genewitch 2y ago

PID techniques useful?

gopher_space 2y ago

> Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly. Looking back at my early work with microservices I'm wondering how much time I would have saved by just manually setting a tongue weight.