6 ms·

This is very creative. Thing to note is that this is not free (or at least it is likely not to be, OpenAI is charging for APIs). We are talking about ~1c per se

This is very creative. Thing to note is that this is not free (or at least it is likely not to be, OpenAI is charging for APIs). We are talking about ~1c per search looking at current GPT pricing (2c per ~750 words in/out). It is also indiscriminate deployed in this way, triggering for every search, even those that chatgpt will show suboptimal responses for, like 'starbucks near me', navigational queries like 'twitter' or anything else chatgpt will be bad in the context of search (a lot of things!). And it is non-trivial to predict which searches GPT will be good for (especially within typical result latency requirements of modern search engines). We are doing some experiments with this at Kagi, and the main trick is to manage cost, possibly through on-demand triggering mechanism (which also can help manage accuracy). One thing to keep in mind is that this is likely going to get better/faster/cheaper in the future.

nilespotter 4y ago
Hey wow, it's the Kagi guy. Offtopic, sorry: I love Kagi. I would say 85% of my searches I stick with it, 15% I tuck tail and head to that one big search engine. Pretty phenomenal IMO, and I have no doubt it will get better. Keep it up!
Waterluvian 4y ago
I forget where I read this recently but a compelling point made was that this is fun as a nerd toy but to deploy it to something general like Google or Gmail would be overwhelmingly expensive. Reminds me of the, “so expensive that only the five richest kings of Europe will own them.” joke from The Simpsons. Eventually it’ll be ridiculously cheap and easy to include anywhere.
- ramraj07 4y ago
  I remember estimates of the worth of each search to Google being 10 cents on average. Google does run Bert on many if not all of its searches anyway? And let’s not forget that searching the entire internet is not cheap either! Google holds the entire index in memory and every search hit goes through thousands of machines to return your results. In other words, running chatGPT might not exactly be a problem for google if it decides to do so.
  - Waterluvian 4y ago
    Google’s search of “the entire internet” is lacking more and more every year. It’s probably not true at all anymore. It’s probably “the sliver of the Internet we prefer you interacted with.” Any time I search these days I’m amazed at how you can exhaust the search in 1-2 pages before you get to “related hits.”
    - Beaver117 4y ago
      Yeah, it's clearly lacking. Either de-indexing many pages or just blocking them from the results, even when I use double quotes. Then I go to Yandex or something and voila, it pops right up. I'm not sure I care enough to pay for Kagi but there's something very wrong with Google (and DDG, and Bing, etc)
      - akiselev 4y ago
        It’s definitely deindexing pages. It’s been years since I’ve been able to locate one of my old HN rants via Google Search when it used to be the first result. Before all I had to use was my username + a politician’s name that I’ve only ever used once. After looking up the comment with Algolia and adding more keywords directly from the post, Google just gives up
        ramraj07 4y ago
        Genuinely curious, can you give both your name and the politician name in double quotes individually? Google has stopped requiring all terms to be present in search, this is a way of forcing it..
        deleted 4y ago
        [deleted]
        akiselev 4y ago
        If I put the two words in quotes, the only result is this unrelated thread: https://news.ycombinator.com/item?id=28162412 https://news.ycombinator.com/item?id=28162412 - I never replied in that thread, so it completely ignored my username despite the quotes but did contain one instance of the other search term. Just double checked Algolia and I actually have three comments that fit the query. Two posted nine years ago and one posted ten years ago. It's been at least three or four years since I was last able to use Google to find my comment. Edit: Turns out if quote the entire first line containing the name, Google finds the comment. It seems they're only purging parts of their index
Terretta 4y ago
For search, abandon Google, not hope. All ye should enter here: https://kagi.com https://kagi.com // as a kagi user, i'd (a) imagine this as a lens at the simplest; just pick text-davinci-003, tokens 3000 to leave room for prompt, temp 0.7, freq 1 - 1.5, presence 0.5, and instead of best of 3 or 5 show repeated calls as if 3 - 5 unique results (b) imagine a richer implementation that summarizes other articles on first SERP, then collates and summarizes those (with the compare/contrast structured synthesis GPT-3 does well when POVs differ), and shows the final rollup summary above the individual summaries, in the right hand column // would also be OK connecting it to my OpenAI token so I'm paying, not you. having done the math, it's nominal cost if not the default mode.
echelon 4y ago
The true killer apps will come when StabilityAI or another group releases an open source equivalent of GPT-3.5. Compare Dall-E to the creative explosion that arose from Stable Diffusion. Nobody is going to build the next Google atop OpenAI APIs except for OpenAI themselves. An open source model and pretrained weights will open the playing field for everyone to compete with Google.
- tzm 4y ago
  Stable diffusion for davinci003 is coming
  - echelon 4y ago
    I can't wait!
- charcircuit 4y ago
  Stable diffusion can run on regular PCs. GPT-3.5 on the hand can't without doing a lot of swap between disk and vram making it sl.
spaceman_2020 4y ago
If it’s possible, please create a family plan or regional pricing. I love Kagi but if I were to buy it for my entire family, the bill would be $600/year - more than a month of rent.
- esperent 4y ago
  In the country where I live (Vietnam) $600 would be 3-4 months wages for many people. Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though.
  - deleted 4y ago
    [deleted]
  - palunon 4y ago
    > Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though. This may be true if your cost per client/customer/etc is either negligible (such as with digital goods delivery), or dependent on their country (eg. retail). Here, the bulk of their cost is computing resources, and they (according to their profile page) don't even make enough to cover it with the current price. I don't think this cost would go down with the customer location. Yeah, it sucks a lot that people in rich countries can afford things people in other countries cannot, but that's kinda what "rich country" means.
    - fnordpiglet 4y ago
      I’m curious about their economics. I would assume there’s a huge upfront fixed cost but the marginal unit cost is tiny. They may not be breaking even or run at even, but that (I expect) is because they’re struggling to overcome that first hump of the fixed cost, but at the margin they’re doing well. I would be flabbergasted if my monthly is going to pay for my search transactions rather than the ability to execute any search at all. In that model you can in fact offer regional pricing as long as there’s a critical mass sufficient to justify the localization and regional edge deployments. You can go even further and let larger (in dollar space) regions subsidize smaller regions while they grow critical mass. Some companies I’ve worked at felt offering their service globally was worth the depressed margins even if they never made it make economic sense simply because it was useful enough - but also it creates a solid global brand. Anyway Kagi is great and I wish them the best of fortunes.
cma 4y ago
Results for the same query can be cached, making the Starbucks near me type queries bypass GPT3 other than the first time across all users.
shepherdjerred 4y ago
It's awesome to hear that Kagi is experimenting with this. Even if it's some extra paid opt-in feature, I'd give it a chance.
fennecfoxy 4y ago
I don't know why people are always so up in arms about cost: if Google switched to a paid rather than ads model and charged 1 credit (1c) per search you made, I'd still use it. People always raise such a ruckus: how dare Youtube offer premium, etc. People get too addicted to free services and become self-entitled.
chrisshroba 4y ago
> it is non-trivial to predict which searches GPT will be good for Since pricing is roughly proportional to response length (and maybe prompt length?), it seems like ChatGPT could use itself to determine if the search is a good fit or not. Give it a prompt like "I want to know if you are likely to generate more immediately useful and actionable results than my web search. The query I am searching is <query>. Should I run you on this query?" I tried running that on some sample queries to see its output, and compiled its responses in this table (note the last one is incorrect in my opinion): | Query | Yes or No | |------------------------------------|-----------| | "starbucks near me" | No | | "twitter" | No | | "python requests get json" | Yes | | "peach and mango cocktail recipe" | Yes | | "hn" | No | | "image of versaille" | Yes | Btw, incredibly, ChatGPT actually made that table for me when I said "Please compile all the queries I just asked you about into an ascii table, where the first column is my query, and the second column is your Yes or No answer." The table it printed was correct but formatted as an HTML table, so I asked it to put it in a code block and I got almost exactly what I pasted above (just had to remove two extraneous spaces that made the lines not line up)
- freediver 4y ago
  Yep, this is one of the amazing things about it. But using ChatGPT to determine whether a query is good for ChatGPT answer does not address the second point I made, one about typical latency requirements (it makes it worse). Normally you want your search results in about 500ms. Using ChatGPT to first figure out if the query is good will take 1-3 seconds then using it again another 1-3 seconds. So we are talking about results in about 4 seconds. Plus it is not 1c per query now it is more likely 1.5c per query as you introduced an extra prompt.

nilespotter 4y ago

Hey wow, it's the Kagi guy. Offtopic, sorry: I love Kagi. I would say 85% of my searches I stick with it, 15% I tuck tail and head to that one big search engine. Pretty phenomenal IMO, and I have no doubt it will get better. Keep it up!

Waterluvian 4y ago

I forget where I read this recently but a compelling point made was that this is fun as a nerd toy but to deploy it to something general like Google or Gmail would be overwhelmingly expensive. Reminds me of the, “so expensive that only the five richest kings of Europe will own them.” joke from The Simpsons. Eventually it’ll be ridiculously cheap and easy to include anywhere.

ramraj07 4y ago
I remember estimates of the worth of each search to Google being 10 cents on average. Google does run Bert on many if not all of its searches anyway? And let’s not forget that searching the entire internet is not cheap either! Google holds the entire index in memory and every search hit goes through thousands of machines to return your results. In other words, running chatGPT might not exactly be a problem for google if it decides to do so.
- Waterluvian 4y ago
  Google’s search of “the entire internet” is lacking more and more every year. It’s probably not true at all anymore. It’s probably “the sliver of the Internet we prefer you interacted with.” Any time I search these days I’m amazed at how you can exhaust the search in 1-2 pages before you get to “related hits.”
  - Beaver117 4y ago
    Yeah, it's clearly lacking. Either de-indexing many pages or just blocking them from the results, even when I use double quotes. Then I go to Yandex or something and voila, it pops right up. I'm not sure I care enough to pay for Kagi but there's something very wrong with Google (and DDG, and Bing, etc)
    - akiselev 4y ago
      It’s definitely deindexing pages. It’s been years since I’ve been able to locate one of my old HN rants via Google Search when it used to be the first result. Before all I had to use was my username + a politician’s name that I’ve only ever used once. After looking up the comment with Algolia and adding more keywords directly from the post, Google just gives up
      - ramraj07 4y ago
        Genuinely curious, can you give both your name and the politician name in double quotes individually? Google has stopped requiring all terms to be present in search, this is a way of forcing it..
        deleted 4y ago
        [deleted]
        akiselev 4y ago
        If I put the two words in quotes, the only result is this unrelated thread: https://news.ycombinator.com/item?id=28162412 https://news.ycombinator.com/item?id=28162412 - I never replied in that thread, so it completely ignored my username despite the quotes but did contain one instance of the other search term. Just double checked Algolia and I actually have three comments that fit the query. Two posted nine years ago and one posted ten years ago. It's been at least three or four years since I was last able to use Google to find my comment. Edit: Turns out if quote the entire first line containing the name, Google finds the comment. It seems they're only purging parts of their index

Terretta 4y ago

For search, abandon Google, not hope. All ye should enter here: https://kagi.com https://kagi.com // as a kagi user, i'd (a) imagine this as a lens at the simplest; just pick text-davinci-003, tokens 3000 to leave room for prompt, temp 0.7, freq 1 - 1.5, presence 0.5, and instead of best of 3 or 5 show repeated calls as if 3 - 5 unique results (b) imagine a richer implementation that summarizes other articles on first SERP, then collates and summarizes those (with the compare/contrast structured synthesis GPT-3 does well when POVs differ), and shows the final rollup summary above the individual summaries, in the right hand column // would also be OK connecting it to my OpenAI token so I'm paying, not you. having done the math, it's nominal cost if not the default mode.

echelon 4y ago

The true killer apps will come when StabilityAI or another group releases an open source equivalent of GPT-3.5. Compare Dall-E to the creative explosion that arose from Stable Diffusion. Nobody is going to build the next Google atop OpenAI APIs except for OpenAI themselves. An open source model and pretrained weights will open the playing field for everyone to compete with Google.

tzm 4y ago
Stable diffusion for davinci003 is coming
- echelon 4y ago
  I can't wait!
charcircuit 4y ago
Stable diffusion can run on regular PCs. GPT-3.5 on the hand can't without doing a lot of swap between disk and vram making it sl.

spaceman_2020 4y ago

If it’s possible, please create a family plan or regional pricing. I love Kagi but if I were to buy it for my entire family, the bill would be $600/year - more than a month of rent.

esperent 4y ago
In the country where I live (Vietnam) $600 would be 3-4 months wages for many people. Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though.
- deleted 4y ago
  [deleted]
- palunon 4y ago
  > Any company that doesn't do regional pricing is only interested in doing business with rich countries. Which sucks but is understandable. I wish they were more honest about it though. This may be true if your cost per client/customer/etc is either negligible (such as with digital goods delivery), or dependent on their country (eg. retail). Here, the bulk of their cost is computing resources, and they (according to their profile page) don't even make enough to cover it with the current price. I don't think this cost would go down with the customer location. Yeah, it sucks a lot that people in rich countries can afford things people in other countries cannot, but that's kinda what "rich country" means.
  - fnordpiglet 4y ago
    I’m curious about their economics. I would assume there’s a huge upfront fixed cost but the marginal unit cost is tiny. They may not be breaking even or run at even, but that (I expect) is because they’re struggling to overcome that first hump of the fixed cost, but at the margin they’re doing well. I would be flabbergasted if my monthly is going to pay for my search transactions rather than the ability to execute any search at all. In that model you can in fact offer regional pricing as long as there’s a critical mass sufficient to justify the localization and regional edge deployments. You can go even further and let larger (in dollar space) regions subsidize smaller regions while they grow critical mass. Some companies I’ve worked at felt offering their service globally was worth the depressed margins even if they never made it make economic sense simply because it was useful enough - but also it creates a solid global brand. Anyway Kagi is great and I wish them the best of fortunes.

cma 4y ago

Results for the same query can be cached, making the Starbucks near me type queries bypass GPT3 other than the first time across all users.

shepherdjerred 4y ago

It's awesome to hear that Kagi is experimenting with this. Even if it's some extra paid opt-in feature, I'd give it a chance.

fennecfoxy 4y ago

I don't know why people are always so up in arms about cost: if Google switched to a paid rather than ads model and charged 1 credit (1c) per search you made, I'd still use it. People always raise such a ruckus: how dare Youtube offer premium, etc. People get too addicted to free services and become self-entitled.

chrisshroba 4y ago

> it is non-trivial to predict which searches GPT will be good for Since pricing is roughly proportional to response length (and maybe prompt length?), it seems like ChatGPT could use itself to determine if the search is a good fit or not. Give it a prompt like "I want to know if you are likely to generate more immediately useful and actionable results than my web search. The query I am searching is <query>. Should I run you on this query?" I tried running that on some sample queries to see its output, and compiled its responses in this table (note the last one is incorrect in my opinion): | Query | Yes or No | |------------------------------------|-----------| | "starbucks near me" | No | | "twitter" | No | | "python requests get json" | Yes | | "peach and mango cocktail recipe" | Yes | | "hn" | No | | "image of versaille" | Yes | Btw, incredibly, ChatGPT actually made that table for me when I said "Please compile all the queries I just asked you about into an ascii table, where the first column is my query, and the second column is your Yes or No answer." The table it printed was correct but formatted as an HTML table, so I asked it to put it in a code block and I got almost exactly what I pasted above (just had to remove two extraneous spaces that made the lines not line up)

freediver 4y ago
Yep, this is one of the amazing things about it. But using ChatGPT to determine whether a query is good for ChatGPT answer does not address the second point I made, one about typical latency requirements (it makes it worse). Normally you want your search results in about 500ms. Using ChatGPT to first figure out if the query is good will take 1-3 seconds then using it again another 1-3 seconds. So we are talking about results in about 4 seconds. Plus it is not 1c per query now it is more likely 1.5c per query as you introduced an extra prompt.