3 ms·
What have you used if i may ask? It seems very simple indeed. What search API is best? Also there is a program called html2text to throw out the html formattin
by emporas 2y ago
What have you used if i may ask? It seems very simple indeed. What search API is best?
Also there is a program called html2text to throw out the html formatting so as to use less tokens. Have you used this or something similar?
- cpursley 2y agoBrave API (Bing is good as well). Here's a little gist (Elixir). It's pretty rudimentary so far and needs refining, but works alright enough (result at bottom): https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e422790 https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e4... The most useful part is probably the prompt and usage of Phi 3 Mini 128K Instruct for web page summarization and Llama 3 for the final summary (of the summaries). I'm parsing out all but minimal content html but might even remove that to keep context length down.
- emporas 2y agoVery nice, thank you!