One aspect that I don't see mentioned is that APIs for search have basically disappeared from the major players, likely due to them being expensive (programs ca

One aspect that I don't see mentioned is that APIs for search have basically disappeared from the major players, likely due to them being expensive (programs can hammer an API faster than a human with a text box) and lacking a revenue model (no human may be involved, making advertising useless). This has caused "search" to devolve into "human using Web browser types natural language in box, human is presented with results (some semantic, most just links to things other people wrote, some advertisments around the side), human reads through results to see if any are useful to them". This is certainly useful, and I rely it all the time, but it's not the pinnacle of what search could be. It's like if `grep` didn't pipe to stdout, but instead popped up an alert box for each line like "Your query '.*' matched the line 'foo'. Try the new McDonalds saver menu today! [Next/Cancel]", it would still be useful, but nowhere near as useful as piping to stdout. Many years ago Google provided an API to their search engines, which applications could build on to be more "smart". This could have paved the way to much better software: for example, imagine a prolog system where all of Google's knowledge could be used by the calculations. That path was mostly abandoned since there's no scalable incentive to make it operate, much like the semantic Web. Rather than opening up databases to empower others, it's much more profitable to keep them walled off behind a few limited, pay-per-use interfaces (e.g. "paying" by showing a human some adverts). Attempts to bypass or abstract over these interfaces are hit with rate limiters, Recaptcha checks, etc.