3 ms·
The easiest way would be running Postgres: https://www.crunchydata.com/blog/postgres-full-text-search-a-search-engine-in-a-database https://www.crunchydata.com
by readyplayernull 2y ago
The easiest way would be running Postgres:
https://www.crunchydata.com/blog/postgres-full-text-search-a-search-engine-in-a-database https://www.crunchydata.com/blog/postgres-full-text-search-a...
The problem is getting all the data. If you try to scrap another search engine it will punish you.
You could think on a distributed collaborative SE that scraps the web from the IP addresses of each device.
- vednig 2y agoYes, based on how web is design there will always be different subsections of the internet which I can scrape but all of the data starting from one site like Wikipedia would only allow me to scrape only one subsection of the internet mainly, truth and news coverage, if I wanted to tap into index of spam ads and untrusted blog posts from this dataset it would never yield correct or even nearly correct result.