As someone who works in this space, ^ this. I would say don't overthink any component, use common crawl (https://commoncrawl.org/ https://commoncrawl.org/) to b

As someone who works in this space, ^ this. I would say don't overthink any component, use common crawl (https://commoncrawl.org/ https://commoncrawl.org/) to build your initial index, use a pagerank implementation that's been thoroughly researched and published, and use off the shelf components from the apache foundation when you can.