6 ms·
Scraped 15 websites, got a good pipeline that ingests and cleans the data once the scraping is done. images get positioned according to simple logic (find top,
by ICodeSometimes 4y ago
Scraped 15 websites, got a good pipeline that ingests and cleans the data once the scraping is done.
images get positioned according to simple logic (find top, find bottom, rotate, etc...)
data is cleaned and units are stripped accordingly (5 inches can be 5in 5 in 5inch 5inches etc...)
so far it's really trivial code so alot of measurements are getting dropped, need to go back in and fix it up and re-scrape/populate the data soon.
- ICodeSometimes 4y agoalso using fp-ts to try out functional programming, it's really perfect for this type of thing, pipe reads really nicely here.