3 ms·
How did you approach the data population? The images of the knives are all consistently positioned, which is impressive, and there seems to be a decent amount o
by andrewmunsell 4y ago
How did you approach the data population? The images of the knives are all consistently positioned, which is impressive, and there seems to be a decent amount of structured data for the number of entries you have
- ICodeSometimes 4y agoScraped 15 websites, got a good pipeline that ingests and cleans the data once the scraping is done. images get positioned according to simple logic (find top, find bottom, rotate, etc...) data is cleaned and units are stripped accordingly (5 inches can be 5in 5 in 5inch 5inches etc...) so far it's really trivial code so alot of measurements are getting dropped, need to go back in and fix it up and re-scrape/populate the data soon.
- ICodeSometimes 4y agoalso using fp-ts to try out functional programming, it's really perfect for this type of thing, pipe reads really nicely here.