40 ms·
One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrappi
by EPendragon 1y ago
One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.