3 ms·

Ask HN: Can I use ML to extract individual images from a webpage screenshot?

6 points by kordlessagain 5y ago

tgflynn 5y ago
Probably but it would be a lot of work. Why do you need to work with screenshots instead of live sites ?
gerardnico 5y ago
Yes. It's called edge detection in computer vision.
proteuskor 5y ago
Yes, a simple approach would be to use a convolutional network for multiple object detection. You would need a training set that was transferable. In your case I think you would best use a training set generator and generate a bunch of webpages with known image locations and sizes. Coursera has a video and links to references on efficient multiple object detection, look at the Andrew Ng courses. I suspect this would be an easy application. You would probably want to use PyTorch or Tensorflow to specify the model and use a GPU or GPU instance if on the cloud to accelerate the training sufficiently to explore hyper-parameters intuitively. Selenium may help with the test set generator if you can extract the image coordinates relative to the browser window with that and then generate a screen shot of that application window. You may need to generate 100k or so such randomly generated pages with multiple rectangles to reach very good accuracy, but that number is very hand wavy. E.g. you need to do your best to generate a uniform distribution of web pages, and then render them with a real browser, extracting the image rectangle coordinates somehow to generate the labeled test set. Then feed that into a convolutional network (or R-CNN) with back-prop's error metric being a function of the R-CNN's output objects vs. the training set labels. This should then converge on correct results and you should plot the convergence of the error metric to see when it asymptotes. If the quality is sufficient at asymptote you are done, if not you may have to change structure and/or hyper-parameters.
- kordlessagain 5y ago
  This is great. Thank you for taking the time to comment! I have the screenshot stuff working with Selenium.
Jugurtha 5y ago
What are you trying to do ?
- kordlessagain 5y ago
  Using an imaging crawler to extract text and images from a webpage to avoid parsing the DOM. Reason include dynamic content loading, diverse HTML usage and consistent training data sets for ML models. I have edge detection running in OpenCV, but it's a bit problematic and brittle. My reasoning is that a ML model would be a bit more robust and useful to others.

tgflynn 5y ago

Probably but it would be a lot of work. Why do you need to work with screenshots instead of live sites ?

gerardnico 5y ago

Yes. It's called edge detection in computer vision.

proteuskor 5y ago

Yes, a simple approach would be to use a convolutional network for multiple object detection. You would need a training set that was transferable. In your case I think you would best use a training set generator and generate a bunch of webpages with known image locations and sizes. Coursera has a video and links to references on efficient multiple object detection, look at the Andrew Ng courses. I suspect this would be an easy application. You would probably want to use PyTorch or Tensorflow to specify the model and use a GPU or GPU instance if on the cloud to accelerate the training sufficiently to explore hyper-parameters intuitively. Selenium may help with the test set generator if you can extract the image coordinates relative to the browser window with that and then generate a screen shot of that application window. You may need to generate 100k or so such randomly generated pages with multiple rectangles to reach very good accuracy, but that number is very hand wavy. E.g. you need to do your best to generate a uniform distribution of web pages, and then render them with a real browser, extracting the image rectangle coordinates somehow to generate the labeled test set. Then feed that into a convolutional network (or R-CNN) with back-prop's error metric being a function of the R-CNN's output objects vs. the training set labels. This should then converge on correct results and you should plot the convergence of the error metric to see when it asymptotes. If the quality is sufficient at asymptote you are done, if not you may have to change structure and/or hyper-parameters.

kordlessagain 5y ago
This is great. Thank you for taking the time to comment! I have the screenshot stuff working with Selenium.

Jugurtha 5y ago

What are you trying to do ?

kordlessagain 5y ago
Using an imaging crawler to extract text and images from a webpage to avoid parsing the DOM. Reason include dynamic content loading, diverse HTML usage and consistent training data sets for ML models. I have edge detection running in OpenCV, but it's a bit problematic and brittle. My reasoning is that a ML model would be a bit more robust and useful to others.