4 ms·

I have a side project to monitor the homepage of every domain and I want to detect the type of site - ecommerce, blog, forum etc. I've just started on it and ul

by inovica 10y ago

I have a side project to monitor the homepage of every domain and I want to detect the type of site - ecommerce, blog, forum etc. I've just started on it and ultimately I want to be able to automatically extract data, such as product information from ecommerce sites. Would these videos help here or is there anywhere else that someone would recommend. I know I sounds very naive here... and I am, but there might be someone here who can give me a steer. I've started looking at AI/ML... or whatever its now called and getting a bit confused!

westbrookt 10y ago
It is possible for subject recognition, but the data extraction would be much more difficult. For the simplest subject recognition, you would need a corpus with all the kinds of subjects that you want to recognize, and many examples of each. It would be very tough as a first project in machine learning. Edit Note: I'm very new to machine learning as well, but this is what I've gathered so far.
- deleted 10y ago
  [deleted]
objclxt 10y ago
> ultimately I want to be able to automatically extract data, such as product information from ecommerce sites A little OT, but although you could probably train models to figure that out, many (if not most) eCommerce sites should already be tagging their sites with semantic tags for product information, since it assists with indexing for search engines. Here, for example, is the Schema.org for a product (https://schema.org/Product https://schema.org/Product).
- inovica 10y ago
  That's really useful. Depends how many of the engines (Magento etc) use this - but I'll look into it. Thank you
fnbr 10y ago
You could absolutely do it. Here's a few things you could do to get started (feel free to email me if you need more specific help, or have any additional questions- my email's in my profile): 1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site. 2. Label the data as belonging to each type of site. This is going to be tedious and take some time. You can also try hiring people via Mechanical Turk to do this. 3. Use [Weka](http://www.cs.waikato.ac.nz/ml/weka/ http://www.cs.waikato.ac.nz/ml/weka/) or [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki https://github.com/JohnLangford/vowpal_wabbit/wiki) to run some preliminary estimations on the model. Weka and VW are great tools as they come with a lot of the configuration done out of the box, so you won't have to write any code to get started. Check your results and see how happy you are with them. Weka has a lot of visualization capacities, so you can see how the data that you've collected aligns with the different types of sites. Now, you can start iterating, which is the key part of any ML project. Consider which aspects of the model you think you can improve on- adding more data, adding more kinds of sites, using a different machine learner.
- afro88 10y ago
  > 1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site. This will be the hardest step. Feature extraction from html for identifying specific things like products, for any given site, is very hard (in my experience, which I will admit is pretty limited). Would love to be proven wrong though.
  - fnbr 10y ago
    I was thinking you could approach it as a document classification problem, where you extract the text from the HTML and work with that. I had a lot of success finding duplicate bug reports by comparing the text from the bug reports with reference documents in a variety of topics (e.g. security, networking, C++), and getting a sense of how similar the text is to the reference text. That gives you a score of how relevant each subject is to the document. You could do something similar here- download the text from 10 ecommerce sites, run some sort of topic extraction algorithm (like LDA) on it and then compare the text from the sites you're trying to classify with the text from the reference sites.

by inovica 10y ago

westbrookt 10y ago

It is possible for subject recognition, but the data extraction would be much more difficult. For the simplest subject recognition, you would need a corpus with all the kinds of subjects that you want to recognize, and many examples of each. It would be very tough as a first project in machine learning. Edit Note: I'm very new to machine learning as well, but this is what I've gathered so far.

deleted 10y ago
[deleted]

objclxt 10y ago

> ultimately I want to be able to automatically extract data, such as product information from ecommerce sites A little OT, but although you could probably train models to figure that out, many (if not most) eCommerce sites should already be tagging their sites with semantic tags for product information, since it assists with indexing for search engines. Here, for example, is the Schema.org for a product (https://schema.org/Product https://schema.org/Product).

inovica 10y ago
That's really useful. Depends how many of the engines (Magento etc) use this - but I'll look into it. Thank you

fnbr 10y ago

You could absolutely do it. Here's a few things you could do to get started (feel free to email me if you need more specific help, or have any additional questions- my email's in my profile): 1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site. 2. Label the data as belonging to each type of site. This is going to be tedious and take some time. You can also try hiring people via Mechanical Turk to do this. 3. Use [Weka](http://www.cs.waikato.ac.nz/ml/weka/ http://www.cs.waikato.ac.nz/ml/weka/) or [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki https://github.com/JohnLangford/vowpal_wabbit/wiki) to run some preliminary estimations on the model. Weka and VW are great tools as they come with a lot of the configuration done out of the box, so you won't have to write any code to get started. Check your results and see how happy you are with them. Weka has a lot of visualization capacities, so you can see how the data that you've collected aligns with the different types of sites. Now, you can start iterating, which is the key part of any ML project. Consider which aspects of the model you think you can improve on- adding more data, adding more kinds of sites, using a different machine learner.

afro88 10y ago
> 1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site. This will be the hardest step. Feature extraction from html for identifying specific things like products, for any given site, is very hard (in my experience, which I will admit is pretty limited). Would love to be proven wrong though.
- fnbr 10y ago
  I was thinking you could approach it as a document classification problem, where you extract the text from the HTML and work with that. I had a lot of success finding duplicate bug reports by comparing the text from the bug reports with reference documents in a variety of topics (e.g. security, networking, C++), and getting a sense of how similar the text is to the reference text. That gives you a score of how relevant each subject is to the document. You could do something similar here- download the text from 10 ecommerce sites, run some sort of topic extraction algorithm (like LDA) on it and then compare the text from the sites you're trying to classify with the text from the reference sites.