4 ms·
You could absolutely do it. Here's a few things you could do to get started (feel free to email me if you need more specific help, or have any additional questi
by fnbr 10y ago
You could absolutely do it. Here's a few things you could do to get started (feel free to email me if you need more specific help, or have any additional questions- my email's in my profile):
1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site.
2. Label the data as belonging to each type of site. This is going to be tedious and take some time. You can also try hiring people via Mechanical Turk to do this.
3. Use [Weka](http://www.cs.waikato.ac.nz/ml/weka/ http://www.cs.waikato.ac.nz/ml/weka/) or [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki https://github.com/JohnLangford/vowpal_wabbit/wiki) to run some preliminary estimations on the model. Weka and VW are great tools as they come with a lot of the configuration done out of the box, so you won't have to write any code to get started.
Check your results and see how happy you are with them. Weka has a lot of visualization capacities, so you can see how the data that you've collected aligns with the different types of sites.
Now, you can start iterating, which is the key part of any ML project. Consider which aspects of the model you think you can improve on- adding more data, adding more kinds of sites, using a different machine learner.
- afro88 10y ago> 1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site. This will be the hardest step. Feature extraction from html for identifying specific things like products, for any given site, is very hard (in my experience, which I will admit is pretty limited). Would love to be proven wrong though.
- fnbr 10y agoI was thinking you could approach it as a document classification problem, where you extract the text from the HTML and work with that. I had a lot of success finding duplicate bug reports by comparing the text from the bug reports with reference documents in a variety of topics (e.g. security, networking, C++), and getting a sense of how similar the text is to the reference text. That gives you a score of how relevant each subject is to the document. You could do something similar here- download the text from 10 ecommerce sites, run some sort of topic extraction algorithm (like LDA) on it and then compare the text from the sites you're trying to classify with the text from the reference sites.
- deleted 10y ago[deleted]
- inovica 10y agoThank you. I'll check these out and also drop you a line