5 ms·

I cooked up a prototype of this project about a year and a half ago. While the results were encouraging, I lost motivation and it just collected dust for all t

I cooked up a prototype of this project about a year and a half ago. While the results were encouraging, I lost motivation and it just collected dust for all that time. In the past 3 weeks I've revived the project and given new life to it. The project started when I wondered: "What if delicious tag signatures were vectors, would their nearest neighbors in n-dimensional space be a good match for similarity?" The answer took awhile to figure out, but it was a yes. For me at least, it wasn't an easy problem to solve; finding the nearest neighbors with each vector having 10 dimensions, with total dimensional space about 50,000. Figuring out a way to solve this in a timely manner was a joy. At the time this project was first started and the first prototype completed, there were absolutely zero sites, other than google's sub-par "similar" link, that offered this. Well, there was "similicious" but its index was tiny, and results weren't too impressive. Now I count 5 full fledged services that do a pretty good job. Some are ranked quit highly and probably make a decent buck. I really think I missed a decent opportunity by not pushing through the last 20% of the project, and losing motivation entirely. But, like they say, better late than never. Questions/comments are welcomed/anxiously awaited.

prawn 16y ago
Just a random, half-baked airport musing, but what if you had a related service that filtered out links which were similar and likely to be competitors. Someone might want to restrict what sites their crowdsourced content might link to?
- photon_off 16y ago
  I've considered the uses of moreofit for competitor-like stuff like this. I could make a service which (when installed on your site) scans users' histories for the top 500 competitor websites, aggregate those results, and let you know which competitors websites your visitors have most often been to. Also, if they've been to X competitor, theres a Y% chance they've been to Z competitor. I haphazardly concluded that the value seems to be in the service itself, not the "finding competitors" aspect. You likely know who your competitors are, or can find out with a 1 time search on a similarity site such as mine. That doesn't mean the idea has no legs. I imagine it's something people would pay for, and probably already are. It's just not my cup of tea at the moment to be censoring or sniffing people's histories.
  - 10ren 16y ago
    The recent posts on how easy it is to launch a start up these days screamed "glut" to me. One solution is to target a niche that wouldn't be well enough served by a universal solution. Anyway, what I'm getting at is that monitoring competitor popularity would be a valuable service (including an early warning system for detecting emerging competitors.) I think many of us do this every so often manually; and/or have google alerts set up for it. But a service that just does this would broaden the appeal - especially if you could somehow show that your estimates are more accurate... I have no idea of how you could do this... I guess you'd need a genuinely accurate measure, and then compare yours and manual searches with it. But if you could demonstrate that credibly and compellingly, it'd be gold. I don't think you need to censor or sniff to do this.
    - photon_off 16y ago
      If I understand you correctly, the service would work as follows: You enter a URL you want to monitor. My service would show you trends of similar websites to watch out for. It would base it on some sort of secret sauce that measures similarity and popularity of competitors. Correct? That's an interesting idea. What types of current websites/businesses do you think could benefit from this right now? Could you give me examples? This could help me see if it's possible to monitor similar websites and develop trends. Measuring a website's popularity is pretty straightforward -- you make some sort of formula based on how often that URL is discussed (from backtype), linked to (various sources), bookmarked, and ranked (alexa, quantcast, et al). It's a matter of mashing up that data with the similarity data (over time) to determine the trends of similar sites.
      - 10ren 16y ago
        That's right - though you enter your own URL. Extra idea: it might also show you complements (products/services that fit in nicely with yours; these are usually pretty obvious in established industries - razors+shaving cream; milk+eggs; tyres +brakepads - but not so obvious in emerging industries. You might not have even thought of the category of a certain complement, let alone the name of it, nor specific brands that do it. It's valuable, because you might consider targeting people who buy the complements. It's a kind of market-research. (Unhelpfully) I think every website could benefit from it. Probably emergent and fast-moving markets would benefit most (because they don't yet know their competitors/complements + it's all changing very fast anyway). A very specific start would be... y-combinator offerings (that have just launched - not the established onces); for fun, you could also do it for emerging languages (like Mirah). These aren't ideal biz markets, but they would gain attention here on HN, and might validate the basic idea (to see if it's possible.) For example, if you were able to find a competitor to a YC company that they didn't know about, it would be pretty compelling evidence (and will surely happen, since startups rarely look very closely for competitors; they are too busy doing. Which is a good thing. Knowing about competitors too early can be artificially discouraging. So this is probably most helpful for companies that already have enough traction that they wouldn't be discouraged. Competitors are good for giving you ideas; for demonstrating market existence (good for raising funds...); for suggesting markets you hadn't thought of. Complements are always good. Perhaps also, companies that are always launching new products might appreciate it: eg Proctor and Gamble (though in their market, there are only a few competitors and they are well-known...). It might help in industries that are often disrupted - ie high-tech industries. Examples might be the established competitors of YC companies. If you could demonstrate legitimacy, big enterprise might pay a lot (perhaps equivalent to how much it would cost to hire someone to do the same work manually). You can be an arms dealer, and sell to both sides. :-) That's the nature of competition. EDIT http://www.mirah.org http://www.mirah.org hasn't got results yet. JRuby's competitors are: Groovy and http://rjb.rubyforge.org.. http://rjb.rubyforge.org... There's a japanese (I think) news site. That might be helpful, if you can detect similarities across languages...
joshu 16y ago
curiously, delicious itself used to have a URL-URL recommender, but it didn't really scale and I had to turn it off.
- photon_off 16y ago
  First off -- thanks for delicious. Not sure if it's apparent, but moreofit uses its rich data. If I might ask, how did the original URL-URL recommended work? Was it simply a query to find other URLs with the most matched tags, or was tag order also taken into account?
  - joshu 16y ago
    It looked at the other URLs the users bookmarked UN the same way, normalized partially for popularity. So I never looked at the global tags - user1's use of tag x was distinct from user2's. Man, I miss that dataset.
todayiamme 16y ago
Can I ask you a question? How computationally expensive is it?
- photon_off 16y ago
  Amazingly, matching can be done with a single MySQL query that will take anywhere between .05 - 2.0 seconds (on average .25 if I do a whole batch of URLs) -- it depends on how popular the tags of the URL are. Those numbers were too large for my liking, so I cooked up a Java program that loads the relevant db tables into memory and does the matching itself. The program uses about 80MB (though it only 'really' uses 35MB -- damn Java heap) of memory and takes .008s to compute 500 matches for a given URL -- which I think is suitably fast. However, its 8ms because I've optimized it quite well -- a brute force match would take URL_INDEX_SIZE^2 * 2 * log2(NUMBER_OF_TAGS_PER_URL) multiplications. The main bottleneck is inserting those 500 rows into a separate table, which is currently taking .1 seconds. I'm sure there's a better way to do this... I'll have to look into it if it becomes a scaling pain.
  - todayiamme 16y ago
    Is there any way to do this once and for all for a link? As you're crawling the database you look up the link and do this once and for all for it. So that the next time a user asks a query if it's over there you can directly return that, if it isn't then you can use more resources to crunch it for them. Is this possible? Also, is there any way I can help you to make this? I don't expect any monetary benefit. I just want to learn.
  - mjw 16y ago
    Interesting. Sounds like you're effectively doing a sparse matrix-vector multiplication for each request? If you want to buy yourself a bit more time performance-wise, might be worth looking into using a numerical computing library for sparse linear algebra which has optimised routines for this (e.g. http://math.nist.gov/spblas/ http://math.nist.gov/spblas/ at a first glance). Looks like there's been some recent work on parallelising this via the GPU too: http://graphics.cs.uiuc.edu/~wnbell/ http://graphics.cs.uiuc.edu/~wnbell/ Some kind of dimensionality reduction is probably one of the next steps scalability-wise though, and should hopefully improve the results too if you tune it right. Latent Semantic Analysis the term to google if you're not already aware of it. Testing the success of your predictive model might be tricky though in the absence of user-specific tagging data. I experimented with this stuff a while back for music recommendations - got as far as doing some simple SVD-based dimensionality reduction but the results weren't as good as I'd hoped and it got put on the backburner. So, respect for polishing this one up for launch :)
lkozma 16y ago
Nice idea, well done. Probably you could use PCA to reduce the dimensionality of the data before doing the nn search. This seems a bit like the netflix movie recommendations data so you could probably use ideas from there. Sorry, probably you already do :)
- lkozma 16y ago
  Oh, I got it now, you use the tags attached to the link. First I thought that you use the individual bookmarking of each user which would be a HUGE binary, sparse matrix, however, that could really be reduced with PCA and it would give a "people with similar tastes to those who bookmarked your site, also bookmarked ... " effect, possibly an even richer source of information. Your service is great anyhow, got to love a site that says that my site is similar to paulgraham.com and joelonsoftware.com http://www.moreofit.com/similar-to/lkozma.net/Top_10_Sites_Like_Lkozma/ http://www.moreofit.com/similar-to/lkozma.net/Top_10_Sites_L...
  - photon_off 16y ago
    I use the tags attached to a URL, as well as their relative magnitudes. In essence, each page has a 10-vector in R^60,000, and moreofit's job is to find kNN. If you click "auto-suggest" moreofit will check your browser history for the top 2500 or so URLs, to give you a jumping off point. I'm not currently logging this data, because I don't think it's nice to, but I think it'd provide a gold mine of information to mull over.
    - lkozma 16y ago
      OK, I got it. Well, each URL is still a (60,000)-dimensional vector, just that 59990 entries happen to be zero and the 10 that are non-zero have certain values which show how many times that tag was chosen. Problem is that not the same ten are nonzero for each URL. Did I get it right? So what I was suggesting was to perform some off-the-shelf projection method on this data, for example principal components analysis (PCA) which will project your data from 60,000 into say 20-dimensions (as you choose) and then for each URL you will have just 20 numbers which are directly comparable across URLs. You can then more easily do k-nn on this data. The advantage is that: 1) you reduce noise and the data will be easier to handle 2) the obtained new "components" can be interpreted as more subtle descriptors of the URLs than the original tags. For example one component could become the "techiness" of the URL, the other the "personalness" the other the "comediness", etc. All this comes out automatically from the data and you can interpret it by looking at what mixture each component will have of the original features (the tags).. This is just a suggestion, I mean, the algorithm seems to be great as it is, but still, the technique seems a bit crude: if one page is labeled "tech" the other "technology", I think it will miss the connection. However, PCA could find out that these features are related. etc. etc. etc.
lkozma 16y ago
Another suggestion: on the result page make the main link for each result go to the "moreofit" page of the given link, rather than the actual external websites. (just like you do for the top pages). You want to keep people on your site :) and it's probably better for pagerank also. Have instead a small arrow icon for the actual external links (that would be useful for the top pages also, there is no external link there at the moment).
- photon_off 16y ago
  Interesting suggestion. There are little "moreofit" links on each result (as well as for the domain of the link)... I guess they are not prominent enough? I'd like to keep the results rather straightforward. What I will do is link them all to a redirect script so I can log which results are most popular and possible create a smarter results algorithm. How does including external links affect pagerank?
  - lkozma 16y ago
    Well, the more internal links and the less external links you have, the better for pagerank. PR just formalizes the random surfer model: what is the probability that someone who just randomly clicks links is on your page after n clicks.
fmora 16y ago
I'm envious. Looks like something that could really take off. Congratulations.

prawn 16y ago

Just a random, half-baked airport musing, but what if you had a related service that filtered out links which were similar and likely to be competitors. Someone might want to restrict what sites their crowdsourced content might link to?

photon_off 16y ago
I've considered the uses of moreofit for competitor-like stuff like this. I could make a service which (when installed on your site) scans users' histories for the top 500 competitor websites, aggregate those results, and let you know which competitors websites your visitors have most often been to. Also, if they've been to X competitor, theres a Y% chance they've been to Z competitor. I haphazardly concluded that the value seems to be in the service itself, not the "finding competitors" aspect. You likely know who your competitors are, or can find out with a 1 time search on a similarity site such as mine. That doesn't mean the idea has no legs. I imagine it's something people would pay for, and probably already are. It's just not my cup of tea at the moment to be censoring or sniffing people's histories.
- 10ren 16y ago
  The recent posts on how easy it is to launch a start up these days screamed "glut" to me. One solution is to target a niche that wouldn't be well enough served by a universal solution. Anyway, what I'm getting at is that monitoring competitor popularity would be a valuable service (including an early warning system for detecting emerging competitors.) I think many of us do this every so often manually; and/or have google alerts set up for it. But a service that just does this would broaden the appeal - especially if you could somehow show that your estimates are more accurate... I have no idea of how you could do this... I guess you'd need a genuinely accurate measure, and then compare yours and manual searches with it. But if you could demonstrate that credibly and compellingly, it'd be gold. I don't think you need to censor or sniff to do this.
  - photon_off 16y ago
    If I understand you correctly, the service would work as follows: You enter a URL you want to monitor. My service would show you trends of similar websites to watch out for. It would base it on some sort of secret sauce that measures similarity and popularity of competitors. Correct? That's an interesting idea. What types of current websites/businesses do you think could benefit from this right now? Could you give me examples? This could help me see if it's possible to monitor similar websites and develop trends. Measuring a website's popularity is pretty straightforward -- you make some sort of formula based on how often that URL is discussed (from backtype), linked to (various sources), bookmarked, and ranked (alexa, quantcast, et al). It's a matter of mashing up that data with the similarity data (over time) to determine the trends of similar sites.
    - 10ren 16y ago
      That's right - though you enter your own URL. Extra idea: it might also show you complements (products/services that fit in nicely with yours; these are usually pretty obvious in established industries - razors+shaving cream; milk+eggs; tyres +brakepads - but not so obvious in emerging industries. You might not have even thought of the category of a certain complement, let alone the name of it, nor specific brands that do it. It's valuable, because you might consider targeting people who buy the complements. It's a kind of market-research. (Unhelpfully) I think every website could benefit from it. Probably emergent and fast-moving markets would benefit most (because they don't yet know their competitors/complements + it's all changing very fast anyway). A very specific start would be... y-combinator offerings (that have just launched - not the established onces); for fun, you could also do it for emerging languages (like Mirah). These aren't ideal biz markets, but they would gain attention here on HN, and might validate the basic idea (to see if it's possible.) For example, if you were able to find a competitor to a YC company that they didn't know about, it would be pretty compelling evidence (and will surely happen, since startups rarely look very closely for competitors; they are too busy doing. Which is a good thing. Knowing about competitors too early can be artificially discouraging. So this is probably most helpful for companies that already have enough traction that they wouldn't be discouraged. Competitors are good for giving you ideas; for demonstrating market existence (good for raising funds...); for suggesting markets you hadn't thought of. Complements are always good. Perhaps also, companies that are always launching new products might appreciate it: eg Proctor and Gamble (though in their market, there are only a few competitors and they are well-known...). It might help in industries that are often disrupted - ie high-tech industries. Examples might be the established competitors of YC companies. If you could demonstrate legitimacy, big enterprise might pay a lot (perhaps equivalent to how much it would cost to hire someone to do the same work manually). You can be an arms dealer, and sell to both sides. :-) That's the nature of competition. EDIT http://www.mirah.org http://www.mirah.org hasn't got results yet. JRuby's competitors are: Groovy and http://rjb.rubyforge.org.. http://rjb.rubyforge.org... There's a japanese (I think) news site. That might be helpful, if you can detect similarities across languages...

joshu 16y ago

curiously, delicious itself used to have a URL-URL recommender, but it didn't really scale and I had to turn it off.

photon_off 16y ago
First off -- thanks for delicious. Not sure if it's apparent, but moreofit uses its rich data. If I might ask, how did the original URL-URL recommended work? Was it simply a query to find other URLs with the most matched tags, or was tag order also taken into account?
- joshu 16y ago
  It looked at the other URLs the users bookmarked UN the same way, normalized partially for popularity. So I never looked at the global tags - user1's use of tag x was distinct from user2's. Man, I miss that dataset.

todayiamme 16y ago

Can I ask you a question? How computationally expensive is it?

photon_off 16y ago
Amazingly, matching can be done with a single MySQL query that will take anywhere between .05 - 2.0 seconds (on average .25 if I do a whole batch of URLs) -- it depends on how popular the tags of the URL are. Those numbers were too large for my liking, so I cooked up a Java program that loads the relevant db tables into memory and does the matching itself. The program uses about 80MB (though it only 'really' uses 35MB -- damn Java heap) of memory and takes .008s to compute 500 matches for a given URL -- which I think is suitably fast. However, its 8ms because I've optimized it quite well -- a brute force match would take URL_INDEX_SIZE^2 * 2 * log2(NUMBER_OF_TAGS_PER_URL) multiplications. The main bottleneck is inserting those 500 rows into a separate table, which is currently taking .1 seconds. I'm sure there's a better way to do this... I'll have to look into it if it becomes a scaling pain.
- todayiamme 16y ago
  Is there any way to do this once and for all for a link? As you're crawling the database you look up the link and do this once and for all for it. So that the next time a user asks a query if it's over there you can directly return that, if it isn't then you can use more resources to crunch it for them. Is this possible? Also, is there any way I can help you to make this? I don't expect any monetary benefit. I just want to learn.
- mjw 16y ago
  Interesting. Sounds like you're effectively doing a sparse matrix-vector multiplication for each request? If you want to buy yourself a bit more time performance-wise, might be worth looking into using a numerical computing library for sparse linear algebra which has optimised routines for this (e.g. http://math.nist.gov/spblas/ http://math.nist.gov/spblas/ at a first glance). Looks like there's been some recent work on parallelising this via the GPU too: http://graphics.cs.uiuc.edu/~wnbell/ http://graphics.cs.uiuc.edu/~wnbell/ Some kind of dimensionality reduction is probably one of the next steps scalability-wise though, and should hopefully improve the results too if you tune it right. Latent Semantic Analysis the term to google if you're not already aware of it. Testing the success of your predictive model might be tricky though in the absence of user-specific tagging data. I experimented with this stuff a while back for music recommendations - got as far as doing some simple SVD-based dimensionality reduction but the results weren't as good as I'd hoped and it got put on the backburner. So, respect for polishing this one up for launch :)

lkozma 16y ago

Nice idea, well done. Probably you could use PCA to reduce the dimensionality of the data before doing the nn search. This seems a bit like the netflix movie recommendations data so you could probably use ideas from there. Sorry, probably you already do :)

lkozma 16y ago
Oh, I got it now, you use the tags attached to the link. First I thought that you use the individual bookmarking of each user which would be a HUGE binary, sparse matrix, however, that could really be reduced with PCA and it would give a "people with similar tastes to those who bookmarked your site, also bookmarked ... " effect, possibly an even richer source of information. Your service is great anyhow, got to love a site that says that my site is similar to paulgraham.com and joelonsoftware.com http://www.moreofit.com/similar-to/lkozma.net/Top_10_Sites_Like_Lkozma/ http://www.moreofit.com/similar-to/lkozma.net/Top_10_Sites_L...
- photon_off 16y ago
  I use the tags attached to a URL, as well as their relative magnitudes. In essence, each page has a 10-vector in R^60,000, and moreofit's job is to find kNN. If you click "auto-suggest" moreofit will check your browser history for the top 2500 or so URLs, to give you a jumping off point. I'm not currently logging this data, because I don't think it's nice to, but I think it'd provide a gold mine of information to mull over.
  - lkozma 16y ago
    OK, I got it. Well, each URL is still a (60,000)-dimensional vector, just that 59990 entries happen to be zero and the 10 that are non-zero have certain values which show how many times that tag was chosen. Problem is that not the same ten are nonzero for each URL. Did I get it right? So what I was suggesting was to perform some off-the-shelf projection method on this data, for example principal components analysis (PCA) which will project your data from 60,000 into say 20-dimensions (as you choose) and then for each URL you will have just 20 numbers which are directly comparable across URLs. You can then more easily do k-nn on this data. The advantage is that: 1) you reduce noise and the data will be easier to handle 2) the obtained new "components" can be interpreted as more subtle descriptors of the URLs than the original tags. For example one component could become the "techiness" of the URL, the other the "personalness" the other the "comediness", etc. All this comes out automatically from the data and you can interpret it by looking at what mixture each component will have of the original features (the tags).. This is just a suggestion, I mean, the algorithm seems to be great as it is, but still, the technique seems a bit crude: if one page is labeled "tech" the other "technology", I think it will miss the connection. However, PCA could find out that these features are related. etc. etc. etc.

lkozma 16y ago

Another suggestion: on the result page make the main link for each result go to the "moreofit" page of the given link, rather than the actual external websites. (just like you do for the top pages). You want to keep people on your site :) and it's probably better for pagerank also. Have instead a small arrow icon for the actual external links (that would be useful for the top pages also, there is no external link there at the moment).

photon_off 16y ago
Interesting suggestion. There are little "moreofit" links on each result (as well as for the domain of the link)... I guess they are not prominent enough? I'd like to keep the results rather straightforward. What I will do is link them all to a redirect script so I can log which results are most popular and possible create a smarter results algorithm. How does including external links affect pagerank?
- lkozma 16y ago
  Well, the more internal links and the less external links you have, the better for pagerank. PR just formalizes the random surfer model: what is the probability that someone who just randomly clicks links is on your page after n clicks.

fmora 16y ago

I'm envious. Looks like something that could really take off. Congratulations.