5 ms·

Giving Away Our Recommendation Engine

297 points by kky 12y ago

ASquare 12y ago
I'm sure plenty of good karma (even the non-HN kind) is headed your way - kudos.
pixelmade 12y ago
I'm curious what the business case was for open sourcing the code. Maybe to create an ecosystem?
- lotsofcows 12y ago
  "We’re giving over a year’s worth of work on our recommendation engine away because we want to earn your business on our platform."
- showerst 12y ago
  From the "What you'll need" section of the first tutorial - A Mortar account. You can sign up for a free Public account with Mortar here. If you want to keep your customized recommendation engine code private, you will need a Solo-level account ($99/month). Beyond that, you'll only pay for your actual usage of AWS cloud services (we never add an upcharge). Kudos for the open source, but it looks like to actually use this for business you'll still need to pay. Unless i'm misreading it, "Open source but you'll still have to go through our platform" is pretty disingenuous.
  - ethanbond 12y ago
    It reads like "open source but not free to make proprietary." First, it's awesome just to see source as something to learn from. Second, it seems reasonable they don't want people forking, modifying then profiting from their work without contributing back to it - either by also releasing source or by paying. I think it's a nice model actually.
    - catern 12y ago
      It's called the GPL. If that's really the model they're trying to create, it would be nice if they just used the GPL.
      - deleted 12y ago
        [deleted]
    - kirushik 12y ago
      It's opensourced under Apache 2.0 license. That means, it's free as in freedom, with all the legal ability to be forked and profited from. AFAIK, the only substantial difference between Apache 2.0 and GPLv3 is that APL2 is not copyleft license. That means, one's fork is not even required to remain opensourced (as far as it contains a reference to original APL-licensed version). So either Mortar opensourced some feature-crippled fragment of their platform, and it relies on features from their proprietary platform heavily; or the statement of requirement Mortar account is property of the Tutorial's approach, not the opensourced code itself.
  - gmisra 12y ago
    The code is all released under the Apache 2.0 license, so calling such an action "disingenuous" is itself disingenuous, (imo).
    - forgottenpass 12y ago
      The code is all released under Apache, but is all the code needed to use this thing released? If the parent poster is accurate that a user still needs to engage with their platform, this conversation is just pedantics and sophistry over what "open" means.
      - rjurney 12y ago
        You can run the code on your local machine, on Amazon ElasticMapReduce, or on your own Hadoop cluster. This really is a give-away of useful open source code.
    - showerst 12y ago
      I'm not trying to start a flame war over the use of the word open, and I think it's great that they're releasing code that others can learn from. It's just that making a big press release and blog post that brags about open sourcing, vs the reality that you can't actually do anything substantial with the code without paying for it... it seems off to me. I get what they're trying to do, but to me the whole point of OS code is that you can self-host, and/or modify it for business use if you so choose. To me this would be better served by advertising "We like you so much, we're giving away access to our service for free for noncommercial and test use, and opening up the code to the library so you can see how it works", but that's less interesting as click bait. Maybe i'm just mis-reading the whole thing and you can self-host.
      - gmisra 12y ago
        Thanks for the clarification. I'm in agreement with your opinions regarding false promises of open source, and also that this is increasingly a problem. However, I don't think that actually applies here. Specifics: 1. Everything in this github repository (https://github.com/mortardata/mortar-recsys https://github.com/mortardata/mortar-recsys) appears to be truly open - it's just a bunch of pig scripts, some java UDF definitions, and some python management code. There doesn't appear to be any dependencies on proprietary MortarData anything. All the code is licensed under the Apache 2.0 license. 2. The blog post states: " You can run this code anywhere. It’s built on widely-adopted open source technologies—Hadoop, Pig, and Python. But we think you’ll want to use our platform."
olidb2 12y ago
FWIW we've been using the mortar platform to run large pig jobs without a fuss at http://datadog.com http://datadog.com and we've been very happy with it. Glad to see them contribute their recommender code too.
- alecsmart1 12y ago
  Can you please suggest why you need a recommendation engine for datadog?
  - olidb2 12y ago
    We don't use the recommendation engine but the underlying platform, which makes it really simple to write and run pig jobs. Though the majority of our business deals with real-time data processing, the ability to crunch numbers in batch without dev or ops overhead is attractive and well worth the price to us.
    - kldavenport 12y ago
      Is this better or similar to Hue?
contingencies 12y ago
So hang on, what exactly is a recommendation engine? They give examples of LinkedIn (people you may know) and Amazon (presumably other people who bought this, so-and-so's list of such-a-subject books). That makes sense, though the segment of businesses that may actually benefit seems limited. Social stuff, sure. Most of us? What's the minimum recommendable-entity/category-or-user threshold that this makes sense for? Is success with these sorts of engines merely a reflector of poor UI design in your normal UX? (Of the above examples, the first seems very unidimensional - in that it's basically a simple graph distance - and the latter also rather rudimentary and often irrelevant). So what exactly is this thing providing? Graph analysis? I think not. It reads more like some kind of raw timestamped user behavioural event data processing to infer relationships between users or products they interact with. Reading through the docs it seems this is a layer on top of Apache Pig (https://pig.apache.org/ https://pig.apache.org/) - a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. I think clarity in explaining this thing could be improved, particularly selling clearly what a recommendation is and when its useful. Using phrases like "award winning" doesn't help. PS. Why all the downvotes? Sheesh.
- alexhawdon 12y ago
  https://www.coursera.org/course/recsys https://www.coursera.org/course/recsys will provide you with a good introduction to the topic
  - contingencies 12y ago
    Thanks. For others who are interested, that course apparently uses a different piece of software called LensKit http://lenskit.grouplens.org/ http://lenskit.grouplens.org/ Could anyone summarize the difference between Pig and LensKit when applied to recommendation systems?
    - rjurney 12y ago
      The mortar recommendation system is written in Pig, and can scale to any data size, even petabytes. The lenskit tool cannot.
dsheth 12y ago
Anyone know of any comparisons between this and Apache Mahout? I've used Mahout's Item-Item recommender in the past, and it's worked well, just wondering if there were advantages to this recommender.
mck- 12y ago
Fwiw, here are two light-weight feature-based recommendation engines I built for Node.js (for situations where you have the cold-start problem and therefore can't rely on user/item based collaborative filtering): Alike [1] and Look-Alike [2] [1] https://github.com/axiomzen/Alike https://github.com/axiomzen/Alike [2] https://github.com/axiomzen/Look-Alike https://github.com/axiomzen/Look-Alike
- yblu 12y ago
  Thanks for sharing. What do you mean by the "cold-start" problem? Just want to know exactly when I can use your engines.
  - elwell 12y ago
    Just speculating: not having a recommendation when you first begin because you don't have any data.
    - mck- 12y ago
      Exactly right. I borrowed that term from Chapter 2 on Collective Intelligence [1] [1] http://shop.oreilly.com/product/9780596529321.do http://shop.oreilly.com/product/9780596529321.do
X4 12y ago
WOW, Awesome Documentation and Product!! Kudos and Greetings from Germany 😊 Those who know what Hadoop, Pig and the whole "Data Science Stack" is, will find this surely useful.

297 points by kky 12y ago

ASquare 12y ago

I'm sure plenty of good karma (even the non-HN kind) is headed your way - kudos.

pixelmade 12y ago

I'm curious what the business case was for open sourcing the code. Maybe to create an ecosystem?

lotsofcows 12y ago
"We’re giving over a year’s worth of work on our recommendation engine away because we want to earn your business on our platform."
showerst 12y ago
From the "What you'll need" section of the first tutorial - A Mortar account. You can sign up for a free Public account with Mortar here. If you want to keep your customized recommendation engine code private, you will need a Solo-level account ($99/month). Beyond that, you'll only pay for your actual usage of AWS cloud services (we never add an upcharge). Kudos for the open source, but it looks like to actually use this for business you'll still need to pay. Unless i'm misreading it, "Open source but you'll still have to go through our platform" is pretty disingenuous.
- ethanbond 12y ago
  It reads like "open source but not free to make proprietary." First, it's awesome just to see source as something to learn from. Second, it seems reasonable they don't want people forking, modifying then profiting from their work without contributing back to it - either by also releasing source or by paying. I think it's a nice model actually.
  - catern 12y ago
    It's called the GPL. If that's really the model they're trying to create, it would be nice if they just used the GPL.
    - deleted 12y ago
      [deleted]
  - kirushik 12y ago
    It's opensourced under Apache 2.0 license. That means, it's free as in freedom, with all the legal ability to be forked and profited from. AFAIK, the only substantial difference between Apache 2.0 and GPLv3 is that APL2 is not copyleft license. That means, one's fork is not even required to remain opensourced (as far as it contains a reference to original APL-licensed version). So either Mortar opensourced some feature-crippled fragment of their platform, and it relies on features from their proprietary platform heavily; or the statement of requirement Mortar account is property of the Tutorial's approach, not the opensourced code itself.
- gmisra 12y ago
  The code is all released under the Apache 2.0 license, so calling such an action "disingenuous" is itself disingenuous, (imo).
  - forgottenpass 12y ago
    The code is all released under Apache, but is all the code needed to use this thing released? If the parent poster is accurate that a user still needs to engage with their platform, this conversation is just pedantics and sophistry over what "open" means.
    - rjurney 12y ago
      You can run the code on your local machine, on Amazon ElasticMapReduce, or on your own Hadoop cluster. This really is a give-away of useful open source code.
  - showerst 12y ago
    I'm not trying to start a flame war over the use of the word open, and I think it's great that they're releasing code that others can learn from. It's just that making a big press release and blog post that brags about open sourcing, vs the reality that you can't actually do anything substantial with the code without paying for it... it seems off to me. I get what they're trying to do, but to me the whole point of OS code is that you can self-host, and/or modify it for business use if you so choose. To me this would be better served by advertising "We like you so much, we're giving away access to our service for free for noncommercial and test use, and opening up the code to the library so you can see how it works", but that's less interesting as click bait. Maybe i'm just mis-reading the whole thing and you can self-host.
    - gmisra 12y ago
      Thanks for the clarification. I'm in agreement with your opinions regarding false promises of open source, and also that this is increasingly a problem. However, I don't think that actually applies here. Specifics: 1. Everything in this github repository (https://github.com/mortardata/mortar-recsys https://github.com/mortardata/mortar-recsys) appears to be truly open - it's just a bunch of pig scripts, some java UDF definitions, and some python management code. There doesn't appear to be any dependencies on proprietary MortarData anything. All the code is licensed under the Apache 2.0 license. 2. The blog post states: " You can run this code anywhere. It’s built on widely-adopted open source technologies—Hadoop, Pig, and Python. But we think you’ll want to use our platform."

olidb2 12y ago

FWIW we've been using the mortar platform to run large pig jobs without a fuss at http://datadog.com http://datadog.com and we've been very happy with it. Glad to see them contribute their recommender code too.

alecsmart1 12y ago
Can you please suggest why you need a recommendation engine for datadog?
- olidb2 12y ago
  We don't use the recommendation engine but the underlying platform, which makes it really simple to write and run pig jobs. Though the majority of our business deals with real-time data processing, the ability to crunch numbers in batch without dev or ops overhead is attractive and well worth the price to us.
  - kldavenport 12y ago
    Is this better or similar to Hue?

contingencies 12y ago

So hang on, what exactly is a recommendation engine? They give examples of LinkedIn (people you may know) and Amazon (presumably other people who bought this, so-and-so's list of such-a-subject books). That makes sense, though the segment of businesses that may actually benefit seems limited. Social stuff, sure. Most of us? What's the minimum recommendable-entity/category-or-user threshold that this makes sense for? Is success with these sorts of engines merely a reflector of poor UI design in your normal UX? (Of the above examples, the first seems very unidimensional - in that it's basically a simple graph distance - and the latter also rather rudimentary and often irrelevant). So what exactly is this thing providing? Graph analysis? I think not. It reads more like some kind of raw timestamped user behavioural event data processing to infer relationships between users or products they interact with. Reading through the docs it seems this is a layer on top of Apache Pig (https://pig.apache.org/ https://pig.apache.org/) - a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. I think clarity in explaining this thing could be improved, particularly selling clearly what a recommendation is and when its useful. Using phrases like "award winning" doesn't help. PS. Why all the downvotes? Sheesh.

alexhawdon 12y ago
https://www.coursera.org/course/recsys https://www.coursera.org/course/recsys will provide you with a good introduction to the topic
- contingencies 12y ago
  Thanks. For others who are interested, that course apparently uses a different piece of software called LensKit http://lenskit.grouplens.org/ http://lenskit.grouplens.org/ Could anyone summarize the difference between Pig and LensKit when applied to recommendation systems?
  - rjurney 12y ago
    The mortar recommendation system is written in Pig, and can scale to any data size, even petabytes. The lenskit tool cannot.

dsheth 12y ago

Anyone know of any comparisons between this and Apache Mahout? I've used Mahout's Item-Item recommender in the past, and it's worked well, just wondering if there were advantages to this recommender.

mck- 12y ago

Fwiw, here are two light-weight feature-based recommendation engines I built for Node.js (for situations where you have the cold-start problem and therefore can't rely on user/item based collaborative filtering): Alike [1] and Look-Alike [2] [1] https://github.com/axiomzen/Alike https://github.com/axiomzen/Alike [2] https://github.com/axiomzen/Look-Alike https://github.com/axiomzen/Look-Alike

yblu 12y ago
Thanks for sharing. What do you mean by the "cold-start" problem? Just want to know exactly when I can use your engines.
- elwell 12y ago
  Just speculating: not having a recommendation when you first begin because you don't have any data.
  - mck- 12y ago
    Exactly right. I borrowed that term from Chapter 2 on Collective Intelligence [1] [1] http://shop.oreilly.com/product/9780596529321.do http://shop.oreilly.com/product/9780596529321.do

X4 12y ago

WOW, Awesome Documentation and Product!! Kudos and Greetings from Germany 😊 Those who know what Hadoop, Pig and the whole "Data Science Stack" is, will find this surely useful.