4 ms·

I'd agree with all that, but add a disclaimer. The rise of cheap and easy solutions for handling Big Data makes a lot of new applications feasible. All those co

I'd agree with all that, but add a disclaimer. The rise of cheap and easy solutions for handling Big Data makes a lot of new applications feasible. All those cool correlations that Amazon does to suggest new books? You can do that at home now. Got a problem that would benefit from running an analysis of hundreds of millions of pages? You don't have to be Google to implement that. It is a bit maddening to see people applying the NoSQL hammer to problems that don't need scalability, but it's partly driven by the realization that they open up opportunities to build entirely new applications.

mcav 17y ago
I know there's Hadoop, etc for map/reduce; what is available for correlations like Amazon?
- bad_user 17y ago
  Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Personally I don't trust NoSql solutions because they aren't mature enough. I've had little problems using MySql/Postgres and never had loss of data ... and I can always come up with a non-relational scheme latter built on top ... when I really need it. The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. I've also used MySql instead of Memcached successfully for a website with tens of millions of hits per day, preferring it because Memcached crashes on me a lot and there are big differences between versions I don't want to worry about ... and yeah, I know async reads are problematic with Mysql/Postgres, but I just threw a web service in front of it, and when using the Memory storage engine it really kicks ass.
  - strlen 17y ago
    > Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Many of the algorithms for recommender systems are especially known for being parallelizable and would be extremely difficult to reliably and quickly implement on top of MySQL (or Postgres or Oracle). I can't even think of a way to do so without making a fairly complex ETL cycle with a separate DWH and a main serving DB (with points at which the site becomes degraded and possibly unavailable). As I've mentioned earlier, with a system like Hadoop, you could start off by implementing these algorithms in Perl or Python (making use extensive libraries already available in the language). Recommendations may not be real time, but there's a huge time difference between doing an even fairly simple "market basket"-type analysis for a data set orders of magnitude difference smaller than Amazon's in MySQL and doing this on top of Hadoop + HBase/Voldemort/Cassandra (i.e., compute in Hadoop, serve from a NoSQL db which offers Hadoop integration). You could even start by doing the computation in Hadoop and loading the data into MySQL (while, perhaps, serving the data from a slave db). > Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Your start-up doesn't need to have tons of users to have tons of data. This is especially true if you're building a start-up to make sense of the data that's already out there (some YC examples are Flightcaster, Directed Edge). For a consumer start-up, ability to have cheap and scalable storage means you can record data much more frequently. > The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. Isn't this the point of separating your data model from your presentation layer? Begin with MySQL (or even simpler, with BerkeleyDB) -- change it to whatever else suits your needs.
- strlen 17y ago
  I think what the parent meant is that there are now tools available to implement the algorithms used by Amazon et al for recommendations. It isn't about having more users, it's about being able to keep track of great deal more data and use that data at great deal more points in the applications. That's what Hadoop, NoSQL et al allow you to do (e.g. Voldemort's and HBase's ability efficiently load and serve data prepared in Hadoop; afaik Cassandra has Hadoop integration now as well). Now even if you are okay with low throughput (you don't see yourself become a major Internet destination), you can still benefit from non-relational approaches. Implementing a recommender system or PageRank on top of a relational database is simply painful and requires expertise in that system's SQL dialect (someone who's done this on top of Oracle won't be able to do this -- in the same amount of time -- on top of MySQL or Postgres). With Hadoop you can start with a Perl or Python script (and make use of libraries like scipy or pysvm) and then gradually re-write the performance critical parts in Java. Even if you aren't and will never get to Facebook's or Amazon's scale, you're now able to build intelligent applications on top of your data. You can also sleep sound at night (i.e., not in front of a terminal tailing your logs or watching ganglia) knowing that a load spike (which could be start of a new financial quarter, holiday shopping season; something as positive as a NYTimes article or -- much more likely -- as negative as a DDOS attack) won't bring you to an outage. The algorithms themselves, however, are very difficult and require a background in machine learning; that is even to use an existing implementation of these algorithms (e.g. Apache Mahout). I am not an expert in these topics. I took a graduate course in machine learning, but that only demonstrated to me how little I know and how much more I have left to learn. It's this expertise that coupled with ability to implement a scalable infrastructure with open source/in-house developed software (instead of relying on commercial vendors) that makes your a business idea (you know, the one that the "business guys" think they can hire bunch of MSCEs to build in ASP.NET/J2EE/PHP/RoR) defensible from some "kid" in Harvard dorms (especially if that kid declined a $1mm starting bonus to work for Microsoft instead of going to Harvard). Don't get me wrong. SQL is an amazingly example of a domain specific language; RDBMs are wonderful works of software engineering. SQL RDBMs are great for many applications (e.g., ERP systems and their functional equivalents like the university registration system patio11 mentioned). Yet they're not made to solve the kinds of problems/create the kind of applications that simply didn't exist when they were created. The fact that they can at times be used for this doesn't mean they should: I can write Russian with Latin instead of Cyrillic Alphabet (or write Lisp in vi instead of emacs), but it's a lot better to use the correct tool.

mcav 17y ago

I know there's Hadoop, etc for map/reduce; what is available for correlations like Amazon?

bad_user 17y ago
Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Personally I don't trust NoSql solutions because they aren't mature enough. I've had little problems using MySql/Postgres and never had loss of data ... and I can always come up with a non-relational scheme latter built on top ... when I really need it. The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. I've also used MySql instead of Memcached successfully for a website with tens of millions of hits per day, preferring it because Memcached crashes on me a lot and there are big differences between versions I don't want to worry about ... and yeah, I know async reads are problematic with Mysql/Postgres, but I just threw a web service in front of it, and when using the Memory storage engine it really kicks ass.
- strlen 17y ago
  > Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Many of the algorithms for recommender systems are especially known for being parallelizable and would be extremely difficult to reliably and quickly implement on top of MySQL (or Postgres or Oracle). I can't even think of a way to do so without making a fairly complex ETL cycle with a separate DWH and a main serving DB (with points at which the site becomes degraded and possibly unavailable). As I've mentioned earlier, with a system like Hadoop, you could start off by implementing these algorithms in Perl or Python (making use extensive libraries already available in the language). Recommendations may not be real time, but there's a huge time difference between doing an even fairly simple "market basket"-type analysis for a data set orders of magnitude difference smaller than Amazon's in MySQL and doing this on top of Hadoop + HBase/Voldemort/Cassandra (i.e., compute in Hadoop, serve from a NoSQL db which offers Hadoop integration). You could even start by doing the computation in Hadoop and loading the data into MySQL (while, perhaps, serving the data from a slave db). > Maybe it is for Amazon which got plenty of data to analyze, but not for a startup. Your start-up doesn't need to have tons of users to have tons of data. This is especially true if you're building a start-up to make sense of the data that's already out there (some YC examples are Flightcaster, Directed Edge). For a consumer start-up, ability to have cheap and scalable storage means you can record data much more frequently. > The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start. Isn't this the point of separating your data model from your presentation layer? Begin with MySQL (or even simpler, with BerkeleyDB) -- change it to whatever else suits your needs.
strlen 17y ago
I think what the parent meant is that there are now tools available to implement the algorithms used by Amazon et al for recommendations. It isn't about having more users, it's about being able to keep track of great deal more data and use that data at great deal more points in the applications. That's what Hadoop, NoSQL et al allow you to do (e.g. Voldemort's and HBase's ability efficiently load and serve data prepared in Hadoop; afaik Cassandra has Hadoop integration now as well). Now even if you are okay with low throughput (you don't see yourself become a major Internet destination), you can still benefit from non-relational approaches. Implementing a recommender system or PageRank on top of a relational database is simply painful and requires expertise in that system's SQL dialect (someone who's done this on top of Oracle won't be able to do this -- in the same amount of time -- on top of MySQL or Postgres). With Hadoop you can start with a Perl or Python script (and make use of libraries like scipy or pysvm) and then gradually re-write the performance critical parts in Java. Even if you aren't and will never get to Facebook's or Amazon's scale, you're now able to build intelligent applications on top of your data. You can also sleep sound at night (i.e., not in front of a terminal tailing your logs or watching ganglia) knowing that a load spike (which could be start of a new financial quarter, holiday shopping season; something as positive as a NYTimes article or -- much more likely -- as negative as a DDOS attack) won't bring you to an outage. The algorithms themselves, however, are very difficult and require a background in machine learning; that is even to use an existing implementation of these algorithms (e.g. Apache Mahout). I am not an expert in these topics. I took a graduate course in machine learning, but that only demonstrated to me how little I know and how much more I have left to learn. It's this expertise that coupled with ability to implement a scalable infrastructure with open source/in-house developed software (instead of relying on commercial vendors) that makes your a business idea (you know, the one that the "business guys" think they can hire bunch of MSCEs to build in ASP.NET/J2EE/PHP/RoR) defensible from some "kid" in Harvard dorms (especially if that kid declined a $1mm starting bonus to work for Microsoft instead of going to Harvard). Don't get me wrong. SQL is an amazingly example of a domain specific language; RDBMs are wonderful works of software engineering. SQL RDBMs are great for many applications (e.g., ERP systems and their functional equivalents like the university registration system patio11 mentioned). Yet they're not made to solve the kinds of problems/create the kind of applications that simply didn't exist when they were created. The fact that they can at times be used for this doesn't mean they should: I can write Russian with Latin instead of Cyrillic Alphabet (or write Lisp in vi instead of emacs), but it's a lot better to use the correct tool.