14 ms·

MemSQL Launches Unlimited Community Edition

198 points by ericfrenkiel 11y ago

phamilton 11y ago
Cue "Call me maybe, MemSQL" Aphyr's posts (taken with appropriate amounts of salt) have become the authority on marketing claims. That said, many solutions are perfectly viable with their shortcomings, but knowing what those shortcomings are is essential.
- nikita 11y ago
  When MemSQL is configured for synchronous durability and all databases are configured with synchronous replication, Jepsen confirms that writes to MemSQL are durable and acknowledged appropriately. As part of testing for the 4.0 release, we used the jepsen network partition test and observed data durability equivalent to Postgres(https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres) e.g. the scalability of a cluster with the durability of a single node machine. As part of running this test, we noticed some opportunities to do some cool performance optimizations. Stay tuned for a blog post to follow!
  - seanp2k2 11y ago
    ^ would love to see the details of this in a blog post, yeah. Please submit that on HN when it's ready; I'm sure I'm not the only one interested :)
    - nikita 11y ago
      will do!
ericfrenkiel 11y ago
Eric, one of the cofounders, here. happy to answer any questions on MemSQL 4 and the community edition. Some new features in MemSQL 4: - fully distributed joins - native geospatial index and datatypes - lots of new SQL surface area - concurrency improvements - analytic optimizer - Spark, HDFS, and S3 connectors
- darkxanthos 11y ago
  What's the catch here? :)
  - deleted 11y ago
    [deleted]
  - ericfrenkiel 11y ago
    http://docs.memsql.com/latest/faq/#what-is-memsql-not-for http://docs.memsql.com/latest/faq/#what-is-memsql-not-for
    - jhugg 11y ago
      It seems like the improvements here are OLAP focused, and welcome ones at that, but the docs and product, if not the marketing, seem to be moving away from operational workloads. From my interpretation of the docs, there are no "transactions" in the Jim Gray / ACID sense of the word. MemSQL offers transactional semantics with READ COMMITTED isolation. This is not just not SERIALIZABLE, it's also not REPEATABLE-READ or SNAPSHOT-READ. For example, imagine a two statement transaction where statement 1 reads a counter value and statement 2 increments it. If two users run this transaction at the same time, the counter could lose an increment. This example is trivial and probably could be done in a single statement, but many other read-then-write operations could cause such an inconsistency. Unless I'm misunderstanding something.
      - jamesblonde 11y ago
        Well if MemSQL supports locks, then you can implement any stronger isolation model using both locks and READ COMMITTED transactions. Do they support row-level locking?
        takeda 11y ago
        Yes, but then you losing the performance benefits.
      - 11y ago
DannoHung 11y ago
Are there any optimizations or explicit support for proximate ordered joins?
- nikita 11y ago
  Could you please elaborate? Do you mean approximate joins as in this talk? http://www2.research.att.com/~divesh/papers/ks2005-aj-tutorial-talk.pdf http://www2.research.att.com/~divesh/papers/ks2005-aj-tutori...
  - DannoHung 11y ago
    No no, sorry, it's much simpler (at least in how it works, no guarantees on implementation complexity, of course). It's an issue that comes up in timeseries databases pretty often. Say I have a table full of quotes and a table full of trades. I want to know what the quote price was at the time the trade occurred. In no-frills SQL, that translates into something like: select * from t left outer join q on q.time=(select max(time) from q where time<=t.time and sym=s) and t.sym = q.sym where date = d and sym = s If you have some sort of support for time proximate joining, the query engine only has to perform a binary search (as long as the indices on the symbol and time columns are appropriate, say symbol partitioned, then time sorted within symbol) to find the correct row from quote to join against. If it doesn't have such support, then a scan is required to find the maximum time value from the quote table that satisfies the constraint on the trade time. Presumably, if you do have support, this wouldn't be the exact query syntax, because that would heavily imply that you want to perform a table scan, or that you could change some aspect of the subquery without affecting performance. Maybe you'd have it be something like this: select * from t left outer join q on before(t.time,q.time) and t.sym = q.sym where date = d and sym = s
    - jchen5 11y ago
      If you just use the no-frills SQL query, we're able to optimize it to do a fast index seek (instead of scan) on q.time, because we know we only have to get the max row. This optimization isn't specific to proximate joins, a simple query like: select max(a) from t where a < 42; will be optimized by memsql.
sql2 11y ago
Goood~
aklarfeld 11y ago
This is way cool. Already spinning it up on my AWS clusters. Such a quick setup. Super stoked to us it.
- rjonesx 11y ago
  It is way cool. I've been an enterprise user for some time now and am excited to get to use it on other projects for which there wasn't cost justification in the past.
stock_toaster 11y ago
> The Community Edition is distributed as an executable > binary and is a free edition of the commercial MemSQL > Enterprise Edition. You are free to download and use > MemSQL Community Edition within your organization. So.. how long until the same thing happens as happened with FoundationDB?
- capkutay 11y ago
  I think the FoundationDB acquisition by a company with no interest in selling enterprise products was an anomaly. A popular, commercial enterprise storage system that actually makes money would be an acquisition target from the likes or Oracle, SAP, EMC, etc...in that scenario, the acquiring company would have significant interest to increase adoption of the product and maintain the developer community versus completely shutting the product down.
  - willvarfar 11y ago
    You mean like Oracle with MySQL? At least in that case the 'community' could move to MariaDB, which is not an option for non-Free databases like MemSQL.
    - ericfrenkiel 11y ago
      you can do the very same - MemSQL uses the MySQL-wire protocol so it works with any MySQL driver and tool.
      - lukaslalinsky 11y ago
        If MySQL was a drop-in replacement for MemSQL, you wouldn't need MemSQL in the first place. The reason you chose MemSQL is probably because it offers something that MySQL doesn't. If I can't take the source and continue using the product, it's a very different situation from MySQL/Oracle.
- rjonesx 11y ago
  While I am an enterprise user of memSQL now, I still have a machine running the beta version 1.0 from years back. It is still stable and still fast as hell.
matt2000 11y ago
Would love to hear from any existing users on their experiences so far (assuming that's allowed under previous licenses). Choosing a database is one of those decisions where I tend to go with the safest, well known option, but maybe I'm missing out.
- rjonesx 11y ago
  I have been an enterprise user and had the luxury of using the 4 beta for the last month or so. I run a cluster of 18 machines with 192 cores and 540GB of RAM. Impressions: memSQL is remarkably stable. I actually have one machine running the old memSQL 1.0 beta that has not rebooted in months. 4.0 has similarly stable. The only problems happen when you run too many other processes on the aggregators (which is really just me being stupid). Speed is great and the wire compliance with mySQL makes it very easy to develop for. To be honest, the "keeping the data in memory" part isn't the best part, it is the query compiling. It is incredibly fast. Often a query that takes 30sec to 1min to execute will compile down to fractions of a second. It is very cool to watch and never gets old. We are looking to literally move all of our internal stuff to memSQL community edition while keeping our customer tools on enterprise.
jansc 11y ago
> While you are free to use Community for your projects, > MemSQL does not support or endorse using it in production. (http://blog.memsql.com/memsql-community-edition/ http://blog.memsql.com/memsql-community-edition/) Ehhh. Do they mean that the Community Edition is only usable for development?
- ericfrenkiel 11y ago
  for critical deployments, the enterprise version has high availability, cross data center replication, and more. you can use the free edition however you please.
- rjonesx 11y ago
  I currently use beta 1.0 for one of my tools and it has been functioning for years without a problem. I think they are just saying don't sue us if you use in production and didn't pay for it :-)
olig15 11y ago
Slightly off-topic: The font weight is too light to read properly on my PC (Widows 8.1, Chrome). Stopped reading because it was too much effort to try and read.
BradRuderman 11y ago
When can we expect windowing functions, specifically row_number, rank, count, lead, lag, by partition.
- aristus 11y ago
  All I can say is "it's on the roadmap." :) I used to do a lot of analytics at Facebook and am now a PM at MemSQL, so it's one of my priorities.
joaojeronimo 11y ago
#FoundationDB
superlogical 11y ago
The CloudFormation cluster generator tool they have is really cool. (http://cloud.memsql.com/cloudformation http://cloud.memsql.com/cloudformation) The templates it generates are pretty complex, I wonder if that is hand written or using some kind tool.. would anyone be able to shed some light on how they did it? Do you think most of it is hand coded? I've been playing around with the .NET SDK in VisualStudio 2013.. it includes a cloud formation project type, and you can type out the JSON with Intellisense which is pretty cool.
- ankrgyl 11y ago
  Hi @superlogical, thank you for using it and for the kind feedback! CloudFormation by itself provides some very basic level of conditional/looping logic, but we found that it was not enough yet to provide a stellar experience. So, when you fill out the form on cloud.memsql.com, we auto-generate a template (we wrote the code to do this) that matches the parameters you filled in, upload it to an S3 bucket on our account, and then expose it as a download or via GET directly to your AWS account (i.e. you own the hardware/database/data).
menegattig 11y ago
Our company (Simbiose) recently did a strong stress test with memSQL with billions of JSON rows, using complex JOIN queries and the results are simply AMAZING.
peterplaylyfe 11y ago
Lol. I got what i asked for in the quora question?

198 points by ericfrenkiel 11y ago

Cue "Call me maybe, MemSQL" Aphyr's posts (taken with appropriate amounts of salt) have become the authority on marketing claims. That said, many solutions are perfectly viable with their shortcomings, but knowing what those shortcomings are is essential.

nikita 11y ago
When MemSQL is configured for synchronous durability and all databases are configured with synchronous replication, Jepsen confirms that writes to MemSQL are durable and acknowledged appropriately. As part of testing for the 4.0 release, we used the jepsen network partition test and observed data durability equivalent to Postgres(https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres) e.g. the scalability of a cluster with the durability of a single node machine. As part of running this test, we noticed some opportunities to do some cool performance optimizations. Stay tuned for a blog post to follow!
- seanp2k2 11y ago
  ^ would love to see the details of this in a blog post, yeah. Please submit that on HN when it's ready; I'm sure I'm not the only one interested :)
  - nikita 11y ago
    will do!

ericfrenkiel 11y ago

Eric, one of the cofounders, here. happy to answer any questions on MemSQL 4 and the community edition. Some new features in MemSQL 4: - fully distributed joins - native geospatial index and datatypes - lots of new SQL surface area - concurrency improvements - analytic optimizer - Spark, HDFS, and S3 connectors

darkxanthos 11y ago
What's the catch here? :)
- deleted 11y ago
  [deleted]
- ericfrenkiel 11y ago
  http://docs.memsql.com/latest/faq/#what-is-memsql-not-for http://docs.memsql.com/latest/faq/#what-is-memsql-not-for
  - jhugg 11y ago
    It seems like the improvements here are OLAP focused, and welcome ones at that, but the docs and product, if not the marketing, seem to be moving away from operational workloads. From my interpretation of the docs, there are no "transactions" in the Jim Gray / ACID sense of the word. MemSQL offers transactional semantics with READ COMMITTED isolation. This is not just not SERIALIZABLE, it's also not REPEATABLE-READ or SNAPSHOT-READ. For example, imagine a two statement transaction where statement 1 reads a counter value and statement 2 increments it. If two users run this transaction at the same time, the counter could lose an increment. This example is trivial and probably could be done in a single statement, but many other read-then-write operations could cause such an inconsistency. Unless I'm misunderstanding something.
    - jamesblonde 11y ago
      Well if MemSQL supports locks, then you can implement any stronger isolation model using both locks and READ COMMITTED transactions. Do they support row-level locking?
      - takeda 11y ago
        Yes, but then you losing the performance benefits.
    - 11y ago

DannoHung 11y ago

Are there any optimizations or explicit support for proximate ordered joins?

nikita 11y ago
Could you please elaborate? Do you mean approximate joins as in this talk? http://www2.research.att.com/~divesh/papers/ks2005-aj-tutorial-talk.pdf http://www2.research.att.com/~divesh/papers/ks2005-aj-tutori...
- DannoHung 11y ago
  No no, sorry, it's much simpler (at least in how it works, no guarantees on implementation complexity, of course). It's an issue that comes up in timeseries databases pretty often. Say I have a table full of quotes and a table full of trades. I want to know what the quote price was at the time the trade occurred. In no-frills SQL, that translates into something like: select * from t left outer join q on q.time=(select max(time) from q where time<=t.time and sym=s) and t.sym = q.sym where date = d and sym = s If you have some sort of support for time proximate joining, the query engine only has to perform a binary search (as long as the indices on the symbol and time columns are appropriate, say symbol partitioned, then time sorted within symbol) to find the correct row from quote to join against. If it doesn't have such support, then a scan is required to find the maximum time value from the quote table that satisfies the constraint on the trade time. Presumably, if you do have support, this wouldn't be the exact query syntax, because that would heavily imply that you want to perform a table scan, or that you could change some aspect of the subquery without affecting performance. Maybe you'd have it be something like this: select * from t left outer join q on before(t.time,q.time) and t.sym = q.sym where date = d and sym = s
  - jchen5 11y ago
    If you just use the no-frills SQL query, we're able to optimize it to do a fast index seek (instead of scan) on q.time, because we know we only have to get the max row. This optimization isn't specific to proximate joins, a simple query like: select max(a) from t where a < 42; will be optimized by memsql.

sql2 11y ago

Goood~

aklarfeld 11y ago

This is way cool. Already spinning it up on my AWS clusters. Such a quick setup. Super stoked to us it.

rjonesx 11y ago
It is way cool. I've been an enterprise user for some time now and am excited to get to use it on other projects for which there wasn't cost justification in the past.

stock_toaster 11y ago

> The Community Edition is distributed as an executable > binary and is a free edition of the commercial MemSQL > Enterprise Edition. You are free to download and use > MemSQL Community Edition within your organization. So.. how long until the same thing happens as happened with FoundationDB?

capkutay 11y ago
I think the FoundationDB acquisition by a company with no interest in selling enterprise products was an anomaly. A popular, commercial enterprise storage system that actually makes money would be an acquisition target from the likes or Oracle, SAP, EMC, etc...in that scenario, the acquiring company would have significant interest to increase adoption of the product and maintain the developer community versus completely shutting the product down.
- willvarfar 11y ago
  You mean like Oracle with MySQL? At least in that case the 'community' could move to MariaDB, which is not an option for non-Free databases like MemSQL.
  - ericfrenkiel 11y ago
    you can do the very same - MemSQL uses the MySQL-wire protocol so it works with any MySQL driver and tool.
    - lukaslalinsky 11y ago
      If MySQL was a drop-in replacement for MemSQL, you wouldn't need MemSQL in the first place. The reason you chose MemSQL is probably because it offers something that MySQL doesn't. If I can't take the source and continue using the product, it's a very different situation from MySQL/Oracle.
rjonesx 11y ago
While I am an enterprise user of memSQL now, I still have a machine running the beta version 1.0 from years back. It is still stable and still fast as hell.

matt2000 11y ago

Would love to hear from any existing users on their experiences so far (assuming that's allowed under previous licenses). Choosing a database is one of those decisions where I tend to go with the safest, well known option, but maybe I'm missing out.

rjonesx 11y ago
I have been an enterprise user and had the luxury of using the 4 beta for the last month or so. I run a cluster of 18 machines with 192 cores and 540GB of RAM. Impressions: memSQL is remarkably stable. I actually have one machine running the old memSQL 1.0 beta that has not rebooted in months. 4.0 has similarly stable. The only problems happen when you run too many other processes on the aggregators (which is really just me being stupid). Speed is great and the wire compliance with mySQL makes it very easy to develop for. To be honest, the "keeping the data in memory" part isn't the best part, it is the query compiling. It is incredibly fast. Often a query that takes 30sec to 1min to execute will compile down to fractions of a second. It is very cool to watch and never gets old. We are looking to literally move all of our internal stuff to memSQL community edition while keeping our customer tools on enterprise.

jansc 11y ago

> While you are free to use Community for your projects, > MemSQL does not support or endorse using it in production. (http://blog.memsql.com/memsql-community-edition/ http://blog.memsql.com/memsql-community-edition/) Ehhh. Do they mean that the Community Edition is only usable for development?

ericfrenkiel 11y ago
for critical deployments, the enterprise version has high availability, cross data center replication, and more. you can use the free edition however you please.
rjonesx 11y ago
I currently use beta 1.0 for one of my tools and it has been functioning for years without a problem. I think they are just saying don't sue us if you use in production and didn't pay for it :-)

olig15 11y ago

Slightly off-topic: The font weight is too light to read properly on my PC (Widows 8.1, Chrome). Stopped reading because it was too much effort to try and read.

BradRuderman 11y ago

When can we expect windowing functions, specifically row_number, rank, count, lead, lag, by partition.

aristus 11y ago
All I can say is "it's on the roadmap." :) I used to do a lot of analytics at Facebook and am now a PM at MemSQL, so it's one of my priorities.

joaojeronimo 11y ago

#FoundationDB

superlogical 11y ago

The CloudFormation cluster generator tool they have is really cool. (http://cloud.memsql.com/cloudformation http://cloud.memsql.com/cloudformation) The templates it generates are pretty complex, I wonder if that is hand written or using some kind tool.. would anyone be able to shed some light on how they did it? Do you think most of it is hand coded? I've been playing around with the .NET SDK in VisualStudio 2013.. it includes a cloud formation project type, and you can type out the JSON with Intellisense which is pretty cool.

ankrgyl 11y ago
Hi @superlogical, thank you for using it and for the kind feedback! CloudFormation by itself provides some very basic level of conditional/looping logic, but we found that it was not enough yet to provide a stellar experience. So, when you fill out the form on cloud.memsql.com, we auto-generate a template (we wrote the code to do this) that matches the parameters you filled in, upload it to an S3 bucket on our account, and then expose it as a download or via GET directly to your AWS account (i.e. you own the hardware/database/data).

menegattig 11y ago

Our company (Simbiose) recently did a strong stress test with memSQL with billions of JSON rows, using complex JOIN queries and the results are simply AMAZING.

peterplaylyfe 11y ago

Lol. I got what i asked for in the quora question?