13 ms·
MemSQL is now free to use for databases with up to 128GB of RAM usage
- setheron 8y agois memSQL a replacement for a full RDBMS in your stack ? Or just for analytics -- siphoning off the data from your master.
- ryanworl 8y agoMemSQL is an "HTAP" (hybrid transactional-analytical processing) database. They have an in-memory row store table engine for the transactional queries and a disk-backed column store table engine for the analytical queries.
- deleted 8y ago[deleted]
- nikita 8y agoYes we have customers that standardize their database workloads on memsql and move off of oracle
- CuriouslyC 8y agoPersonally, I would not use memsql as a first line RDBMS. It lacks too many useful features, and it doesn't return consistent results for all queries (I've found inconsistencies when using group by with having, for instance).
- ruw1090 8y agoIf you have an example of the query you thought was returning inconsistent results, feel free to make a post over at https://www.memsql.com/forum/ https://www.memsql.com/forum/ with a repro.
- gacba 8y agoBeen using MemSQL for a enterprise-level financial services client since mid-2017. We have this in production and are running it in a multi-TB cluster. We've not seen ANY of these issues here and are heavy, crazy query users. Their support has been nothing but on-the-spot and very helpful.
- CuriouslyC 8y agoI totally admit that the inconsistency might be a result of misconfiguration by maintainers at my company. That being said, being able to shoot yourself in the foot so subtly and badly via configuration seems like a pretty strong anti-feature.
- manigandham 8y agoSince you aren't sharing any actual details and are switching between blaming the product to blaming your devs and config (where no product can magically keep you from breaking settings), your comments come across as rather disingenuous. Why not share a clear example of exactly what happened, or post on their forum with details, so we can all judge for ourselves?
- manigandham 8y agoIt can be, but if you're using the rowstore tables (best for single row lookups and constant modifications) then you'll have to keep all that data in RAM which can be limiting.
- supperdave 8y agoSweet!
- truth_seeker 8y agoDoes the free version supports Horizontal sharding and async/sync DISK writes for data mirroring from RAM ?
- no1youknowz 8y agoFrom the announcement: > You can do almost anything with MemSQL, using the free tier, that you can do if you have an Enterprise license, including capabilities and production use. The differences are that you can only configure the free tier of MemSQL to use up to 128GB of RAM usage, and support is only community support; for paid MemSQL support, you need an Enterprise license.
- lbriner 8y agoThe bit I can't see on the FAQ is about how it is as fast as RAM but protects against data loss using disk. I think most engines guarantee Durability by assuming that once on disk, it won't go anywhere but if it's in RAM, it is susceptible to power outage? If it gets written to disk, it's not as fast as RAM?
- ryanworl 8y agoThe concept in databases that you’re looking for is called group commit. Transactions running concurrently batch writes together before flushing to disk. This means the minimum latency for committing a transaction is the speed of fsync, but the throughput can be as high as the disk bandwidth. There are other reasons why an in memory database can run faster than a disk based database, such as not having a buffer pool manager, but I don’t think that’s what you were worried about.
- janetacarr 8y agoIt might use a Write-Ahead log like postgres to guarantee data integrity between memory and disk.
- nikita 8y agoThat’s right and it writes it on two separate nodes if u enable high availability
- rickn 8y ago[Director of Product Management for MemSQL] MemSQL has two storage modes Rowstore and Columnstore. The Rowstore is "in-memory" and the columnstore is "on-disk" but those are oversimplifications. The rowstore data is stored in memory but we keep a snapshot of the data on disk. We also keep the transaction log (a record of all changes since the snapshot was taken) also on disk. So queries can be satisfied fully from memory (because that is where the current data lives) but writes go to memory and to the transaction log on disk. If the machine reboots then the snapshot is loaded from disk back into memory and the transaction log is replayed. When that is complete you are back to where you were when the machine rebooted with no loss of committed data. Columnstore data is always stored on disk although we use a row store in front of the column store that is hidden from the user but acts as a buffer of sorts so that writes in the column store can be pretty fast. More details on how the columnstore works can be found here: https://docs.memsql.com/concepts/v6.7/columnstore/#how-the-memsql-columnstore-works https://docs.memsql.com/concepts/v6.7/columnstore/#how-the-m...
- quangio 8y agoWhat is the benefit of using MemSQL over some other free in memory databases like Apache Ignite ? I see that they have better documentation and support (edit: + competition on Codeforces which winners rarely receive their T-Shirts). What about other things?
- nikita 8y agoMemSQL CEO here. There are a few: - MemSQL is transactional and writes transactions on disk - MemSQL has an excellent implementation of SQL with mature query optimization and query execution. And it get better every release. This is from 6.5 https://www.memsql.com/blog/6.5-performance/ https://www.memsql.com/blog/6.5-performance/ - MemSQL has in-memory and on-disk data storage so you can use MemSQL to store petabytes - MemSQL has columnstores and vectorized query processing: https://news.ycombinator.com/item?id=16617098 https://news.ycombinator.com/item?id=16617098 - MemSQL supports geospatial, fulltext search, and json - MemSQL allows you to stream data from kafka in one command: https://docs.memsql.com/sql-reference/v6.5/create-pipeline/ https://docs.memsql.com/sql-reference/v6.5/create-pipeline/
- noir_lord 8y agoGiven the availability of open source solutions why would I in 2018 build a critical part of my application on a closed platform? Genuinely not intended as snark, I'm just curious why memsql is so compelling that I would consider it.
- siscia 8y agoSort of shameless plug, but if you are looking into such solution you could have a look also at https://redisql.com/ https://redisql.com/ It is a redis module that embed SQLite, bringing on the table a lot of advantages. Extremely fast and with it you can even upgrade your Redis instance to be your only database. There is not a huge company behind it, but I really do my best to support it. I don't believe to have disappointed any of our users so far. Also the tech documentation: http://redbeardlab.tech/rediSQL/references/ http://redbeardlab.tech/rediSQL/references/
- edoceo 8y agoThis is awesome! Exactly what I was looking for. Basically a SQLite server-node. Replace my homebrew Go thing.
- siscia 8y agoI am very glad you find it interesting! For any issues don't hesitate to contact me directly or through GitHub. Also if you need it for some open source/(do the world a better place) kind of project I am more than happy to provide the PRO version free of any charges or obligations. Of course it apply to anybody reading!
- timmorgan 8y agoCool project! I will check this out. Note: you have a typo on your home page: "never loose a bit" 'loose' => 'lose'
- 8y ago
- shin_lao 8y agoI don't understand the high frequency use case they describe. High frequency trading is something very different from "12,000 transactions a minute". What is exactly the use case? Pre-deal checks? Post-deal checks? Book replay? Or is it just a simulation? It's not very clear.
- PeterisP 8y agoHFT can easily be 12000 transactions a minute with a requirement for each of these transactions to be very fast. It's not (necessarily) about high throughput, but about low latency.
- hendzen 8y agoNo real HFT system is doing a * database transaction * in the critical trading path. HFT systems are not built like web applications. They are typically built as a tight event loop, reading market data packets directly from the network card, doing a tiny bit of computation and then writing to a userspace TCP stack for order entry. I guess you could be using MemSQL for post-order or trade analysis but then it would probably overkill since a lot of that can be done considerably slower.
- shin_lao 8y agoAdditionally 128 GB is just about 20 minutes worth of data.
- jandrese 8y agoHFT firms have plenty of cash to buy the unlimited version.
- hendzen 8y agoHFT ain’t what it used to be.
- manigandham 8y ago
- no1youknowz 8y agoI've been using MemSQL in development since March 2017. I've seen it evolve into one of the fastest databases for columnar storage and analytical workloads. Having in-memory rowstore as well removed the need to have Aerospike in my infrastructure and simplified the whole stack. This announcement, that it's now free to 128GB pretty much saved me from having to do a kickstarter to raise funds for my little SaaS project. If anyone from MemSQL is reading, really thanks for doing this. I think it's a very shrewd move. Should definitely see an upswing of potential customers adding it to their stack and when they grow, become enterprise customers.
- starik36 8y agoIt says that he 128GB version comes with High Availability features. Does that mean if I have 2 nodes, each one is limited to 64GB? Or each one can have 128?
- hanson5b 8y agoA typical setup for HA would be 2 aggregator nodes and 2 leaf nodes. You can allocate the memory however you want. For example, you could give 32GB to all 4. Or give 16GB to each aggregator and 48GB to each leaf. (I'm a MemSQL Product Manager by the way.)
- zedpm 8y agoDo you have recommendations on provisioning memory in aggregators vs leaves? That is, would one of those configurations make for sense than the other for typical workloads?
- brd529 8y agoIn general more memory should go to the leaves. All data is stored there and MemSQL will push processing to the leaves whenever possible.
- rickn 8y agoI am the Director of Product Management for MemSQL. The 128 GB limit applies to the whole cluster. So if you have two nodes in the cluster they would each have to be 64GB or less. If you have four nodes they would all have to be 32 GB or less. To have a highly available system we recommend 4 nodes (a master aggregator, a child aggregator and two leaf nodes). You can read more about the cluster architecture here: https://docs.memsql.com/concepts/v6.7/distributed-architecture/ https://docs.memsql.com/concepts/v6.7/distributed-architectu...
- dominotw 8y agohttps://www.youtube.com/watch?v=_vloWsdPCDs https://www.youtube.com/watch?v=_vloWsdPCDs CMU had guest lecture by memsql founder about the architecture. Not sure how much of it changed in past 2 yrs tho.
- polskibus 8y agoWhat happens if I set hard limit on MemSQL process to use 128GB RAM and it decides it needs more? Will it die demanding more memory or will it behave like traditional rdbms and manage buffers to work within limits?
- ruw1090 8y agoThe best thing to do in this situation would be to set the maximum memory on the MemSQL server via the `maximum_memory` system variable. The server will then internally manage it's memory usage against this upper bound and fail only specific operations which can't be performed without additional memory.
- polskibus 8y agoWhat are the specific operations? Will another batch of INSERTs / UPDATEs work? Does it matter if I use the analytical part of MemSQL ?
- ruw1090 8y agoI mean that an operation that needs more memory than is available will fail, but the server itself will remain operational. An simple example would be loading more data into an in-memory table (row store) than there is available memory. If you're using the columnstore then your storage won't be limited to memory.
- polskibus 8y agoThat sucks, it's similar to Hekaton though. I was hoping there was some kind of a fallback mode when the DB has to juggle memory buffers at a performance cost.
- qeternity 8y agoSeems with this they've also deprecated the developer version which was unlimited capacity but lacked enterprise features. Also, no mention of pricing except to "contact sales".
- no1youknowz 8y agoI asked back in 2017 and it was $25k. This post 7 months ago [0] mentions the same. It's probably still at this price, but if you are really interested. Yeah, contact sales. [0]: https://news.ycombinator.com/item?id=16617827 https://news.ycombinator.com/item?id=16617827
- elvinyung 8y agoHow is MemSQL for OLTP nowadays? Has anyone had any success consistently/scalably using it for true HTAP-ish use cases?
- manigandham 8y agoIt's always been good at OLTP and great at OLAP. We used it for years as a single HTAP solution during the 5.x versions for heavy adtech applications.
- elvinyung 8y agoThanks! Just curious, do you use the same tables for both OLTP and OLAP, or do you have some kind of pseudo-ETL process to transform between OLTP and OLAP schemas?
- manigandham 8y agoThe schemas are the same. Tables behave the same way logically, and its just performance and physical semantics that are different. We would have recent data in rowstore and move older data into columnstore. You can easily join/union between both for queries. Also some constantly changing data (like budget counters) would always remain in rowstore with many lookups and updates per-second.
- moolcool 8y agoThis might be a stupid question, but what is the advantage of this solution over an in-memory sqlite database?
- haneefmubarak 8y agoFrom what they claim, MemSQL supposedly scales well across a cluster of servers (and of course the usual, fail over, features, and other fun stuff).
- ericd 8y agoI’m no db expert, but doesn’t SQLite have very non-granular locking that makes it hard to scale?
- manigandham 8y agoDistributed. Scalable. OLTP + OLAP queries. High availability. Very fast performance for reads and writes. They are entirely different systems. Sqlite is meant for self-contained applications that need some relational data persistence with a single file for storage, not for accessing as a central database with many clients storing TBs and scaling across nodes.
- marmaduke 8y agoSQLite is a replacement for fopen (Quote from their docs)
- riku_iki 8y ago> Distributed is MemSQL shared nothing or shared everything, or can be mix of both?
- manigandham 8y agoShared nothing. It has leaf nodes that store data and do local processing, and aggregator nodes that run queries spread to the leaf nodes and return the results.
- 8y ago
- cft 8y agoWhy cannot I use free open source MySQL heap or memory storage engine? It also provides clustering/replication
- deleted 8y ago[deleted]
- tintor 8y agoYou can. :)
- raitom 8y agoIs there a way on this release to create auto-scale rules like on RDS?
- deleted 8y ago[deleted]
- furicane 8y agoI tried to come up with a smart and polite comment but I can't. The target audience for memsql aren't developers or engineers. It's the management that has no idea about IT. I don't like closed-source solutions. I don't want to book a demo. I want to be able to read the source. I want to install it, use it, benchmark it, be sure that the results are 100% accurate. Sadly, I see this post as marketing ploy and I can't find any nice words for this product.
- manigandham 8y agoYou can install it, use it, benchmark it, and check everything yourself. It's not open-source, but there is plenty of closed-source proprietary software, and plenty of buyers who care about solving their problems and paying money to get that done (and ensure the vendor stays alive). If you don't want to use a closed-source product then that's your prerogative, but I don't see how you're making a dev/engineering decision by ignoring a product because of that.
- moocowtruck 8y agobecause the source is important, it tells the buyer they have a future if they no longer like your services; closed source means they are locked into whatever fresh hell might come upon your company which in turn unleashes fresh hell upon their decision to buy in.. in 2018+ it's just a smart decision to make
- shin_lao 8y agoGood luck with the source code of a database you didn't write...
- jessedhillon 8y agoI recently was able to unfuck my way out of a 300GB data loss resulting from a failed DB upgrade. By looking into the commit history of PostgreSQL, finding the commit with the PG_CATALOG_VERSION I needed, and compiling from that revision, I was able to re-run the upgrade with the parameters I needed. I'm not sure what I would have done if that had been MS SQL Server or something else.
- emmanueloga_ 8y agoAnother instance of the Wikipedia page for a product [1] being more useful than the main site to describe it: * MemSQL is a distributed, in-memory, SQL database management system. * It is a relational database management system (RDBMS). * It compiles Structured Query Language (SQL) into machine code, via termed code generation. * On April 23, 2013, MemSQL launched its first generally available version of the database to the public. * MemSQL is wire-compatible with MySQL. * MemSQL can store database tables either as rowstores or columnstores (The OLAP vs OLTP part I guess). * A MemSQL database is a distributed database implemented with aggregators and leaf nodes. * MemSQL durability is slightly different for its in-memory rowstore and an on-disk columnstore. * A MemSQL cluster can be configured in "High Availability" mode. * MemSQL gives users the ability to install Apache Spark as part of the MemSQL cluster, and use Spark as an ETL tool. The main value proposition seems to be the distributed nature, which probably makes it easier to setup out of the box than, say, trying to setup a cluster MySQL or PostgreSQL databases which are not "natively distributed". Also, probably most useful when the data is "big enough" vs resources available on any single server or when reliability is very important. 1: https://en.wikipedia.org/wiki/MemSQL https://en.wikipedia.org/wiki/MemSQL
- pritambarhate 8y agoOne thing I would like to see is the commitment to keep it free till X years at least. OK, MemSQL has "decided" to give it for free up to 128GB of RAM usage now. But they can "decide" 2 years later that they want to charge for any MemSQL usage. Then what options the "small" businesses, who have adopted MemSQL, have? When it comes to commercial products, it is generally good to check if you can afford their paid offering, then only make the tool core part of your infrastructure. For databases, it's better to stick with fully open source popular options from a long-term perspective.