4 ms·

> I, personally, have seen many projects burning thousands a month in database costs because they prefer replicating the environment for testing branches. If t

> I, personally, have seen many projects burning thousands a month in database costs because they prefer replicating the environment for testing branches. If this is the premise for serverless database it's a weak start. If you really need lightweight DBMS for testing just run MySQL or PostgreSQL in Docker. If you really need access to production-like data (e.g., a lot of it so you get realistic distributions) run the same DBMS on cheap hardware or cheap instances. In both cases you can use persistent volumes and shut down when things are not in use. Few people really care if it takes a few minutes to spin up the test environment. As for the main point of Aurora offering a "Serverless" architecture it looks as if what they've really done is enabled the DBMS compute layer to scale up and down quickly. I wonder if this optimization fell out of pushing redo log management down into the storage layer. (See Section 3.1 https://www.allthingsdistributed.com/files/p1041-verbitski.pdf https://www.allthingsdistributed.com/files/p1041-verbitski.p... for details.)

hodgesrm 9y ago
Just read the news release and they are claiming exactly that. Good on them--the Aurora/Redshift team has done some really outstanding work.
marknadal 9y ago
Exactly! It is also ridiculous that this "serverless" phrase is being used to push this agenda. "Serverless" DBs are also very pricey if you consider everything. Instead, it isn't actually that hard to do some work loads at huge scale for cheap, two good articles on this (one discord, one ours) - https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7 https://blog.discordapp.com/how-discord-stores-billions-of-m... - https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 (100M records/day for $10 total cost, server, disk, S3 backup!)
- hodgesrm 9y ago
  Thanks for the references! Just read the blog article. It was kind of obvious from first paragraph it would be a Cassandra deployment story. ;) Gun is new to me. You have a good way of handling distributed consistency. It's intriguing that the system still can reach consistency even if every node loses network connectivity temporarily. Is there any academic work behind this?
  - marknadal 9y ago
    Thank you! Yes, there is currently just a whitepaper we're publishing with Stanford and that was also reviewed by colleagues at MIT. However it isn't up to snuff yet to be posted publicly, so shoot me an email mark@gunDB.io and I'll send you the link. For non-academics though, I did a comic strip explainer for the layperson (as distributed systems are often hyped up with elitist jargon) here: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html ! The prototype shown in the video was specific to append-only data and was done last year, we recently rewrote the system to be more generalizable using a radix trie structure and have released an alpha of it with the Radix Storage Engine (RSE) in the main repo: https://github.com/amark/gun https://github.com/amark/gun Let me know if you need help with any/all of that, or more links/resources (docs are somewhat scarce on it currently, sadly), and I'll do what I can! Although, I can't resist but to leave this one last animated explainer gif that shows what a radix looks like: http://gun.js.org/see/radix.gif http://gun.js.org/see/radix.gif
Clubber 9y ago
I think his main point was cost savings in environments that don't need constant uptime, like Dev and QA DB's in a 9-5ish shop. You can also throw a light DB (mySql, SqlExpress, Postgres) on a VM that you already have running 24/7, that would make the additional cost zero. Of course you are still going to need to curate your dev/qa data (through replication or periodic backup/restore) because shit data is hard to code against, not to mention debug against.
tytytytytytytyt 9y ago
> If this is the premise for serverless database it's a weak start. He also seems to imply that you have to keep your test db's running all day instead of simply paying for them by the hour. :|

hodgesrm 9y ago

Just read the news release and they are claiming exactly that. Good on them--the Aurora/Redshift team has done some really outstanding work.

marknadal 9y ago

Exactly! It is also ridiculous that this "serverless" phrase is being used to push this agenda. "Serverless" DBs are also very pricey if you consider everything. Instead, it isn't actually that hard to do some work loads at huge scale for cheap, two good articles on this (one discord, one ours) - https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7 https://blog.discordapp.com/how-discord-stores-billions-of-m... - https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 (100M records/day for $10 total cost, server, disk, S3 backup!)

hodgesrm 9y ago
Thanks for the references! Just read the blog article. It was kind of obvious from first paragraph it would be a Cassandra deployment story. ;) Gun is new to me. You have a good way of handling distributed consistency. It's intriguing that the system still can reach consistency even if every node loses network connectivity temporarily. Is there any academic work behind this?
- marknadal 9y ago
  Thank you! Yes, there is currently just a whitepaper we're publishing with Stanford and that was also reviewed by colleagues at MIT. However it isn't up to snuff yet to be posted publicly, so shoot me an email mark@gunDB.io and I'll send you the link. For non-academics though, I did a comic strip explainer for the layperson (as distributed systems are often hyped up with elitist jargon) here: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html ! The prototype shown in the video was specific to append-only data and was done last year, we recently rewrote the system to be more generalizable using a radix trie structure and have released an alpha of it with the Radix Storage Engine (RSE) in the main repo: https://github.com/amark/gun https://github.com/amark/gun Let me know if you need help with any/all of that, or more links/resources (docs are somewhat scarce on it currently, sadly), and I'll do what I can! Although, I can't resist but to leave this one last animated explainer gif that shows what a radix looks like: http://gun.js.org/see/radix.gif http://gun.js.org/see/radix.gif

Clubber 9y ago

I think his main point was cost savings in environments that don't need constant uptime, like Dev and QA DB's in a 9-5ish shop. You can also throw a light DB (mySql, SqlExpress, Postgres) on a VM that you already have running 24/7, that would make the additional cost zero. Of course you are still going to need to curate your dev/qa data (through replication or periodic backup/restore) because shit data is hard to code against, not to mention debug against.

tytytytytytytyt 9y ago

> If this is the premise for serverless database it's a weak start. He also seems to imply that you have to keep your test db's running all day instead of simply paying for them by the hour. :|