4 ms·

"My Laptop Is Faster Than Your Cloud" is similar to the things I often say. I've long dismayed at the fact that cloud instances of databases(RDS), build servers

"My Laptop Is Faster Than Your Cloud" is similar to the things I often say. I've long dismayed at the fact that cloud instances of databases(RDS), build servers, and various other things I need have often been less powerful than the laptop I'm using and at the same time so costly that a few months of rent on them would pay for my laptop. On the other hand, Postgres can be shockingly fast. With a little care I've been able to make it beat the stuffing out of most other DBs and various big data tools like Spark(running locally, not massively scaled out of course). The article starts out saying it's 80x faster than Postgres. That level of difference from Postgres doesn't impress me, it makes me stop reading and become deeply suspicious that the author either doesn't know anything about how to make PG run well or is cherry picking one of the very few tasks PG doesn't do well or has had a regression in.

ttunguz 4y ago
Author here: I agree with you on benchmarks. It's hard to compare different databases well. MotherDuck is an analytics database optimized for reading with columnar compression. Postgres is more of a transactional/general purpose database. Tuning it well for analytics would surely improve performance. Most people won't tune a db with custom indices though because it can be hard, so purpose built solutions like this offer value in those cases. The article isn't purely about performance but also ease of use. MD is an in process database so starting with it is very easy. Glad we agree on the power within a laptop! It's underappreciated.
- m_ke 4y ago
  *motherduck is not a database, it's a company commercializing a great open source project called duckdb.
  - carterschonwald 4y ago
    this is an important detail I couldn't figure out from the linked materials!
  - ttunguz 4y ago
    Good addition. I should have been more clear.
- danielmarkbruce 4y ago
  How this is easier than using snowflake/bigquery etc? The situation where it seems useful is if I'm using a jupyter notebook and I want to use it to pull data in but it's too much to fit into memory, and sqlite is too slow... which seems like a pretty specific situation to be in? And it's not data that is part of some business process that needs updating frequently? Or I'm just missing the point entirely? I see another post about huge parquet datasets - but how do I get them? Is everyone at my company comfortable with me moving them onto my machine even if I can? Is the dataset so big it won't fit into memory? Can't I just connect my jupyter notebook to snowflake anyway? Then I'm only pulling down the data I want instead of all the data. I guess maybe the real question is - exactly who is the buyer of this product, how much will they spend on it, and who is the user?
KiranRao0 4y ago
While this is true that a local computer is often faster than a single cloud instance, I've often noticed code running on cloud instances (dev servers, etc) end up finishing their task more quickly. It's usually because most tasks I do end up being limited by the network connection to other resources (Postgres DBs, etc). The dev server being in the same datacenter as the resource is a huge advantage in terms of both latency and bandwith. Again, this is mostly observations from my own workloads, and other people's use cases may be different.
- ttunguz 4y ago
  Yes fair point. I see that pattern when running background tasks locally vs on heroku's workers. In this case, most of the analysis is happening locally without a ton of network use.
- wkdneidbwf 4y ago
  exactly. people complaining that an ec2 or rds instance isn’t as fast as their laptop are missing the point. go ahead and host your production apps on your laptop then. enjoy yourself!
- wink 4y ago
  You're assuming a certain competency in running the cloud setup. I once had a customer where I've actually used a ~3y old middle of the line laptop (not even a dev one) to prove that our software is not at fault for performance problems because it was running all nodes of the production setup/load better than their cloud setup.
m_ke 4y ago
duckdb is really competing with things like snowflake, where compute is separate from storage. Loading data into postgres and tuning it can be a pain in the ass, with duckdb I can query huge parquet datasets in cloud storage without spinning up a VM. It already replaced pandas and dask for most of my common data eng tasks.
- ttunguz 4y ago
  Yes, that's right.
- mr_toad 4y ago
  > I can query huge parquet datasets in cloud storage without spinning up a VM. You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?
  - danielmarkbruce 4y ago
    Yeah, I thought the exact same thing.
- danielmarkbruce 4y ago
  >> it already replaced pandas and dask for most of my common data eng tasks This is interesting - why did you replace pandas/dask with this? What are you getting out of it?
  - m_ke 4y ago
    It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc. I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
    - danielmarkbruce 4y ago
      Gotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.

ttunguz 4y ago

Author here: I agree with you on benchmarks. It's hard to compare different databases well. MotherDuck is an analytics database optimized for reading with columnar compression. Postgres is more of a transactional/general purpose database. Tuning it well for analytics would surely improve performance. Most people won't tune a db with custom indices though because it can be hard, so purpose built solutions like this offer value in those cases. The article isn't purely about performance but also ease of use. MD is an in process database so starting with it is very easy. Glad we agree on the power within a laptop! It's underappreciated.

m_ke 4y ago
*motherduck is not a database, it's a company commercializing a great open source project called duckdb.
- carterschonwald 4y ago
  this is an important detail I couldn't figure out from the linked materials!
- ttunguz 4y ago
  Good addition. I should have been more clear.
danielmarkbruce 4y ago
How this is easier than using snowflake/bigquery etc? The situation where it seems useful is if I'm using a jupyter notebook and I want to use it to pull data in but it's too much to fit into memory, and sqlite is too slow... which seems like a pretty specific situation to be in? And it's not data that is part of some business process that needs updating frequently? Or I'm just missing the point entirely? I see another post about huge parquet datasets - but how do I get them? Is everyone at my company comfortable with me moving them onto my machine even if I can? Is the dataset so big it won't fit into memory? Can't I just connect my jupyter notebook to snowflake anyway? Then I'm only pulling down the data I want instead of all the data. I guess maybe the real question is - exactly who is the buyer of this product, how much will they spend on it, and who is the user?

KiranRao0 4y ago

While this is true that a local computer is often faster than a single cloud instance, I've often noticed code running on cloud instances (dev servers, etc) end up finishing their task more quickly. It's usually because most tasks I do end up being limited by the network connection to other resources (Postgres DBs, etc). The dev server being in the same datacenter as the resource is a huge advantage in terms of both latency and bandwith. Again, this is mostly observations from my own workloads, and other people's use cases may be different.

ttunguz 4y ago
Yes fair point. I see that pattern when running background tasks locally vs on heroku's workers. In this case, most of the analysis is happening locally without a ton of network use.
wkdneidbwf 4y ago
exactly. people complaining that an ec2 or rds instance isn’t as fast as their laptop are missing the point. go ahead and host your production apps on your laptop then. enjoy yourself!
wink 4y ago
You're assuming a certain competency in running the cloud setup. I once had a customer where I've actually used a ~3y old middle of the line laptop (not even a dev one) to prove that our software is not at fault for performance problems because it was running all nodes of the production setup/load better than their cloud setup.

m_ke 4y ago

duckdb is really competing with things like snowflake, where compute is separate from storage. Loading data into postgres and tuning it can be a pain in the ass, with duckdb I can query huge parquet datasets in cloud storage without spinning up a VM. It already replaced pandas and dask for most of my common data eng tasks.

ttunguz 4y ago
Yes, that's right.
mr_toad 4y ago
> I can query huge parquet datasets in cloud storage without spinning up a VM. You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?
- danielmarkbruce 4y ago
  Yeah, I thought the exact same thing.
danielmarkbruce 4y ago
>> it already replaced pandas and dask for most of my common data eng tasks This is interesting - why did you replace pandas/dask with this? What are you getting out of it?
- m_ke 4y ago
  It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc. I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
  - danielmarkbruce 4y ago
    Gotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.