3 ms·

where did you get the 99% metric ? Most of the companies even with a single saas product have insane amount of data these days. Not just application data, Ther

where did you get the 99% metric ? Most of the companies even with a single saas product have insane amount of data these days. Not just application data, There is also a whole lot of analytical data collected at every step of the product usage cycle.

jpgvm 5y ago
Generally you don't have your main application database also serve as your data warehouse for lots of reasons. Generally you offload that to purpose built systems that favor those aforementioned tradeoffs or onto services that run them for you, i.e BigQuery, Snowflake, etc. Your main application database is unlikely to need sharding unless you really do have a phenomenal amount of customers or you need regional sharding to meet legal requirements about data sovereignty for example.
KingOfCoders 5y ago
From managing databases over the last two decades (starting with Ingres/mSQL), hardware has growing much faster (RAM/CPU/NVMe) than the data needs. I remember the first time we put 2Tb RAM into a machine to handle most analytics in memory. From my experience TimescaleDB is fast and takes a lot of data in. If you're multi tenant, it's usually easy to shard. And we do dataware housing on BigQuery, no need to have your own machine and manage the database. Of course there are people who need unlimited horizontal scaling.
- LoriP 5y ago
  TimescaleDB also offers a good lot of supplementary functions to the PostgeSQL core product to help with time series data analysis... saves a lot of SQL acrobatics!
traceroute66 5y ago
> where did you get the 99% metric ? Most of the companies even with a single saas product have insane amount of data these days. I will just leave this link here .... https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...
- cutemonster 5y ago
  And this one -- StackOverflow, one single live db: https://stackexchange.com/performance https://stackexchange.com/performance
mosselman 5y ago
You are right, the metric should probably be 99.999999999999999%. 99% is way to conservative.
geon 5y ago
If you do sas, you are already the 1 %.

jpgvm 5y ago

Generally you don't have your main application database also serve as your data warehouse for lots of reasons. Generally you offload that to purpose built systems that favor those aforementioned tradeoffs or onto services that run them for you, i.e BigQuery, Snowflake, etc. Your main application database is unlikely to need sharding unless you really do have a phenomenal amount of customers or you need regional sharding to meet legal requirements about data sovereignty for example.

KingOfCoders 5y ago

From managing databases over the last two decades (starting with Ingres/mSQL), hardware has growing much faster (RAM/CPU/NVMe) than the data needs. I remember the first time we put 2Tb RAM into a machine to handle most analytics in memory. From my experience TimescaleDB is fast and takes a lot of data in. If you're multi tenant, it's usually easy to shard. And we do dataware housing on BigQuery, no need to have your own machine and manage the database. Of course there are people who need unlimited horizontal scaling.

LoriP 5y ago
TimescaleDB also offers a good lot of supplementary functions to the PostgeSQL core product to help with time series data analysis... saves a lot of SQL acrobatics!

traceroute66 5y ago

> where did you get the 99% metric ? Most of the companies even with a single saas product have insane amount of data these days. I will just leave this link here .... https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...

cutemonster 5y ago
And this one -- StackOverflow, one single live db: https://stackexchange.com/performance https://stackexchange.com/performance

mosselman 5y ago

You are right, the metric should probably be 99.999999999999999%. 99% is way to conservative.

geon 5y ago

If you do sas, you are already the 1 %.