3 ms·

Yup, experiencing that at work quite a bit. People are currently arguing about big data solutions for a data set of 300Gb or so, growth in the area of 100ish Gb

by tetha 8y ago

Yup, experiencing that at work quite a bit. People are currently arguing about big data solutions for a data set of 300Gb or so, growth in the area of 100ish Gb / year. I'm still arguing that this could be an in-memory dataset for postgres, and we wouldn't need that with a good schema. Or other people are wondering if we could replace all of our relational databases with kafka. While complaining about inconsistent data sets. Well let's talk about advantages of a good relational schema first. Maybe I'm turning into a grumpy old admin/DBA... but mariadb or postgres used well just solve so many problems.

alexandercrohde 8y ago
>> Maybe I'm turning into a grumpy old admin/DBA No you're not; you're saving the company tens or hundreds of thousands of dollars over the coming years.
freehunter 8y ago
I once asked if Postgres would have any negative performance impact from holding 1.8m records and once the laughing stopped and they realized I was serious I got a good lesson on how robust Postgres can be.
- kod 8y ago
  I once worked for a company that maxed out a 32 bit autoincrement primary key on a mysql table. Relational databases can hold a lot more than people give them credit for.
  - icedchai 8y ago
    Yep. I worked at a company that had to migrate to 64-bit IDs. This was with MySQL 5.0 on servers with HDs, not SSD. Some of those "alter table" commands took all night!
  - pickle-wizard 8y ago
    Oh man, I hope they saw that one coming ahead of time.
  - erik_seaberg 8y ago
    A colleague describes this problem as "running out of numbers".
EdwardDiego 8y ago
> Or other people are wondering if we could replace all of our relational databases with kafka. Unless you're using your DB as a data transport, dear God why. And if you are, dear God why. I work with Kafka and it's very good for a certain subset of problems. Replacing RDBMSes is not one of them.
mtrn 8y ago
I managed complex datasets of a few TB with standard Unix tools just fine. If you are lucky, you have a sustained read and processing speed of 200M/s (or more with, if you are put in some work), which allows you to sift through a TB in a good hour. Sometimes I am wondering, how people will rationalize the layer bloat in 5-10 years, when hardware went through a few more upgrades.

by tetha 8y ago

alexandercrohde 8y ago

>> Maybe I'm turning into a grumpy old admin/DBA No you're not; you're saving the company tens or hundreds of thousands of dollars over the coming years.

freehunter 8y ago

I once asked if Postgres would have any negative performance impact from holding 1.8m records and once the laughing stopped and they realized I was serious I got a good lesson on how robust Postgres can be.

kod 8y ago
I once worked for a company that maxed out a 32 bit autoincrement primary key on a mysql table. Relational databases can hold a lot more than people give them credit for.
- icedchai 8y ago
  Yep. I worked at a company that had to migrate to 64-bit IDs. This was with MySQL 5.0 on servers with HDs, not SSD. Some of those "alter table" commands took all night!
- pickle-wizard 8y ago
  Oh man, I hope they saw that one coming ahead of time.
- erik_seaberg 8y ago
  A colleague describes this problem as "running out of numbers".

EdwardDiego 8y ago

> Or other people are wondering if we could replace all of our relational databases with kafka. Unless you're using your DB as a data transport, dear God why. And if you are, dear God why. I work with Kafka and it's very good for a certain subset of problems. Replacing RDBMSes is not one of them.

mtrn 8y ago

I managed complex datasets of a few TB with standard Unix tools just fine. If you are lucky, you have a sustained read and processing speed of 200M/s (or more with, if you are put in some work), which allows you to sift through a TB in a good hour. Sometimes I am wondering, how people will rationalize the layer bloat in 5-10 years, when hardware went through a few more upgrades.