5 ms·

I worked on a MySQL-on-RDS setup once where the company was splurging (and being charged through the nose, ofc) on a backing EBS volume with 20k IOPS. Meanwhile

by WJW 5y ago

I worked on a MySQL-on-RDS setup once where the company was splurging (and being charged through the nose, ofc) on a backing EBS volume with 20k IOPS. Meanwhile nobody had bothered to update the default settings, which limited all background work to use no more than 100 IOPS. Needless to say, this DB had severe performance problems. Managed databases in the cloud are a bit like ORMs: they significantly reduce the amount of knowledge needed 99% of the time. Sadly, when the 1% of the time hits this also means that nobody on the team has built up the DBA skills required for the big hairy problem. I see this pattern repeated all the time when consulting for startups that just had their first real traction and now the magic database box is no longer working as before.

mcherm 5y ago
I think DB tuning seems like a perfect business opportunity for specialist consulting. Doing a really good job requires deep expertise but then it only needs to be repeated occasionally (if the operational profile or the data size changes significantly). But I don't see this model much. I mostly hear of companies struggling to develop internal expertise or going without. Why?
- BrentOzar 5y ago
  > But I don't see this model much. Why? It does exist (I'm one of them), and it's much more common on expensive databases like Microsoft SQL Server and Oracle. With free databases, it's relatively inexpensive to throw more hardware at the problem. When licensing costs $2,000 USD per CPU core or higher, there's much more incentive for companies to bring an expert in quickly to get hardware costs under control.
  - redis_mlc 5y ago
    Yup, SQL Server performance and licensing (!) consultants came out of retirement after MS doubled the licensing costs in 2012. Managers don't like paying for optimization consultants, but it's the easiest way to both improve current performance and scale a master. Note that Oracle positions MySQL as a replacement for SQL Server, as "moderate-performance databases." :) Source: DBA. https://metrixdata360.com/pricing-series/1500-price-increase-on-sql-server/ https://metrixdata360.com/pricing-series/1500-price-increase...
- Intermernet 5y ago
  I did this for years. I was a server optimisation consultant. I'd always advise my clients that increasing the specs of the hardware would probably be cheaper in the short term than hiring me to go through the various configurations with a fine toothed comb. I also promised a fixed price report on what needed to be done, as well as a fixed price quote on doing the actual work. I found that this level of transparency and honesty never failed, and the work was always approved. I'd also offer training for the internal team. I've pretty much completely changed careers now, but I found that this model did work well. It involved a lot of constant research and occasionally a huge amount of stress, but I loved it!
- apavlo 5y ago
  > I think DB tuning seems like a perfect business opportunity I got you fam: https://ottertune.com/ https://ottertune.com/
idiocrat 5y ago
Call it driving with hand brakes on. You have bought your 1000 hps car and are driving it as-is since you left the car dealer's parking place. Nobody told you to release the hand brakes.
- WJW 5y ago
  Quite so. Of course, in most countries all drivers will have had mandatory driving lessons before they got their license and the instructor will have told them about the handbrake. No such license requirement exists for databases and it shows.
- daemoncoder 5y ago
  I like to say they should remove the block from under the accelerator pedal.
throwaway20371 5y ago
> Sadly, when the 1% of the time hits this also means that nobody on the team has built up the DBA skills required for the big hairy problem. I see this pattern repeated all the time when consulting for startups that just had their first real traction and now the magic database box is no longer working as before. I call this the Kubernetes effect. 99% of the time, people think it's easy. Then the 1% hits and they have to completely rebuild it from the ground up. For us it was cluster certificate expiration and etcd corruption, and some other problems we couldn't ever pin down. The other effect (which I'm sure has a name) is when they depend on it for a long time, but then a problem starts to occur, and because they can't figure out a way to solve it, they move to a completely different tech ("We had problems with updating our shards so we moved to a different database") and encounter a whole new problem and repeat the cycle.

by WJW 5y ago

mcherm 5y ago

I think DB tuning seems like a perfect business opportunity for specialist consulting. Doing a really good job requires deep expertise but then it only needs to be repeated occasionally (if the operational profile or the data size changes significantly). But I don't see this model much. I mostly hear of companies struggling to develop internal expertise or going without. Why?

BrentOzar 5y ago
> But I don't see this model much. Why? It does exist (I'm one of them), and it's much more common on expensive databases like Microsoft SQL Server and Oracle. With free databases, it's relatively inexpensive to throw more hardware at the problem. When licensing costs $2,000 USD per CPU core or higher, there's much more incentive for companies to bring an expert in quickly to get hardware costs under control.
- redis_mlc 5y ago
  Yup, SQL Server performance and licensing (!) consultants came out of retirement after MS doubled the licensing costs in 2012. Managers don't like paying for optimization consultants, but it's the easiest way to both improve current performance and scale a master. Note that Oracle positions MySQL as a replacement for SQL Server, as "moderate-performance databases." :) Source: DBA. https://metrixdata360.com/pricing-series/1500-price-increase-on-sql-server/ https://metrixdata360.com/pricing-series/1500-price-increase...
Intermernet 5y ago
I did this for years. I was a server optimisation consultant. I'd always advise my clients that increasing the specs of the hardware would probably be cheaper in the short term than hiring me to go through the various configurations with a fine toothed comb. I also promised a fixed price report on what needed to be done, as well as a fixed price quote on doing the actual work. I found that this level of transparency and honesty never failed, and the work was always approved. I'd also offer training for the internal team. I've pretty much completely changed careers now, but I found that this model did work well. It involved a lot of constant research and occasionally a huge amount of stress, but I loved it!
apavlo 5y ago
> I think DB tuning seems like a perfect business opportunity I got you fam: https://ottertune.com/ https://ottertune.com/

idiocrat 5y ago

Call it driving with hand brakes on. You have bought your 1000 hps car and are driving it as-is since you left the car dealer's parking place. Nobody told you to release the hand brakes.

WJW 5y ago
Quite so. Of course, in most countries all drivers will have had mandatory driving lessons before they got their license and the instructor will have told them about the handbrake. No such license requirement exists for databases and it shows.
daemoncoder 5y ago
I like to say they should remove the block from under the accelerator pedal.

throwaway20371 5y ago

> Sadly, when the 1% of the time hits this also means that nobody on the team has built up the DBA skills required for the big hairy problem. I see this pattern repeated all the time when consulting for startups that just had their first real traction and now the magic database box is no longer working as before. I call this the Kubernetes effect. 99% of the time, people think it's easy. Then the 1% hits and they have to completely rebuild it from the ground up. For us it was cluster certificate expiration and etcd corruption, and some other problems we couldn't ever pin down. The other effect (which I'm sure has a name) is when they depend on it for a long time, but then a problem starts to occur, and because they can't figure out a way to solve it, they move to a completely different tech ("We had problems with updating our shards so we moved to a different database") and encounter a whole new problem and repeat the cycle.