4 ms·

With drives like this, the approach of "throw more hardware at it" continues working for databases to the point where most database loads in the world can be ha

With drives like this, the approach of "throw more hardware at it" continues working for databases to the point where most database loads in the world can be handled on a single machine.

rosser 10y ago
As someone who's worked with PostgreSQL for a decade now, I've very definitely seen cases where memory bandwidth is the bottleneck, not disk bandwidth. I'm pretty sure the IO subsystem would still be a narrower pipe than main memory, even if you stuck Optane devices in every available connector at the same time and striped across them. EDIT: Don't mistake me, I'm very excited about the potential of Optane devices in the workloads my databases handle (though, since my postgres machines are all in the cloud right now, that remains a purely theoretical question). It's just not a panacea. Then again, nothing is.
zzzcpan 10y ago
"where most database loads in the world can be handled on a single machine" I don't think it works like that. There is always some point when at certain number of users avoiding downtime becomes more important, than handling a load and hardware performance stops matter as much.
- Shivetya 10y ago
  high availability systems can counter this and if properly implemented you could share certain workloads with both the primary and HA systems; basically anything which doesn't change data. the its memory and storage would be a boon to dynamic indexing and common table expressions. maybe this new technology will bring single level storage as IBM has employed on some machines to the public. Where all storage is treated as one resource and only the machine knows the difference.
jrockway 10y ago
Scaling beyond what fits inside one computer is a reasonable concern that this addresses, but ultimately that one computer can be sucked up by a tornado at an inconvenient time, so distributed systems will always be necessary for availability and durability.
- yjftsjthsd-h 10y ago
  Resiliency, yes, but maybe not performance. Still a great win :)
peterwwillis 10y ago
There are faster drives. Load is just one factor in database design. For example, if you want fast query returns, the speed of light will limit you if you're using one machine and your query comes from around the world. Then there's the whole high availability thing. One single machine just doesn't fit every scenario.
kartan 10y ago
The more powerful the computers, the more data we want to aggregate and process. All that tracking that companies are gathering has to be used somewhere. Also SaaS means that one provider holds data of thousands of companies. Salesforce, for example, has more than 100,000 customers. So the scale is still too big for one machine (or two or three for redundancy).
- brianwawok 10y ago
  Sure but Salesforce has natural sharding. Customers never need to see each other's data. So can a single DB hold the largest customer? That is more the question here...
  - snuxoll 10y ago
    > Customers never need to see each other's data. Correct and incorrect at the same time. Separate orgs may not need direct access to each others rows in the multi-tenant database, but there are plenty of use cases where data does need to be shared between them (companies with multiple orgs, like ours - business parters, etc) and Salesforce has tooling to handle this (Salesforce to Salesforce, Lightning Connect).

With drives like this, the approach of "throw more hardware at it" continues working for databases to the point where most database loads in the world can be handled on a single machine.

rosser 10y ago

As someone who's worked with PostgreSQL for a decade now, I've very definitely seen cases where memory bandwidth is the bottleneck, not disk bandwidth. I'm pretty sure the IO subsystem would still be a narrower pipe than main memory, even if you stuck Optane devices in every available connector at the same time and striped across them. EDIT: Don't mistake me, I'm very excited about the potential of Optane devices in the workloads my databases handle (though, since my postgres machines are all in the cloud right now, that remains a purely theoretical question). It's just not a panacea. Then again, nothing is.

zzzcpan 10y ago

"where most database loads in the world can be handled on a single machine" I don't think it works like that. There is always some point when at certain number of users avoiding downtime becomes more important, than handling a load and hardware performance stops matter as much.

Shivetya 10y ago
high availability systems can counter this and if properly implemented you could share certain workloads with both the primary and HA systems; basically anything which doesn't change data. the its memory and storage would be a boon to dynamic indexing and common table expressions. maybe this new technology will bring single level storage as IBM has employed on some machines to the public. Where all storage is treated as one resource and only the machine knows the difference.

jrockway 10y ago

Scaling beyond what fits inside one computer is a reasonable concern that this addresses, but ultimately that one computer can be sucked up by a tornado at an inconvenient time, so distributed systems will always be necessary for availability and durability.

yjftsjthsd-h 10y ago
Resiliency, yes, but maybe not performance. Still a great win :)

peterwwillis 10y ago

There are faster drives. Load is just one factor in database design. For example, if you want fast query returns, the speed of light will limit you if you're using one machine and your query comes from around the world. Then there's the whole high availability thing. One single machine just doesn't fit every scenario.

kartan 10y ago

The more powerful the computers, the more data we want to aggregate and process. All that tracking that companies are gathering has to be used somewhere. Also SaaS means that one provider holds data of thousands of companies. Salesforce, for example, has more than 100,000 customers. So the scale is still too big for one machine (or two or three for redundancy).

brianwawok 10y ago
Sure but Salesforce has natural sharding. Customers never need to see each other's data. So can a single DB hold the largest customer? That is more the question here...
- snuxoll 10y ago
  > Customers never need to see each other's data. Correct and incorrect at the same time. Separate orgs may not need direct access to each others rows in the multi-tenant database, but there are plenty of use cases where data does need to be shared between them (companies with multiple orgs, like ours - business parters, etc) and Salesforce has tooling to handle this (Salesforce to Salesforce, Lightning Connect).