5 ms·

"The team removed table joins and complex queries from the database layer." As a database person, I dislike reading this. I understand their need and priority

"The team removed table joins and complex queries from the database layer." As a database person, I dislike reading this. I understand their need and priority to keep things running. But I would have loved it if they emphasized that doing something like this has so many disadvantages. Normalization, referential integrity and a powerful query language bring so many benefits. I see young people oftentimes seeing relational vs. NoSQL as two totally valid opinions, like Coke vs. Pepsi. And not as one thing runs the world for decades and is perfect for >90% of use cases, and the other one is for niche cases and fast hyper-scalers.

t8sr 3y ago
I don’t like it either, but apparently semi-structured data is the future. It’s proving easier to build systems that tolerate bad data and broken references than it is to get our data normalized and cleaned. I’m the guy on most teams pontificating about SQL and data-oriented code, but even I’ll concede that large scale live systems probably shouldn’t serve from an SQL database.
- koliber 3y ago
  Do they have only one huge table in the db that contains all of their entities? From the key structure that seems reasonable: shard id, type, object id. If that is the case, is a relational db still the right place to keep such data? It seems that if the goal is to store this kind of data in one or a few huge sharded tables, a different tool could be optimizing to do the job better. It seems that I’m missing something…
  - rwultsch 3y ago
    A pin was a 1.2 KB json blob. There were other tables but pins was the big one. Why MySQL? It did not destroy data like the alternatives. how storage became efficient https://medium.com/pinterest-engineering/evolving-mysql-compression-part-1-7f8b09666589 https://medium.com/pinterest-engineering/evolving-mysql-comp... https://medium.com/pinterest-engineering/evolving-mysql-compression-part-2-2c3eb0101205 https://medium.com/pinterest-engineering/evolving-mysql-comp...
rwultsch 3y ago
Before I joined in late 2013, they had not known how to run schema change without downtime. Once we fixed the kernel the db’s were nearly completely untaxed in terms of performance. They did however need large instances due to disk usage.
tmsh 3y ago
In 2024 all major cloud providers have NoSQL (AWS DynamoDB and its equivalents) that will easily scale to beyond this use case, managed purely by good single table design. That is the best practice for this and most basic use cases — not RDBMS in 2024. Why? key/value data stores have consistent hashing which scales to planet scale built in. No need for sharding or caching or other bs that brings down slack and GitHub every other month.
- rsanek 3y ago
  do you have a reference to how this might be accomplished? how do you avoid high read / write latencies if you put everything into a single table / key-value pair per user?
  - Zanfa 3y ago
    I think they're referring to single-table data design as described in https://www.youtube.com/watch?v=HaEPXoXVf2k https://www.youtube.com/watch?v=HaEPXoXVf2k It's quite clever and scales well, but completely unnecessary for the vast majority of CRUD apps.
    - evntdrvn 3y ago
      IMO it’s also difficult when you’re in the initial phases of a new company/product, because the system is evolving fast and that doesn’t play nice with single-table-design needing upfront design to work well. I have had much better luck sticking to a plain old relational DB stack for those phases, because it’s fast to iterate, and then consider moving to single-table NOSQL when the system/DDD has “gelled” and traffic is starting to hockey-stick
- Denvercoder9 3y ago
  Most applications don't need planet scale, though.
- adrianN 3y ago
  I know cases where caches on top of a kv store improved performance a lot. I don’t think it’s as simple as you claim.
fakecontent000 3y ago
[dead]
jimlikeslimes 3y ago
Like another comment pointed out they seem to store pins as a JSON blob in a single column anyway. They probably should've used Drupal. Really, this is pinterest. They store 'boards' and 'pins', you may as well have a bit of fun with it. What use cases do they have for the features you list? They probably only used a database for write transactions? I deeply value database technology and use it daily, but I don't think something simple like this needs to worry too hard about it.

t8sr 3y ago

I don’t like it either, but apparently semi-structured data is the future. It’s proving easier to build systems that tolerate bad data and broken references than it is to get our data normalized and cleaned. I’m the guy on most teams pontificating about SQL and data-oriented code, but even I’ll concede that large scale live systems probably shouldn’t serve from an SQL database.

koliber 3y ago
Do they have only one huge table in the db that contains all of their entities? From the key structure that seems reasonable: shard id, type, object id. If that is the case, is a relational db still the right place to keep such data? It seems that if the goal is to store this kind of data in one or a few huge sharded tables, a different tool could be optimizing to do the job better. It seems that I’m missing something…
- rwultsch 3y ago
  A pin was a 1.2 KB json blob. There were other tables but pins was the big one. Why MySQL? It did not destroy data like the alternatives. how storage became efficient https://medium.com/pinterest-engineering/evolving-mysql-compression-part-1-7f8b09666589 https://medium.com/pinterest-engineering/evolving-mysql-comp... https://medium.com/pinterest-engineering/evolving-mysql-compression-part-2-2c3eb0101205 https://medium.com/pinterest-engineering/evolving-mysql-comp...

rwultsch 3y ago

Before I joined in late 2013, they had not known how to run schema change without downtime. Once we fixed the kernel the db’s were nearly completely untaxed in terms of performance. They did however need large instances due to disk usage.

tmsh 3y ago

In 2024 all major cloud providers have NoSQL (AWS DynamoDB and its equivalents) that will easily scale to beyond this use case, managed purely by good single table design. That is the best practice for this and most basic use cases — not RDBMS in 2024. Why? key/value data stores have consistent hashing which scales to planet scale built in. No need for sharding or caching or other bs that brings down slack and GitHub every other month.

rsanek 3y ago
do you have a reference to how this might be accomplished? how do you avoid high read / write latencies if you put everything into a single table / key-value pair per user?
- Zanfa 3y ago
  I think they're referring to single-table data design as described in https://www.youtube.com/watch?v=HaEPXoXVf2k https://www.youtube.com/watch?v=HaEPXoXVf2k It's quite clever and scales well, but completely unnecessary for the vast majority of CRUD apps.
  - evntdrvn 3y ago
    IMO it’s also difficult when you’re in the initial phases of a new company/product, because the system is evolving fast and that doesn’t play nice with single-table-design needing upfront design to work well. I have had much better luck sticking to a plain old relational DB stack for those phases, because it’s fast to iterate, and then consider moving to single-table NOSQL when the system/DDD has “gelled” and traffic is starting to hockey-stick
Denvercoder9 3y ago
Most applications don't need planet scale, though.
adrianN 3y ago
I know cases where caches on top of a kv store improved performance a lot. I don’t think it’s as simple as you claim.

fakecontent000 3y ago

[dead]

jimlikeslimes 3y ago

Like another comment pointed out they seem to store pins as a JSON blob in a single column anyway. They probably should've used Drupal. Really, this is pinterest. They store 'boards' and 'pins', you may as well have a bit of fun with it. What use cases do they have for the features you list? They probably only used a database for write transactions? I deeply value database technology and use it daily, but I don't think something simple like this needs to worry too hard about it.