5 ms·

The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they rea

The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many engineers working below the waterline. Rather than work on the actual customer-facing problem, let's port the backend to do event-sourcing/cqrs, move all our infrastructure to k8s, change language from x to y etc. These are all what I would call "internal goals" (ie they may or may not be necessary or even essential to progress but are not directly customer-visible in their outcomes even if they may enable customer features to be built or indirectly improve the customer experience later) and need to be held to an extremely high level of scrutiny. If you're amazon/google/meta and you need to do this because of extreme user scale I might believe you. If you're CERN or someone and you need to do this because of absolutely ridiculous data scale I might believe you. The idea that it's better for figma to write their own sharding solution than it is to port to one of the existing ones just doesn't pass even the most basic sniff test.

djtango 3y ago
I don't have a read on this - do we know that Figma isn't doing difficult stuff that warrants proprietary solutions?
- jiggawatts 3y ago
  I just read through several thousand lines of code re-implementing the concept of a distributed queue from the ground up... for an application that has maybe a few hundred users. And doesn't need queues, at all. This issue is so pervasive that we've all just assumed that it must be necessary.
  - djtango 3y ago
    I couldn't get the context of this response - is this application you read unrelated to the featured article? I just read the article and from what I can tell the Figma team made a somewhat reasonable sounding decision
    - jiggawatts 3y ago
      Yes, unrelated. My point was that wheel-reinvention is a curse of the software industry because it's just so easy to reinvent every wheel on a whim. DevOps and is no different. How many large orgs have their own build tooling, or some special sauce around large repos?
- winrid 3y ago
  Absolutely not. Citus would have solved this problem. Or move to MySQL and use PlanetScale etc. Second best option is ability to easily create prod environments, and then give those to your biggest customers (bigname.figma.com) etc. No single figma customer will go beyond an i3.metal for the DB, or the app.
  - djtango 3y ago
    So I just read the article - they were on RDS so Citus wasn't an option. They also stated it was too risky to migrate data stores on the timeline they were working within Those all seem like measured engineering decisions AFAICT
    - winrid 3y ago
      That doesn't sound right. Do data dump from prod for initial sync and then setup replication from RDS to new cluster. Once synced do switch. Then you're off RDS and can shard on Citus.
      - raverbashing 3y ago
        Yes but they probably wouldn't want to migrate off RDS.
      - nathanappere 3y ago
        This doesn't work with the constraint of "staying on AWS" though.
        winrid 3y ago
        Yes it does.
      - djtango 3y ago
        a) they didn't want to move off RDS b) this is a pretty big hand wave over migrating your persistence store on a moving product and moving engineering teams The coordination alone usually takes months
thibaut_barrere 3y ago
I don't read the story this way personally (not saying that these scenarios do not occur, but I feel the narrative detailed in the original article makes sense even without "chasing cool problems").
thih9 3y ago
The article suggests a different reason. What would be your approach if you wanted to stay on RDS? > So, now, let me speculate. The real reason why Figma reinvented the wheel by creating their own custom solution for sharding might be as straightforward as this — Figma wanted to stay on RDS, and since Amazon had decided not to support the CitusData extension in the past, the Figma team had no choice but to develop their own sharding solution from scratch.
- MrJohz 3y ago
  This rings a lot more true for me as well: a lot of the overly complicated decisions I've made haven't been because I wanted to try something interesting out (although occasionally it's been a factor), but more because I've ended up backed into a corner by previous decisions, factors outside my control, and limited time. Even when the simpler solution is obvious (which isn't always the case), it often takes a more complicated journey to get there. And balancing short term vs long term complexity is a challenge in its own right.
- mbesto 3y ago
  Fair. But it doesn't really explain why they wanted to stay on RDS. This is their reasoning: > over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. So they had in house expertise to run performantly on RDS but that same experience couldn't be translated to switching over to it running on EC2 + Citus? Rather they used another non-experience concept of building their own sharding? That left me scratching my head.
  - lopkeny12ko 3y ago
    I was puzzled by this as well. RDS is a managed, cloud product. You don't run it. The whole point is that AWS runs it for you, no?
    - thih9 3y ago
      It’s Postgres, large dbs will need some level of config and maintenance. > Common DBA tasks for Amazon RDS for PostgreSQL https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appen...
fhd2 3y ago
I suppose Figma might just be beyond the "let's find the fastest/cheapest way to get this working" point. I believe it makes sense for a company in that stage to mess about a bit, find different (maaaybe even better) ways of doing things, keep the engineering work interesting to attract/retain talent, be OK with the inevitable waste involved in that game. If you're chasing the global maximum, you shouldn't get too obsessed with local maxima. That said, I've seen plenty of unprofitable startups with high burn rate play this game. That seems a bit suicidal to me.
- thih9 3y ago
  > I suppose Figma might just be beyond the "let's find the fastest/cheapest way to get this working" point. The article implies otherwise. E.g. it quotes Figma saying: “Given our very aggressive growth rate, we had only months of runway remaining.”
  - fhd2 3y ago
    Right, I was thinking of Figma in 2024 for some reason (they seem to have conquered the market, I'll just assume they're profitable with that pricing they have), the article talks about Figma in 2022, from what I gather. Should have read properly.
aerhardt 3y ago
I can buy your comment as an interesting and even credible hypothesis, but the absolutes which you deal in (“doesn’t pass even the most basic sniff test”) are damning. You are clearly lacking huge amounts of information and context and are passing your own assumptions as hard facts. Also, I’m assuming Amazon or Google will sometimes roll their own solutions on problems of a scale in the same ballpark as Figma’s. But anyhow, what’s the scale at which this becomes acceptable, exactly? Is there a magical number which serves as a universal threshold? Or is there - like in all engineering decisions - a very concrete economic case for which you and I both lack a lot of the requisite context and inputs?
- CoastalCoder 3y ago
  I like the approach you took for questioning an unqualified claim. Seems like a useful argument design pattern.
- dbuser99 3y ago
  In this particular case of sharding a postgresql solution, in my opinion, the parent is right. Any major cloud provider would give companies of their scale assistance. This is their bread and butter. The posts likely hide the requirement of stay on aws, but we don’t know they did not talk about that. Likewise cockroach or yugabyte were also available options.
jitl 3y ago
We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client. In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand damage alone. Possibly even company ending. In such a situation, failure is not an option, and you must pick the least risky solution. Moving to an unmanaged cluster system and figuring out your own point-in-time backup/restore, access control provisioning, etc etc has a lot more unknown unknowns than sticking with the managed database vendor you know. The potential failure scenarios of Citus have scary worst cases - we get backup and restore wrong but it seems to work fine in test, then we move to Citus, then something breaks and we can’t restore from backup after all. It’s equally bad to mis-estimate the amount of time needed to bring up the new system. Let’s say you estimate 6 months to get parity with RDS built in features needed to survive disaster and start moving data over, but instead it takes 10 months. Is there enough time left to finish before going hard down? The clock is ticking. Staying with RDS keeps a whole class of new risk out of the picture. At least here at Notion, NO ONE wanted to build something complicated for fun. We really wanted the company we’d spent years working for and on-call for to survive. Our story: https://www.notion.so/blog/sharding-postgres-at-notion https://www.notion.so/blog/sharding-postgres-at-notion
- hobs 3y ago
  Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.
  - jitl 3y ago
    It's not just manage Postgres, it's manage a Citus cluster - (unmanaged Postgres + postgres experts + time for them to implement their stuff) just gets us to parity with RDS but doesn't solve our sharding problem. We asked our Postgres consultants & networks to see if we could find Citus experts we could bring on full-time but didn't have great success. Most of the experts we talked to suggested application level sharding, and it seems like it worked out okay.

djtango 3y ago

I don't have a read on this - do we know that Figma isn't doing difficult stuff that warrants proprietary solutions?

jiggawatts 3y ago
I just read through several thousand lines of code re-implementing the concept of a distributed queue from the ground up... for an application that has maybe a few hundred users. And doesn't need queues, at all. This issue is so pervasive that we've all just assumed that it must be necessary.
- djtango 3y ago
  I couldn't get the context of this response - is this application you read unrelated to the featured article? I just read the article and from what I can tell the Figma team made a somewhat reasonable sounding decision
  - jiggawatts 3y ago
    Yes, unrelated. My point was that wheel-reinvention is a curse of the software industry because it's just so easy to reinvent every wheel on a whim. DevOps and is no different. How many large orgs have their own build tooling, or some special sauce around large repos?
winrid 3y ago
Absolutely not. Citus would have solved this problem. Or move to MySQL and use PlanetScale etc. Second best option is ability to easily create prod environments, and then give those to your biggest customers (bigname.figma.com) etc. No single figma customer will go beyond an i3.metal for the DB, or the app.
- djtango 3y ago
  So I just read the article - they were on RDS so Citus wasn't an option. They also stated it was too risky to migrate data stores on the timeline they were working within Those all seem like measured engineering decisions AFAICT
  - winrid 3y ago
    That doesn't sound right. Do data dump from prod for initial sync and then setup replication from RDS to new cluster. Once synced do switch. Then you're off RDS and can shard on Citus.
    - raverbashing 3y ago
      Yes but they probably wouldn't want to migrate off RDS.
    - nathanappere 3y ago
      This doesn't work with the constraint of "staying on AWS" though.
      - winrid 3y ago
        Yes it does.
    - djtango 3y ago
      a) they didn't want to move off RDS b) this is a pretty big hand wave over migrating your persistence store on a moving product and moving engineering teams The coordination alone usually takes months

thibaut_barrere 3y ago

I don't read the story this way personally (not saying that these scenarios do not occur, but I feel the narrative detailed in the original article makes sense even without "chasing cool problems").

thih9 3y ago

The article suggests a different reason. What would be your approach if you wanted to stay on RDS? > So, now, let me speculate. The real reason why Figma reinvented the wheel by creating their own custom solution for sharding might be as straightforward as this — Figma wanted to stay on RDS, and since Amazon had decided not to support the CitusData extension in the past, the Figma team had no choice but to develop their own sharding solution from scratch.

MrJohz 3y ago
This rings a lot more true for me as well: a lot of the overly complicated decisions I've made haven't been because I wanted to try something interesting out (although occasionally it's been a factor), but more because I've ended up backed into a corner by previous decisions, factors outside my control, and limited time. Even when the simpler solution is obvious (which isn't always the case), it often takes a more complicated journey to get there. And balancing short term vs long term complexity is a challenge in its own right.
mbesto 3y ago
Fair. But it doesn't really explain why they wanted to stay on RDS. This is their reasoning: > over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. So they had in house expertise to run performantly on RDS but that same experience couldn't be translated to switching over to it running on EC2 + Citus? Rather they used another non-experience concept of building their own sharding? That left me scratching my head.
- lopkeny12ko 3y ago
  I was puzzled by this as well. RDS is a managed, cloud product. You don't run it. The whole point is that AWS runs it for you, no?
  - thih9 3y ago
    It’s Postgres, large dbs will need some level of config and maintenance. > Common DBA tasks for Amazon RDS for PostgreSQL https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appen...

fhd2 3y ago

I suppose Figma might just be beyond the "let's find the fastest/cheapest way to get this working" point. I believe it makes sense for a company in that stage to mess about a bit, find different (maaaybe even better) ways of doing things, keep the engineering work interesting to attract/retain talent, be OK with the inevitable waste involved in that game. If you're chasing the global maximum, you shouldn't get too obsessed with local maxima. That said, I've seen plenty of unprofitable startups with high burn rate play this game. That seems a bit suicidal to me.

thih9 3y ago
> I suppose Figma might just be beyond the "let's find the fastest/cheapest way to get this working" point. The article implies otherwise. E.g. it quotes Figma saying: “Given our very aggressive growth rate, we had only months of runway remaining.”
- fhd2 3y ago
  Right, I was thinking of Figma in 2024 for some reason (they seem to have conquered the market, I'll just assume they're profitable with that pricing they have), the article talks about Figma in 2022, from what I gather. Should have read properly.

aerhardt 3y ago

I can buy your comment as an interesting and even credible hypothesis, but the absolutes which you deal in (“doesn’t pass even the most basic sniff test”) are damning. You are clearly lacking huge amounts of information and context and are passing your own assumptions as hard facts. Also, I’m assuming Amazon or Google will sometimes roll their own solutions on problems of a scale in the same ballpark as Figma’s. But anyhow, what’s the scale at which this becomes acceptable, exactly? Is there a magical number which serves as a universal threshold? Or is there - like in all engineering decisions - a very concrete economic case for which you and I both lack a lot of the requisite context and inputs?

CoastalCoder 3y ago
I like the approach you took for questioning an unqualified claim. Seems like a useful argument design pattern.
dbuser99 3y ago
In this particular case of sharding a postgresql solution, in my opinion, the parent is right. Any major cloud provider would give companies of their scale assistance. This is their bread and butter. The posts likely hide the requirement of stay on aws, but we don’t know they did not talk about that. Likewise cockroach or yugabyte were also available options.

jitl 3y ago

We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client. In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand damage alone. Possibly even company ending. In such a situation, failure is not an option, and you must pick the least risky solution. Moving to an unmanaged cluster system and figuring out your own point-in-time backup/restore, access control provisioning, etc etc has a lot more unknown unknowns than sticking with the managed database vendor you know. The potential failure scenarios of Citus have scary worst cases - we get backup and restore wrong but it seems to work fine in test, then we move to Citus, then something breaks and we can’t restore from backup after all. It’s equally bad to mis-estimate the amount of time needed to bring up the new system. Let’s say you estimate 6 months to get parity with RDS built in features needed to survive disaster and start moving data over, but instead it takes 10 months. Is there enough time left to finish before going hard down? The clock is ticking. Staying with RDS keeps a whole class of new risk out of the picture. At least here at Notion, NO ONE wanted to build something complicated for fun. We really wanted the company we’d spent years working for and on-call for to survive. Our story: https://www.notion.so/blog/sharding-postgres-at-notion https://www.notion.so/blog/sharding-postgres-at-notion

hobs 3y ago
Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.
- jitl 3y ago
  It's not just manage Postgres, it's manage a Citus cluster - (unmanaged Postgres + postgres experts + time for them to implement their stuff) just gets us to parity with RDS but doesn't solve our sharding problem. We asked our Postgres consultants & networks to see if we could find Citus experts we could bring on full-time but didn't have great success. Most of the experts we talked to suggested application level sharding, and it seems like it worked out okay.