4 ms·

Interesting insights and refreshing to see large scale providers still relying on „old-school“ tech! What I did not fully grasp - isn’t Vitess it’s own databas

Interesting insights and refreshing to see large scale providers still relying on „old-school“ tech! What I did not fully grasp - isn’t Vitess it’s own database system built on top of a K/V store similar to e.g. CockroachDB? Here it sounds like they only use pieces of Vitess in front of regular MySQL primaries with sharding? It’s also curious that they sharded based on domains. I assume at some point the QPS on specific domains (e.g. Gist) will be so high that partitioning no longer is effective? edit:// Looks like Vitess is indeed a set of tools on top of MySQL that allow for these type of scaling operations. It is not re-implementation of MySQL using different underlying technologies.

alexpotato 5y ago
I remember reading a post from Jeremy Zawodny back in the early 2000s that was basically: "Start virtually sharding your DB data so that when you have to physically shard it, it's a lot easier."
- baybal2 5y ago
  The whole of Alibaba websites (Aliexpress, TaoBao, Tmall) rely on a single MySQL database for stock count keeping.
  - matesz 5y ago
    Also they developed their own storage engine for MySQL (X-Engine) which is based on LSM data structure and using FPGA to accelerate levels compaction [1], whitepaper [2]. "In addition, FPGA hardware is used to accelerate the compaction process, further maximizing the performance of the system. This marks the first time that hardware acceleration was applied to the storage engine of an OLTP database." [1] https://www.alibabacloud.com/blog/new-insights-into-x-engine https://www.alibabacloud.com/blog/new-insights-into-x-engine... [2] https://dl.acm.org/doi/pdf/10.1145/3299869.3314041 https://dl.acm.org/doi/pdf/10.1145/3299869.3314041
    - baybal2 5y ago
      404
      - matesz 5y ago
        https://www.alibabacloud.com/blog/new-insights-into-x-engine-the-new-storage-engine-of-rds-for-mysql_596209 https://www.alibabacloud.com/blog/new-insights-into-x-engine...
WatchDog 5y ago
> In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth. We’ll detail the tooling, linters, and Rails improvements related to this in a future blog post. They make a clear distinction between "vertical partitioning" and "horizontal partitioning", I think there are probably bigger wins to be had, and probably bigger challenges in the horizontal partitioning than the vertical. All of this work only got them a 2 factor reduction in load, whereas with an ideal horizontal solution, they could scale it infinitely.
d3nj4l 5y ago
refreshing to see large scale providers still relying on „old-school“ tech! Somewhat OT but Github is really "old school" - they don't have an SPA frontend and render most things server side the classic Rails way, with lots of UX enhancements coming from judicious use of JS and websockets for interactions. That's one of the things I find very endearing about their platform.
- mediumdeviation 5y ago
  That approach unfortunately also makes keeping state updates consistent very difficult. Some pet peeves - when reviewing code, if you set up some file filters then scroll down to trigger more files to load, those newly loaded files are not affected by the previously set filters. Or occasionally the page would helpfully offer to reload to show new commits, but clicking that link loses the current white space and file filter settings. SPA frameworks aren’t a silver bullet here, and you can definitely fix these without using one, but these bugs are definitely much harder to make since the DOM is rendered from a single source of truth instead of being patched piecemeal.
  - d3nj4l 5y ago
    That’s a good point. I’ve faced a couple of issues like that doing pure server side rails and mostly dealt with them by not having “infinite” scroll, though I guess that’s not an option in all cases. I think Stimulus can go some way in making experiences like that possible without a full SPA.
qeternity 5y ago
Vitess is to MySQL as Citus is to Postgres
- xdanger 5y ago
  And Microsoft owns Citus and Github.
tybit 5y ago
What you’re describing sounds more like TiDB. Like a MySQL flavoured cockroach DB with its own K/V store TiKV.
paulfurtado 5y ago
Vitess can be thought of as a very smart MySQL proxy with additional MySQL management features built in. Centralized vtgates proxy to vttablet services which run on the same host as MySQL, the tablet then queries the local MySQL server. On top of this, a lot of magic can be built

alexpotato 5y ago

I remember reading a post from Jeremy Zawodny back in the early 2000s that was basically: "Start virtually sharding your DB data so that when you have to physically shard it, it's a lot easier."

baybal2 5y ago
The whole of Alibaba websites (Aliexpress, TaoBao, Tmall) rely on a single MySQL database for stock count keeping.
- matesz 5y ago
  Also they developed their own storage engine for MySQL (X-Engine) which is based on LSM data structure and using FPGA to accelerate levels compaction [1], whitepaper [2]. "In addition, FPGA hardware is used to accelerate the compaction process, further maximizing the performance of the system. This marks the first time that hardware acceleration was applied to the storage engine of an OLTP database." [1] https://www.alibabacloud.com/blog/new-insights-into-x-engine https://www.alibabacloud.com/blog/new-insights-into-x-engine... [2] https://dl.acm.org/doi/pdf/10.1145/3299869.3314041 https://dl.acm.org/doi/pdf/10.1145/3299869.3314041
  - baybal2 5y ago
    404
    - matesz 5y ago
      https://www.alibabacloud.com/blog/new-insights-into-x-engine-the-new-storage-engine-of-rds-for-mysql_596209 https://www.alibabacloud.com/blog/new-insights-into-x-engine...

WatchDog 5y ago

> In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth. We’ll detail the tooling, linters, and Rails improvements related to this in a future blog post. They make a clear distinction between "vertical partitioning" and "horizontal partitioning", I think there are probably bigger wins to be had, and probably bigger challenges in the horizontal partitioning than the vertical. All of this work only got them a 2 factor reduction in load, whereas with an ideal horizontal solution, they could scale it infinitely.

d3nj4l 5y ago

refreshing to see large scale providers still relying on „old-school“ tech! Somewhat OT but Github is really "old school" - they don't have an SPA frontend and render most things server side the classic Rails way, with lots of UX enhancements coming from judicious use of JS and websockets for interactions. That's one of the things I find very endearing about their platform.

mediumdeviation 5y ago
That approach unfortunately also makes keeping state updates consistent very difficult. Some pet peeves - when reviewing code, if you set up some file filters then scroll down to trigger more files to load, those newly loaded files are not affected by the previously set filters. Or occasionally the page would helpfully offer to reload to show new commits, but clicking that link loses the current white space and file filter settings. SPA frameworks aren’t a silver bullet here, and you can definitely fix these without using one, but these bugs are definitely much harder to make since the DOM is rendered from a single source of truth instead of being patched piecemeal.
- d3nj4l 5y ago
  That’s a good point. I’ve faced a couple of issues like that doing pure server side rails and mostly dealt with them by not having “infinite” scroll, though I guess that’s not an option in all cases. I think Stimulus can go some way in making experiences like that possible without a full SPA.

qeternity 5y ago

Vitess is to MySQL as Citus is to Postgres

xdanger 5y ago
And Microsoft owns Citus and Github.

tybit 5y ago

What you’re describing sounds more like TiDB. Like a MySQL flavoured cockroach DB with its own K/V store TiKV.

paulfurtado 5y ago

Vitess can be thought of as a very smart MySQL proxy with additional MySQL management features built in. Centralized vtgates proxy to vttablet services which run on the same host as MySQL, the tablet then queries the local MySQL server. On top of this, a lot of magic can be built