3 ms·
No information in this article. How much data? We don't even know if it's GB/TB/PB/EB? How many files/objects? How many read IOPS are needed? How many write IO
by user5994461 10y ago
No information in this article.
How much data? We don't even know if it's GB/TB/PB/EB? How many files/objects? How many read IOPS are needed? How many write IOPS? What's the current setup on AWS? What's the current cost? What are they hosting? Can it scale horizontally? How do they shard jobs/users? What's running on Postgre? What's running on Ceph? What's running on NFS? How much disk bandwidth is used? How much network bandwidth is used?
How are we supposed to review their architecture if they don't explain anything...
I bet that there is a valid narrative where PostgreSQL and NFS was their doom, but I'd need data to explain that ^^
- connorshea 10y agoUnfortunately I don't have deep knowledge of our infrastructure, so I can't answer all of these questions myself. That said, a decent chunk of this info can be found in the discussions on our Infrastructure issue tracker[1]. The last infrastructure update[2] includes some slide decks that contain more data (albeit it's now a little under 2 months old). Looking at our internal Grafana instance, it looks like we're using about 1.25 TiB combined on NFS and just under 16 TiB on Ceph. We're working on migrating the data currently hosted on Ceph back to NFS soon[3]. I'll get someone from the infrastructure team to respond with more info. [1]: https://gitlab.com/gitlab-com/infrastructure/issues https://gitlab.com/gitlab-com/infrastructure/issues [2]: https://about.gitlab.com/2016/09/26/infrastructure-update/ https://about.gitlab.com/2016/09/26/infrastructure-update/ [3]: https://gitlab.com/gitlab-com/infrastructure/issues/711 https://gitlab.com/gitlab-com/infrastructure/issues/711
- flik 10y ago>>If one of the hosts delays writing to the journal, then the rest of the fleet is waiting for that operation alone, and the whole file system is blocked. When this happens, all of the hosts halt, and you have a locked file system; no one can read or write anything and that basically takes everything down. I have seen similar issues where a GC pause on one server, freeze the entire cluster. Is this one single monolithic file system? On the service side, can the code be asynchronous with request queues for each shard? This can help free up threads from getting blocked and serve requests for other shards.