3 ms·
> I can get someone from the backend performance team to comment too if you'd like :) Sure! > the biggest bottleneck at this point is with the file system. W
by dineshp2 10y ago
> I can get someone from the backend performance team to comment too if you'd like :)
Sure!
> the biggest bottleneck at this point is with the file system.
Would something like AWS EFS [1] solve the problem?
It might not be a practical solution from a cost perspective though. Eg: for 50TB of data at $0.3/GB/month
0.3*1024*50 = $15360/month
[1] https://aws.amazon.com/efs/ https://aws.amazon.com/efs/
- pcarranza 10y agoThe main issue with this is that this would be a vendor specific solution that would lock everybody. We build a product that you can host yourself. If we only solve the scaling issue by pushing it down to a specific vendor, then there is no actual solution. The way we are facing the problem is first by enabling a really easy and simple form of sharding (that would solve most of the issues that a lot of big customers may face), and then by using an open source underlaying filesystem that can scale reasonably well.