We initially had this issue as well. So we put a small team of developers to work on the problem and were able to implement our own version of the metastore tha

We initially had this issue as well. So we put a small team of developers to work on the problem and were able to implement our own version of the metastore that supported optimistic concurrency and hidden partitioning. It has significantly improved reliability in stress tests (variable velocity ball bouncing in a hyper-dimensional click-cube environment). however, the larger size of our object store is increasing our monthly S3 costs.