jamwt

8 ms·

61.
by jamwt 10y ago
Correct, they're all 64-bit linux. We might use some Rust on 32-bit soon though. So this is interesting to us.
62.
Stud No More(blog.jamwt.com)
2 points by jamwt 11y ago|0 comments
63.
by jamwt 11y ago
We actually use several different servers from different vendors, and some of them are more customized than others.
64.
by jamwt 11y ago
Live writes are written out with 4x redundancy in the local zone, then asynchronously replicated out to the remote zone. Some time later, it is erasure coded into a more space-efficient format, independently in each zone.
65.
by jamwt 11y ago
We use a custom version of jemalloc, with profiling enabled so that we can use it. We also tweak jemalloc pretty heavily for our workload.
66.
by jamwt 11y ago
Yep, vendors do sometimes provide stock to large customers ahead of general availability.
67.
by jamwt 11y ago
SATA SMR disks, zone management is ZAC.
68.
by jamwt 11y ago
Magic Pocket is a block storage system, and all storage in it is append only. We'll go into more detail on the on-disk format in a blog post. But, yes, we use the SMR disks on an HBA, and we directly address the disk on a LBA (and zo
69.
by jamwt 11y ago
We do want to minimize that--compute costs money, and storage is irreducible. But you can only reduce aggressively if your larger distributed system is great at repairs, since the failure of a single box kicks off 1PB of repair activity!
70.
by jamwt 11y ago
Yes, balancing links and distributing work w/out congestion collapse became a really interesting and nontrivial problem.
71.
by jamwt 11y ago
I mean, if at home you need 400k IOPS, 1GB/s of writes, and 2.2GB/s of reads... go for it! I sure don't, but more power to you. :-)
72.
by jamwt 11y ago
Building a massive, super-reliable storage system was not really feasible given Dropbox's resources back when it first started up.
73.
by jamwt 11y ago
You can substantially change the economics if you design your own power distribution systems.
74.
by jamwt 11y ago
> Are you using spindle or ssd or flash? (admittedly I dont know if you consider flash and ssd to be the same) We have flash caches (NVMe) in the machines, but the long-term storage devices are spindle--PMR and SMR. > What is your pri
75.
by jamwt 11y ago
Awesome! Thanks for being a customer. We'll continue to work hard to make Dropbox better and better.
76.
by jamwt 11y ago
Yeah, our hardware teams put a lot of qualification time into hardware profiles before we buy in bulk. And part of what we test is power utilization, heat, etc, under normal software loads vs extreme loads, to make sure we don't excee
77.
by jamwt 11y ago
We probably "pulled forward" every two months or so. Back in early 2015, sometimes we'd have breakages that took a bit of work to remedy. But post 1.0, the language has been very stable. As usual, we actually have more chal
78.
by jamwt 11y ago
Right now, they talk over RPC. In the future, we might embed Rust libraries as C libraries into go and other languages.
79.
by jamwt 11y ago
http://googleforstudents.blogspot.com/2012/06/site-reliabili...
80.
by jamwt 11y ago
We have an in-house futures-based framework (inspired by Finagle) built on mio (non-blocking libevent like thing for rust). All I/O is async, but application work is often done on thread pools. Those threads are freed as soon as poss
81.
by jamwt 11y ago
Sometimes; but depreciation costs are usually more than either of those (for storage).
82.
by jamwt 11y ago
We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.
83.
by jamwt 11y ago
We use Reed Solomon variants with local reconstruction codes.
84.
by jamwt 11y ago
I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like
85.
by jamwt 11y ago
We have several dozen very, very good engineers working on a few discrete teams--infrastructure security, product security, etc. I don't feel comfortable getting any more specific than that; we keep security information pretty close
86.
by jamwt 11y ago
We still have a ton of Python. James is referring to the infrastructure software layer--storage services, monitoring, etc. That particular part of Dropbox is mostly Go. But our web controller code, for example, is millions of lines of Pyt
87.
by jamwt 11y ago
We're on rustc-serialize. JSON is not really a part of our data pipeline, just our metrics pipeline. So the performance of the library is not especially critical.
88.
by jamwt 11y ago
Yes, a 1PB+ failure domain only made sense because Magic Pocket is very good at automatic cluster management and repair.
89.
by jamwt 11y ago
We use 3rd parties for the "obvious" stuff. Like, we're not going to reinvent json serialization. But we typically don't use any 3rd party frameworks on the backend. So things like service management/discovery, r
90.
by jamwt 11y ago
Little known fact, but already we have more business customers than Box: https://www.dropbox.com/business/customers

More ›