4 ms·
Well then, tell us about the system YOU built to batch process millions of images that doesn't use something "weak sauce" like NFS. In the real world, you use
by packetslave 13y ago
Well then, tell us about the system YOU built to batch process millions of images that doesn't use something "weak sauce" like NFS.
In the real world, you use what works, not the flavor of the month.
- stass 13y agoGiven that for images one does not need to do random I/O, any kind of object storage system will do much better than NFS or any other kind of network filesystem with POSIX semantics guarantees. And no, in real world, you don't use what works, you come up with a design other people won't have to painfully maintain for the next 5 years.
- dmourati 13y agoSure, I built a system for a company specializing in image processing and uploads/sharing called Eye-Fi. We used MogileFS for image storage. Resizing millions of images was a daily task. NFS is not the right tool but is frequently mistaken for such.
- packetslave 13y agoNFS can be the right solution, depending on the problem. Storage backend for serving live images to users? Probably not. Taking uploaded images and storing a copy (after processing), maybe. Depends on the amount of IOPS you expect to need to throw at those copies. I haven't seen any posts from Etsy on their normal image pipeline, so it's hard to say. It sounds like this was a one-time conversion. Their live image serving is on Akamai, but nothing says they're back-ending the origin servers on NFS. I have to imagine with the amount of ops talent at Etsy, they're not using NFS because they're too dumb to know about a better option. I'm guessing John Allspaw knows a thing or two about how to serve images at high volume.
- dmourati 13y agoThe thought that Etsy had good ops people was what lead to my surprise in relying on NFS for anything of the sort. Two mitigating facts: one, this article is from 2010, two, this may have been a one-time thing. Still, surprised and dismayed that they would let NFS into production.
- code_duck 13y agoEtsy grew quite quickly, after having been founded by people with little experience running a website at scale. Once an experienced and competent team came on board, I believe they spent a lot of effort recovering from poor decisions made by inexperienced people under duress.
- icelancer 13y ago>Well then, tell us about the system YOU built to batch process millions of images that doesn't use something "weak sauce" like NFS. dmourati was being a jerk but boy, this particular response is hilarious. I suggest Googling his name with some image processing terms if you feel like eating crow.