14 ms·
Flickr – A Year Without a Byte
- keeganjw 10y agoWow, this is really impressive! A whole year without any new storage for such a large enterprise is downright miraculous.
- khaledtaha 10y agoYes, but I would like to see this against their metrics. Were they growing or shrinking in terms of activity at the same time? And by how much? I suspect that Flickr's past the explosive growth stage so an optimization like this is timely.
- WatchDog 10y agoHard to really judge without knowing how much excess storage they started the year with.
- londons_explore 10y agoSome large organisations store as many as 15 copies of each bit of user data... When you have a database which keeps redundant master slave replicas and mutation logs, a backup system which keeps many previous backups on-site and offshore on tapes, a storage system that has many RAID mirrors per host, a distributed filesystem which stores replicated chunks of data across an entire datacenter to handle host failure, and you keep copies of the entire lot in multiple datacenters for accessibility during planned downtime and natural disasters.
- blakesterz 10y agoInteresting read... They're just using "S3 costs" as a comparison we can all understand? They don't use S3 do they?
- cschmidt 10y agoRight, they're just using an understandable basis, without having to talk about what they pay.
- twiss 10y agoThe first link in the article [1] says Yahoo deploys Ceph internally for object storage. [1]: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-ob...
- revelation 10y agoGotta get the capex down for new management moving in.
- git-pull 10y agoI'll tell you what Flickr isn't spending money or time on: Support. I have a flickr pro account from 6 or so years ago with hundreds of photos on it. I've tried over 10 times over a year to contact their support and get turned over to Tech Support in India that won't even read into your case! Of course, the original email address I used for my flickr was deleted, so none of the avenues on Yahoo Help (which is where they redirect you) work. Not to mention the password may be reset after all the leaks Yahoo had. So when I see these people on @FlickrHelp on Twitter (No replies) and Flickr having office parties, it really makes me feel quite disappointed! Yeah sure, real human touch! Former paying customer who just wants to login his account with tons of priceless photos. And they have a thread of like thousands of people who can't get into their accounts [1] At least the employees are having fun with data compression. Sad I can't talk to an actual human to get access to my account! [1] https://www.flickr.com/help/forum/en-us/72157668446997150/ https://www.flickr.com/help/forum/en-us/72157668446997150/
- acomjean 10y agoI have the same issue. Fortunately I can log in if I remember just correctly what email I used for my Yahoo! account. my browser fortunately remembers and the credentials in lightroom still allow my to put stuff up there.
- foota 10y agoYou could probably pull your password out if you really need it.
- nemeth88 10y agoIf your browser has the password saved then you can extract it now and save it to a more secure place, like a password manager. Presumably you could then use it to update your Yahoo! account email to a current, working address. As noted here - http://stackoverflow.com/questions/30013032/prevent-user-to-find-password-through-firebug-chrome-dev-tools http://stackoverflow.com/questions/30013032/prevent-user-to-... you can get it using your browser's web inspector dev tool.
- acd 10y agoSwitching over to erasure based storage such as Minio could bring down the cost even further. https://www.minio.io/ https://www.minio.io/
- fnord123 10y agoMinio is an interesting new project but they seem to be favouring "features" over stability. For now it would be better to go with a proven system like WD Active Archive or Scality.
- Veratyr 10y agoThey're already on Ceph-based Cloud Object Store, which uses 8/3 erasure coding: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-ob... Erasure coding isn't something new, unknown or unique to Minio, it's built into Ceph, GlusterFS and Openstack Swift, the largest distributed object stores.
- londons_explore 10y agoI want to see a distributed erasure coding system. For example, the data is distributed across 10 datacenters, and every object is available in one datacenter without requiring slow costly network roundtrips, but if that datacenter becomes unavailable it can be recovered from a combination of the other 9.
- rakoo 10y agoI'm not aware of any company doing that (I know backblaze does that but at the pod level, not at the datacenter level... because they only have one) You may be interested in Tahoe-LAFS though (https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs). It has many good things in it, one of them is that all files get erasure-encoded so that k nodes out of n are needed to restore the file. When you set a node to be a storage provider (such as S3, GCS, ...), then you effectively have erasure encoding over providers: If S3 is down, you can still retrieve your data from the rest of the providers. Some people have actually tried to do it, as described in the first section in (https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TipsTricks https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TipsTricks).
- Nition 10y agoReally interesting that (if I'm reading the article right) you can take an already-compresed JPEG, recompress it losslessly using another technique to get better compression than the original compressed JPEG, and then decompress it to the original JPEG again. The concept makes sense but I'd never thought of that before.
- Niksko 10y agoNo, I think you've misunderstood. What they're saying is that you can take a JPEG compressed image, decompress to raw pixels, and then recompress with JPEG more efficiently (if you're careful, no specifics on how this is done), and you save space. That's why they mention that they're doing it very carefully, because you've got to make sure that when you decompress the new optimized image that it pixel for pixel matches the original decompressed image.
- deleted 10y ago[deleted]
- gwern 10y agoIt could be either. For example, I believe zpaq or 7z or one of the super-cutting edge compression tools actually does compress JPGs losslessly and provide you bit-for-bit the same binary file you put in (but saving 10-20% inside the archive).
- camtarn 10y agoNition is correct, I think. As far as I can tell, 'raw pixels' is not quite accurate. Both PackJPG and Lepton seem to decompress as far as frequency-domain coefficients, but no further. That means they don't repeat the lossy stages in JPEG encoding - the transformation of pixel data to the frequency domain, and the discarding of high frequency infomation. It looks like PackJPG and Lepton do some interesting tricks with the coefficients to essentially make them more amenable to the following lossless compression. Both PackJPG and Lepton have their own file format, so they're definitely not outputting standard JPEG images. Here's the Lepton page: https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/ https://blogs.dropbox.com/tech/2016/07/lepton-image-compress... And for more detail, here's the paper that PackJPG is based on: http://www.elektronik.htw-aalen.de/packjpg/_notes/PCS2007_PJPG_paper_final.pdf http://www.elektronik.htw-aalen.de/packjpg/_notes/PCS2007_PJ...
- tiffanyh 10y agoTL;DR; the longtail (rarely accessed images) becomes really expensive. So to save storage (and thus cost), both highly compress and dynamically generate rarely accessed images.
- bbcbasic 10y agoI always wondered if YouTube hates people who put lots of unlisted family videos for friends that get say 10 or so views. It's costing in them storage but with virtually no ad revenue.
- digler999 10y agodisks have hit 10tb per 3.5" disk, which means you could have petabytes in each rack. I think we're at the point now where the materials are almost "free" compared to the amount of information you can fit on them. Plus, they mine the shit out of that data. Even if they're not earning ad revenue on it, they're tracking location, usage statistics, I wouldn't be surprised if their user agreement includes the ability to have AI "watch" the video and try to mine data about what's happening in the video the same way they mine your email to build smarter and more effective consumer profiles about you. So even with nobody watching your video, I wouldn't put it past the "G-men" to find a way to eke some profit off it.
- niftich 10y agoIt does. Our automated systems analyze your content (including emails) to provide you personally relevant product features, such as customized search results, tailored advertising, and spam and malware detection. This analysis occurs as the content is sent, received, and when it is stored. [1] [1] https://www.google.com/policies/terms/ https://www.google.com/policies/terms/
- nikofeyn 10y agoi (dis)like how it spins it as a benefit to you.
- omarforgotpwd 10y agoWell sure, it's not so hard to go a year without adding another byte of storage if you're Flickr ;)... let's see Instagram or Facebook do it. Are people even still uploading things to Flickr?
- swrobel 10y agoThought I was the only one who found this really sad
- ascorbic 10y agoThe article says that on a busy day they'll have 25 million photos, 80 terabytes uploaded. Not Facebook scale, but then nor is anything.
- laurentdc 10y agoI think it's a different demographic. Photographers still use and make projects or series on stuff like Flickr or Behance. Instagram and Facebook are more "blog-like" and aimed at snapshots of daily life, holidays etc to share with friends. It's normal imho that these are much higher in volume.
- gdulli 10y agoClicking the link and reading can answer that question and fix your lack of object permanence regarding Flickr's existence.
- random_upvoter 10y agoFor an image junkie like me Flickr is still my favorite internet black hole, after all these years. Nothing comes close to it. No matter how much time you have spent on Flickr, you will always find some wonderful collection of images you have never seen before, on any subject imaginable. I really dread that some day Yahoo will manage to casually destroy it.
- sean_patel 10y ago> get turned over to Tech Support in India How do you know they are in India? Accent? Asking because India offshore support peeps used by Dell, Walmart etc are all give white christian names - like Mary, John, Adam etc - and also undergo 3 months of rigorous 'American Accent' training. I know because 2 of my Indian cousins ( I am American-Indian born and raised here) work at such call centers in Mumboi and Chenna respectively. So it's quite difficult to discern that they are Indian cos the Companies that hire them spend millions of $ trying to disguise their voice and tone to make them sound like they are local / American.
- josh64 10y agoI'm Australian but even I can tell the accents are fake for the most part. Also, the terrible VoIP quality is a dead giveaway. edit: I should mention they use "American" accents for Australian customers too.
- krisroadruck 10y agobe a whole lot cooler if they'd spend those millions just hiring Americans in the first place. 3 months of voice training isn't going to trump having english as your first/primary language your whole life, and those voip connections are the worst.
- t0mas88 10y agoSeriously? I would bet that there are more Indians speaking English and willing to work a call center job than Americans. The whole "Hire Americans!!1!" kind of viewpoint smells like Trump and honestly: The world deserves better.
- Dylan16807 10y agoIt's not about nationalism. The job is talking to people, so it's important to be mutually intelligible. Willingness to work saves the company money. It doesn't help the user. What helps the user are things like clear speaking and good phone connections. It's possible to do that in any country, but the relative cost varies, and outsourcing implies cutting corners.
- pmlnr 10y agoOptimization done right. After a certain point, datacenter growth (both physically and logically) gets so brutal that you need to consider running things more efficiently.
- imaginenore 10y agoThey just postponed the inevitable. You can't optimize significantly forever. They will have to start getting more storage soon.
- wpietri 10y agoEvery meal you eat is postponing the inevitable. But I imagine you still think it's worth doing.
- zaroth 10y agoIndeed. They spent ~$1m of engineering talent and took on some significant technical debt in order to save recurring annual datacenter cost. I'd say if they reduced their datacenter annual cost by at least $5m it was money well spent. From the sound of it they actually saved a lot more than that. And by increasing their overall storage efficiency it's a feature that pays dividends even when they do ultimately add more storage in the future.
- Sargos 10y agoThis is fascinating stuff. I would love to see what the Google Photos team is doing in detail (but that will likely never happen).
- ec109685 10y agoGoogle uses lossy compression on its images, so it has more flexibility than Flickr does with regards to optimizations.
- londons_explore 10y agoThey likley still store the originals though, even though they don't make them available for download. YouTube does that, so that when a new video compression format comes along, they can transcode into it.
- webmaven 10y agoGoogle uses lossy compression on it's unlimited free "high quality" image storage[0]. Storing original uncompressed images eats into the 15GB free storage budget, and uses paid storage upgrades after that. Unless you have a Pixel phone (free unlimited original uncompressed image storage). [0] https://support.google.com/photos/answer/6220791 https://support.google.com/photos/answer/6220791
- irfanka 10y agoSo Indians are not "actual humans"?! Wow...
- paulddraper 10y agoI'd call Indians actual humans, but the second part "to get access to my account" might not have happened.
- pyre 10y agoWhile the wording may not have been the best, the "Indian call centre" is just a place where people have been turned into drones (these exist in the US too, but the fact that it's been out-sourced to India reflects more on the "cost-cutting" mentality around support that the company itself has). They have a script to follow and not much agency to do anything (and possible penalties for escalating issues rather than "resolving" them at the first tier of support).
- the_duke 10y agoTo be fair, most issues that call centers deal with are rather simplistic, and don't really justify employing highly qualified people there. Apart from those not being interested in that job anyway. Of course I'm mad too when I have to deal with a clueless call center rep, especially when they don't move you up to higher tier support even if they have no idea how to handle the issue. But as long as they do that, I'm fine with the concept.
- pyre 10y agoThe problem is that every issue that is escalated costs more money to resolve, so there is pressure from managers to make sure that the number of escalated calls is minimal. The reality is that this is a horrible metric to go by. The issue that a caller wants resolved it what dictates the level of support that they need, and the company really has no control over the volume of calls that require more than tier 1 support (other than building a decent product, I guess).
- dang 10y ago
- harryf 10y ago> Peter Norby, Teja Komma, Shijo Joy and Bei Wu formed the core team for our zero-storage-budget project. Many others assisted the effort. Looks like someone's hoping to get hired
- siavosh 10y agoI hope these engineers get a few million $ bonuses.
- photogrammetry 10y agoI'm sort of surprised to see you're getting downvoted by all the greedy wanna-be zillionaire founders. If they had any brains, they'd be the ones developing this compression code and getting the million-dollar bonuses.
- siavosh 10y agoHah yeah I was surprised by the downvotes. My point was that if you were in finance, these sorts of numbers generate massive bonus'...but not in other fields sadly.
- Rebelgecko 10y agoIs anyone else confused by the baseline thumbnails and current thumbnails bar graph? I'm not really sure what it is measuring
- ascorbic 10y agoThe file sizes of the different thumbnail sizes. Baseline shows all the different sizes they used before, and current thumbnails shows the space saved by switching to just two thumbnail sizes.
- lolc 10y agoThe way I read it is that they eliminated intermediate thumbnail sizes. So the graph shows the before and after of storage space used by generated thumbnails.
- thisnotmyacc 10y agoThey chose really poor colour gradients to show the differences. The image would have been clearer if it used more than purple and what looks like the most minor variation on light blue.
- gist 10y ago> We’ve pencilled out a model where we place one copy on a slower, but underutilized, tape-based system while leaving the other on disk. Store images on tape? What about degradation of the tape overtime? Certainly seems to be a factor compared to hard drive degradation.
- JaggedJax 10y agoI have no personal experience, but think an LTO tape would last longer than a HDD for long term storage. LTO tapes are rated to 30 years of storage and I doubt anyone would be using an HDD to store data that long.
- londons_explore 10y agoWith drives, you are typically checking every few hours that data is readable. If it becomes unavailable, you create a new replica right away. With tapes, even though they might last longer, you typically wouldn't scan and check the data as regularly. That means, if it does go unavailable, there is a larger window of time for other replicas to fail in before re-replication completes.
- Insanity 10y agoThis is really interesting to read! Also quite surprised by some of the statistics, I had no clue Flickr still saw this much activity.
- amelius 10y ago> "There are several accepted resize algorithms, but to retain the Flickr “look”, we implemented the same Lanczos resize and kernel sharpening algorithms that we’ve used for years in CUDA." How exactly is the Flickr "look" defined?
- bahro 10y agoFlickr's resized images have a characteristically (over)sharpened appearance.
- amelius 10y agoDoes that mean that any artistic depth-of-field effect is (partially) eliminated by the Flickr look?
- roywiggins 10y agoOrdinary image sharpening tends to just make existing edges look sharper; dof blurriness obliterates the edges. Strong sharpening on a strongly DoF-blurred image might actually make the visual contrast between in-focus and out-of-focus areas stronger.
- WhitneyLand 10y ago- h.265 would help a lot, eventually this (or equivalent patent free codec) will be mainstream for storage and devices. - They say cost is 0.03/GB? Doesn't backblaze b2 charge 0.005/GB? Why isn't B2 a real option?
- adventured 10y agoCan B2 reliably handle something the size of Flickr, with its consumer access demands (vs an enterprise/business focused service with far lower requests and bandwidth usage), as opposed to knowing for certain that AWS can? That has to be a rather dramatic consideration for a service the scale of Flickr. It's a near certainty that AWS isn't going away for the next decade at least; the same cannot be said with even remotely the same confidence for Backblaze. It's the difference between 99% and 90% confidence that your provider is going to still be around and offering a high quality service ten years out.
- toomuchtodo 10y agoIts very likely Backblaze is going to outlive Flickr.
- imaginenore 10y agoThey said S3 was $0.03 per GB, not their own storage.
- deleted 10y ago[deleted]
- detaro 10y agoS3 is an example the reader might be familiar with, they use their own storage and haven't published what that costs for them as far as I know.
- deleted 10y ago[deleted]
- malisper 10y agoFor a less ad hoc approach to reducing storage costs, I suggest looking into the ZFS filesystem. Compression is completely transparent in ZFS. Once you enable compression in ZFS, all of your files will automatically be compressed when written, and decompressed when read. I am currently managing a Postgres cluster with a petabyte of data in it. We found ZFS to be a great way to reduce overall storage costs. We just switched our machines to machines running ZFS, and we were suddenly using 1/3rd the amount of disk space. Although it took us a while to learn all of the gotchas of ZFS, it wound up saving us a huge amount of $$$. (As I understand it, ZFS would not have helped in Flickr's case. Since JPEGs are already compressed, ZFS would not have provided any benefit. Flickr was able to save storage by using an ad hoc compression algorithm.)
- devty 10y agoThe article suggested that they had to carefully balance the processing cost of compressing and decompressing data and the storage cost. Did the use of ZFS in your system incur noticeable processing cost in your system? Any noticeable increase in latency on your system?
- malisper 10y agoWhen we first switched to ZFS, we had some ingestion issues. It turned out the problems were caused by poor architecture decisions we made and was merely exaggerated by ZFS. So yes, there was a noticeable increase in latency, but ZFS was only a tiny piece of it.
- brokenmachine 10y agoHow much RAM do you need for a Petabyte ZFS cluster? I investigated ZFS for my home server and it was recommended to have 1Gb RAM for every Tb, and much more if you enabled deduplication/compression (I forget which).
- olavgg 10y agoThe 1GB ram per TB is wrong. More memory definitely helps, especially if you add an L2ARC cache which can use a lot of memory just for mapping. But in most cases you are probably fine with just 1-2GB of ram. I would never use deduplication on ZFS, it is very slow even if you have enough RAM. And in most cases, savings are less than 10%.
- devty 10y agoIs there a name for an exploit where a malicious client requests rarely-accessed contents that has been tucked/compressed away in order to overwork their server?
- niftich 10y agoDenial-of-service
- lobster_johnson 10y agoCache pollution. https://en.m.wikipedia.org/wiki/Cache_pollution https://en.m.wikipedia.org/wiki/Cache_pollution
- londons_explore 10y agoMost large web services today would die a horrible death if someone pulled off an effective cache pollution attack. For example, if a piece of client side malware, for every Vimeo video a user views, loads in the background a random rarely viewed video, it would probably lead to days of downtime.
- natch 10y agoAnother way to reach this goal is to make sure features of the service remain unfriendly to users. Case in point, when providing a search interface, only return as many results as you want to, not as many as you have. And don't bother coalescing spammy results all from the same account into one expandable item; instead let them flood the results because they used keyword tag spam, and then cut off the search results after a few pages. Make it hard to navigate by hiding everything behind hashes, to prevent fair use downloads. Keep tags in beta for 15+ years. Of course, when usage goes down, that helps with the problem quite a bit. A poor experience, even for viewing content, lessens engagement and leads to lower usage and fewer uploads. Sadly, I'm afraid a much more extreme data storage reduction approach awaits faithful users of Flickr. When Yahoo! bought a large photo blogging site in Taiwan, it simply shut it down with about six months notice, deleting everything as it did.
- superic 10y ago> Case in point, when providing a search interface, only return as many results as you want to, not as many as you have. Do you have an example of this? Seems like a bad way to run search. > instead let them flood the results because they used keyword tag spam Tag spam largely doesn't work but it's not impossible. Flagging these results makes them go away and brings them to our attention. > Make it hard to navigate by hiding everything behind hashes, to prevent fair use downloads. Huh? > Keep tags in beta for 15+ years. Double-plus huh? Flickr is only 13ish years old. It's not ready to get a drivers license yet. > Sadly, I'm afraid a much more extreme data storage reduction approach awaits faithful users of Flickr. Nope! Unless you know something I don't? > When Yahoo! bought a large photo blogging site in Taiwan, it simply shut it down with about six months notice, deleting everything as it did. That sucks. Which one?
- natch 10y ago無名小站 (wretch.cc)
- rocky1138 10y agoRegarding the lossless JPG compression change: the review strategy... was that done manually by eye or automatically using some sort of image comparison library?
- tehlike 10y agoi wonder if they considered decompressing on the client. not that it's a great way especially on mobile, but i was curious how numbers played out :)
- ComodoHacker 10y ago>On a very high-traffic day, Flickr users upload as many as twenty-five million photos. These photos require an average of 3.25 megabytes of storage each, totalling over 80 terabytes of data. >increasing camera resolution, burst mode and the addition of short animations (Live Photos) have increased bytes-per-image rapidly enough >Users only rarely delete or change images once uploaded. I'm very curious, how much of all this tr.. sorry, sweet memories are never ever viewed after, say one week from upload.
- TeeWEE 10y agoFlickr SUCKS donkeyballs. They have a non existing support team. And disabled all login mechanisms except yahoo. Im locked out of my account, and cant delete my pictures anymore.
- devwastaken 10y agotheir OEmbed endpoint isn't very good, either. Seems they'l be packing up someday.
- ideonexus 10y agoOne area that may be impacted by this strategy is bulk-downloading your photos. My wife set her phone to automatically backup all her photos to flickr. Thinking she could get them back at anytime, she started deleting them off her phone. After three months, I discovered she was doing this and begged her to also backup her files to an external hard drive. Every year we sync all our family photos to have redundant backups. When she went to get the three months of backups from flickr she got "download error" after error. She sent me this link and hypothesized that the bulk-download feature is no longer working because of the need to now first decompress the files before transmitting them. Luckily, she was able to get the 25 gigs of family photos down using a third-party application, but it's another reminder to never wholly trust the "cloud."
- NEDM64 10y agoThat's how it's free.