13 ms·
Intern Impact: Brotli compression for Play Store app downloads
- painted 10y agoSo she used a compression algorithm developed by other googlers? So what? Don't get me wrong, I'm sure she did a lot of work for it, but looks like a lot of people would have been able to do that, there is nothing innovative in what she did, right?
- sp332 10y agoIt's not innovative, and yet it had a large impact on Google and their customers. Innovation isn't everything.
- jeron 10y agosometimes just applying the same thing in a different place can make a huge difference
- got2surf 10y agoIt's easy for people (especially people who like tech) to forget this. "Innovation" in the sense of better algorithms or improved computations are great, but they don't solve problems - applications do. That's why many 'hot' areas (like machine learning, for example) have a large gap between industry and academia. Researchers focus on "innovation" in terms of new approaches, aiming for marginally higher performance, and requiring increasingly complex models/infrastructure to unlock the remaining performance. On the other hand, combining an existing technology with industry knowledge of how to apply it can have a huge impact.
- geodel 10y agoI think it is mostly young IT worker recruitment ad, and also a 'Women in technology' angle. I personally wouldn't be too happy if my modest contributions as novice described as 'massive improvement for millions of our users'.
- recursive 10y agoHow about a modest improvement for millions of users?
- softawre 10y agoYou wouldn't be happy if somebody gave you a virtual high five for your work when you were early career?
- GunlogAlm 10y agoI'd be happy, but happier if it came without the hyperbole (personally).
- geodel 10y agoI guess I remain critical of my efforts. For example, I would not call my sneaky effort to use old veggies and other food items in refrigerator as massive cooking innovation to keep the planet green.
- Aeolun 10y agoI'd be happy to put, "Implemented compression alghorith that saves the company 1.5PB in data transfer a day." on my CV. That's $15000 a day saved on data traffic. You are worth $5M+ per year now (at the same scale).
- Ar-Curunir 10y ago
- sidcool 10y agoAfter any idea is implemented, there are people who say the same. Coz most ideas are out there, they need dedicated implementation.
- rokhayakebe 10y agoHonestly genuine question: What did you accomplish that was technically as significant by her age/experience?
- NetStrikeForce 10y ago> So she used a compression algorithm developed by other googlers? Yes. > So what? So she saved 1.5PB per day - which no other Googler did. Nor you or me. She did. > Don't get me wrong, I'm sure she did a lot of work for it, but looks like a lot of people would have been able to do that, there is nothing innovative in what she did, right? If it was that easy, why wasn't done already? Why did she have to come along to do it? You are confusing easy to implement with easy to come up with the idea. In hindsight it is a no-brainer, but somehow the almighty Google didn't do in years of Play Store. Go figure, right? P.S.: Please support your downvotes with a reply.
- ng12 10y agoWhat makes you think she came up with the idea? Read the article -- it's clear that it was her "project". She's an intern at Google, they're generally on a short leash.
- rocky1138 10y ago> they're generally on a short leash How do you know that?
- ng12 10y agoMany friends who had gone there. It was one of the reasons I turned them down. It's just part of being a large tech company -- they get hordes of interns and generally want to get something out of them in a short period of time. Most are explicitly attached to one or two projects which can be completed in ~2 months.
- NetStrikeForce 10y ago> What makes you think she came up with the idea? It might have not been her sole idea or even her idea, I'll give you that. However there's no evidence it was not her idea of partly her idea. > Read the article Please, this is the kind of comment not welcomed here. I did read the article before commenting. You are trying very hard to read between the lines, when Google is claiming once and again that her work saved 1.5PB/day - I'm not sure why will you try so hard to read a hidden message in Google's words and dismiss what's written in plain English. > it's clear that it was her "project". So what? See my previous paragraph. > She's an intern at Google, they're generally on a short leash. Generally or always? In the US or in Europe? It was the second time she was interning at Google (it's in the article). Again, you are trying very hard to read between the lines and guess. This engineer did something that saved users 1.5PB/day. She did, no one else did. Other people did other stuff. To each their own - I'm not sure why do we have to downplay her achievement though. P.S.: This is the kind of shit women have to put up with constantly. It might look like it's not important, but when you have to go through this every day it takes a toll on you. The worst part is that we don't even want to acknowledge it.
- danellis 10y agoYou're missing the point of the blog post, which is "Hey, students, Google deals with massive amounts of data, so even as an intern you can have a huge impact."
- sp332 10y agoThis page is consistently crashing my Firefox content process. I'm running 51.0.1 64-bit on Win10. Anyone else having this problem?
- SnaKeZ 10y agoFirefox 51.0.1 (64 bit) Ubuntu works ok.
- snuxoll 10y ago51.0.1 on Fedora 25, no issues here.
- gcp 10y agoLook in about:crashes, you can click the links and see if there is already a bug filed for it. Given that you're on release and this is a Google blog rather than an obscure site, it's unlikely to be a problem in Firefox itself. Check gfx drivers, plugins, etc.
- demonshalo 10y agoSo she did this for free? :D please tell me this is a paid internship!
- akhilcacharya 10y ago.... I hope the rock you live under is heated, they're paid $6.6K a month + 9K-12K housing in the US.
- deleted 10y ago[deleted]
- ThePawnBreak 10y ago2700 GBP a month, plus 1000 GBP housing (for a summer). Because Europe.
- gambiting 10y agoShe started in Krakow, Poland. 2700 GBP/month is what the prime minister of Poland makes, it's an absurdly high salary for the local market there. I've friends who work as professional C++ programmers there and they make about 1000-1500 GBP/month working for huge corporations.
- ThePawnBreak 10y agoShe worked on something completely different in Krakow, and earned a lot less (about 1000 euro I think). The compression work was in London, where 2700 GBP doesn't go a long way at all. Rent is about 1000 GBP for a room, and 2700 GBP is about 1800 net, so that leaves 800 GBP per month to spend on food, transportation and other expenditures. I happen to know the person in the article. I was also an intern at Google this summer in Europe.
- gurgus 10y agoFor an intern, that is insanely good. edit: In fact, for a lot of people, that is insanely good! Apologies if I came across as an asshole when I said "for an intern".
- Swizec 10y agoPied Piper?
- okreallywtf 10y agoThere isn't much information but this reads more like an advertisement for google internships than anything else. Not to denigrate her work, she could very well be brilliant and have gone above and beyond, but from how it reads they could be blowing it up to make it seem like every intern has a huge impact and you could too! Either way good for her, but not sure why this is so high up on HN.
- bobdole1234 10y agoIt's almost like Google might be recruiting.
- dmoy 10y agoActually it is odd timing considering intern recruiting is slowing way down for most companies around now.
- ljk 10y agothere's always next year
- ChuckMcM 10y agoI suspect this is exactly correct. First, it plays to a lot of colleges who are having career fairs this month as their seniors consider what to do after graduation. Second it plays to what has been the biggest job selector for new graduates over the last 5 years, "Will I be able to make an impact?". The number one way to discourage people from going to BigCo is to point out that they are going to spend 3 - 5 years just getting enough of a political network developed in order to be given a chance at an assignment that might move the needle slightly. Last year I was walking around a recruiting event at CU and just listening to the pitches being made by companies Facebook, Microsoft, Google, and LinkedIn (and in full disclosure was making the same sort of pitches for IBM with "Hey its the Watson group, its the most important company project and we're building one of the most important projects in it!" all to counter the fact that it was hundreds of thousands of employees, most of which were not having much impact at all. You need stories where you can pitch them "if you're brilliant enough and work hard enough you can be the person who changes the world." otherwise the siren song of startups of "You will probably eat lunch with the CEO at least once a month, and everything you do will be really impactful." will carry the day.
- 4twilight 10y agoWhy she doesn't wear a Pied Piper t-shirt? However, I'm more interested if the Erick's position is still vacant in the venture? (Jin Yang's will work as well for first 2 years, I suppose).
- Syzygies 10y agoCan we get her to work for DropBox? Every time my iPad GoodReader syncs my 1,000+ papers, it has to check every file. It boggles the mind that they don't support some version of change records.
- falloutx 10y agoI don't whats the config of your computer, but Dropbox works like a charm for me. I have more than 600 gigs of data synced btwn Dropbox and my computer and it works pretty nicely. I never had to manually check whether a file has been transferred or not.
- tln 10y agoMaybe the solution should start with GoodReader? I don't have syncing issues with the core app.
- orliesaurus 10y agoSo wait, if I understand this article correctly she applied the compression because someone told her to? Or did she research herself and apllied the whole thing? I agree this is a bit like a "we re hiring interns" post
- arenaninja 10y agoI would be excited if I had taken an internship and someone had told me to do this :)
- orliesaurus 10y agoyeah sure nothing wrong with that - it's just that the article is a tad clickbait
- jonatron 10y ago1.5M GB = 1.5 PB?
- jdcarter 10y ago> her work resulted in saving users an expected 1.5 petabytes (that's 1.5 million gigabytes) of data each day. I'm guessing this is not a measure of data at rest, but data transferred over the network. The couple samples listed on the page ranged from 2.5% improvement to 20.3% (vs. zLib) so I guess they're extrapolating that out to all app downloads and updates across the world. Nicely done. More generally, we've seen some great advances in compression lately. I've been using Facebook's zStandard [1] for compression in a product I'm currently working on, and I've been extremely pleased with both its speed and compression ratio. The days of "just use zLib" are coming to a close. [1]: https://github.com/facebook/zstd https://github.com/facebook/zstd
- rdtsc 10y agoAre you worried at all about their patents stance. I currently I think it says if you litigate with Facebook you lose the license. Otherwise I agree zstd is looking like a very nice improvement in an area where most people think nothing happens. I especially dictionary compression bit.
- jdcarter 10y agoThat's a good point, the downside of using new algorithms is playing by the rules of their patent licenses. In this situation I'm not concerned about litigation with Facebook, but I could foresee other products where that might be an issue. Aside: anywhere you store compressed data at rest, you need to store (in a header somewhere) the algorithm used to compress that data. If you need to change algorithms down the road--e.g. you do get into a lawsuit with Facebook--you'll need that header to know how to decompress old data vs. new.
- jontro 10y agoLooks like they're using the BSD license in this project, https://github.com/facebook/zstd/blob/dev/LICENSE https://github.com/facebook/zstd/blob/dev/LICENSE So no need to worry about the patent clause in this case right?
- 10y ago
- diimdeep 10y agoClick bait.
- adtac 10y agoNo it's not. That's literally what I would summarise the entire article as: Google Intern's work saves over 1.5M GB per day.
- PedroBatista 10y agoPlease fix the title, its 1.5 PB, not GB
- sp332 10y agoIt says 1.5M GB == 1,5000,000 GB == 1.5 PB
- fredsted 10y ago1.5 megagigabytes.
- rocky1138 10y agoIt's still a stupid way to write it.
- Someone1234 10y agoThat's confusing. To use an analogy, imagine if someone wrote: "$1.5M B" instead of 1.5 quadrillion or 1,500,000,000,000,000. You'd be confused, and rightly so. A lot of people would mistakenly read it either as $1.5M or $1.5B, neither of which is right. In this case a lot of people are misreading it as 1.5 GB/Day instead of 1.5 PB/Day. PS - The way Google uses it in the Blog post is pretty clear, they're describing what a petabyte is. My issue is with the HN title only.
- leephillips 10y agoI'm sure I'm strange, but I find "$1.5M B" clearer than talk of "quadrillion"s, because I'm not confident I know what power of 10 a quadrillion is, but when I see M B I just add 6+9. Even then, the British sometimes say things like "thousand million", because billion over there used to mean 10^12, but now means 10^9 (which they used to call a milliard); as with other wordy number-words they've succumbed to American usage¹. ¹https://en.oxforddictionaries.com/explore/how-many-is-a-billion https://en.oxforddictionaries.com/explore/how-many-is-a-bill...
- PedroBatista 10y agoI didn't even saw that. good trick :)
- divbit 10y agoI don't know much about gigabytes, but that seems like a lot edit: (I'm guessing the downvotes are because I phrased it like a meme, but to clarify, this was a genuine compliment in response to a 'look what this person did' type post- it's inspiring stuff)
- Yuioup 10y agoMathematicians are the true programmers. I wish I was one.
- treehau5 10y agoEven mathematicians rely on abstractions every day to do their jobs. Don't sell yourself short.
- chanandler_bong 10y agohttps://xkcd.com/435/ https://xkcd.com/435/
- cdnsteve 10y agoNGINX: https://github.com/google/ngx_brotli https://github.com/google/ngx_brotli Apache: https://lyncd.com/2015/11/brotli-support-apache/ https://lyncd.com/2015/11/brotli-support-apache/ Node.js: https://hacks.mozilla.org/2015/11/better-than-gzip-compression-with-brotli/ https://hacks.mozilla.org/2015/11/better-than-gzip-compressi...
- bluedino 10y agoOn the other end of the spectrum, how much more energy has been used by the millions of Android phones uncompressing the app, applying the patch, and re-compressing the data?
- bhouston 10y agoI bet switching to LZMA would have saved even more. LZMA beats Brotli nearly every time. zStandard would likely have worked better as well. Brotli is very slow to compress.
- gsylvie 10y agoLZMA is much slower at decompress compared to Brotli. https://www.opencpu.org/posts/brotli-benchmarks/ https://www.opencpu.org/posts/brotli-benchmarks/
- adtac 10y agoApps are usually in the order of 40-50 MB. From a back of the hand calculation, the a 50 MB file when compressed with Brotli is 14.5 MB and LZMA is 17.5 MB. The difference of 3 MB translates to an additional 8 seconds of data transfer in a 3.1 Mbps 3G connection (an average one). This [1] states that a LZMA has a decompression speed of 70 MB/s, which is about 0.7 seconds. The 334 MB/s speed of Brotli does the same in about 0.15 seconds. So the additional overhead of LZMA (compared to Brotli) in decompression is just 0.65 seconds. Given these orders of magnitudes, I think optimizing for compression ratio is a much better option than decompression speed. [1] https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro...
- bhouston 10y agoIn my tests and others LZMA seems to beat Brotli by quite a bit on non-text cases (DB, 3D meshes): https://news.ycombinator.com/item?id=13589048 https://news.ycombinator.com/item?id=13589048 I am concerned that the Brotli v LZMA v GZip paper you cite is not fully representative as it is written by the the Brotli team.
- bhouston 10y agoI just installed brotli on my ubuntu machine and compared it compressing a executable. Brotli: 4.77MB -> 1.21MB (8.5s) xz: 4.77MB -> 1.129MB (1.4s) LZMA does better than Brotli in the case of binaries by a fair bit every time.
- MtL 10y agoMakes you wonder how much they'd save by using Courgette, like the Chrome team does.
- jordache 10y agoshe didn't create a compression algorithm. More akin to enabling GZIP in IIS...
- 16bytes 10y agoIf you had an intern that was responsible for turning GZIP on in IIS for a website that had 1B users it starts to become much more of an accomplishment. Even small changes at that scale require careful analysis and coordination.
- jordache 10y agotrue. However my response was more to the click baity title giving the automatic impression an intern came up with an innovative approach that netted tremendous result.
- dajohnson89 10y agoThe amount of negativity in the comments section here is astounding. How could you not be excited and happy for this promising young woman's achievement? No, her work will not put her on the shortlist for a Turing Award. But it is something any engineer should be proud of, and has real impact for millions of users. You have a right to be unimpressed, but if you're taking the time to say "So what?" or "This is just a recruiting ad" then you should probably rethink. I never thought I'd say this, but the negativity here really indicates the kind of latent discrimination that so many URMs & women in tech complain about. I have literally no other explanation for it -- a senior engineer at Google could have implemented this compression and it would still be HN worthy, and nobody would be calling the blog article a fluffy PR piece.
- whatshisface 10y agoI wouldn't call this discrimination - I don't think HN has ever treated early-career projects with a light touch. There's a curmudgeonly atmosphere in nearly all of these threads!
- tonmoy 10y agoOn the contrary, I have seen projects that change desktop wallpapers get LOTS of praise
- djrogers 10y ago> a senior engineer at Google could have implemented this compression and it would still be HN worthy, No, I doubt it would be. How many of the hundreds of little features in the google play store have been posted on HN with an article about the person who implemented them? Also, I find it more than a little presumptuous of you to assume that any scoffing is due to sexism. I see the exact same cynicism and lack of awe in the posts below that I have come to expect from HN - regardless of gender or color of the person involved.
- deleted 10y ago[deleted]
- 10y ago
- bricss 10y agoTake a look at LZ5 algorithm -> https://github.com/inikep/lz5 https://github.com/inikep/lz5
- mnml_ 10y agothats like 50 million dollars a year (in egress cost)
- jknoepfler 10y agoThe phrasing makes it clear that this is not intended to wow a tech audience. It's a Google ad to parents, or something.
- forgottenpass 10y agoI bet your hacker excellent skills cannot save 1.5 PB traffic per day. Your own shitty[0] remark highlights why your post doesn't have much of a point. The reason the impact of this change is worth mentioning at all is because google already wields the centralized control needed to effect change to such a massive volume of internet traffic on it's whims. Edit: [0] "shitty" is not a quality rating, but my assessment of the attitude embodied in your comment. Not specifically the bit I quoted, but how it fits into your assessment of contribution. You're acting "shitty" by implying the fact "everyone" can't do it is a matter of skill, while knowing full well that access is the more important factor here.
- therealmarv 10y agoAlthough you don't deserve an reply for your rudeness. Let's say I can get into a position and make some good work and it counts (good, is not it?). Let's say you think you can also make something meaningful and even better and you are currently not in the position to currently make it count for anybody (why then disrespect others people work?). This is something you are lacking: respect.
- dang 10y ago> Although you don't deserve an reply for your rudeness > something you are lacking: respect Please don't break the HN guidelines by being uncivil and making this site worse, even if someone else has behaved badly. Reacting like this creates a downward spiral.
- dang 10y ago> Your own shitty[0] remark You can't do this kind of name-calling on HN, so please don't. You've also been posting quite a few ranty comments lately. Please don't do that; we're trying to for a higher quality of discussion than that here. When hot under the collar, please cool down before posting. We detached this subthread from https://news.ycombinator.com/item?id=13581770 https://news.ycombinator.com/item?id=13581770 and marked it off-topic.
- arenaninja 10y agoPretty cool that an intern was given this level of confidence. Less data for updating/installing applications is good no matter how you slice it
- mbesto 10y agoI've worked with a fair number of people that graduated from the Mathematics and Informatics at Babeș-Bolyai University. I'm generally very impressed by them, and is just another data point of areas of that world that get overlooked.
- iamleppert 10y agoThis compression technique seems to be based on the fact they have previous installation of an app that can be diffed and patched, so it wouldn't receive any benefit from first installations, only updates. But still might be worth it for many applications. I remember I investigated a way to send and apply diffs of javascript code (using a js version of patch) and store in the browser using localstorage. However, at the time the performance wasn't good enough when compared in an end to end benchmark. However, this has got me wondering as a general corollary for application delivery...would it just make more sense to use something like a well-pruned and compact git repo, and make the connections over HTTP with gzip compression? I'm not sure how space efficient the git repo is but may seem like an interesting project. I'm wary of using any Google technology, open source or not if it can be done yourself in an afternoon. Does such a thing even exist?
- captn3m0 10y agoWhat you want is http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/ All objects in git packfiles are already compressed, so you aren't gaining much by adding another layer of compression.
- jfasi 10y agoI've seen replies about how this is a "simple library swap" and so doesn't deserve the attention it recognition it has received. As some who works at Google but not anywhere remotely near this project, but with experience in similar projects, I'd like to shed some light on why this isn't a simple library swap, and seems from far away to have been both a tremendous accomplishment and a wonderful learning experience. First off, there is no such thing as a library swap at Google. Our codebase is quite large. Like shockingly overwhelmingly large. Executing a change like this is almost certainly not a case of "swapping out one configuration line for another." It requires writing new code, testing it appropriately, updating any integration tests, updating documentation, etc. But the real fun starts when you're done coding... There's the issue of frontend and backend. Serving Brotli-compressed data is great, but what if you're app doesn't support it? If you're lucky, this will be handled by the underlying network layer but then you have to deal with... Rollout. I don't know how many servers are dedicated to app updates, but I imagine it's a lot. I also imagine they're distributed geographically, across regions and probably even continents. Getting all those servers to support new features is a delicate, time consuming process where any misstep will result in users noticing. It's not coding, but that's why it's called "software engineering" and not "coding engineering." But then once you're servers are all up and running you have to deal with... Versioning. Updating backend servers is bad enough, but at least you control them. What about that zoo of Android versions out in the wild? How do you ensure they all support this changes? Short answer: you don't. You design a strategy that will allow the rollout to happen gradually over a period of time, and closely monitor it to make sure nothing unintended is happening. Then how do you turn down the old feature? When do you turn it down? You need to build and properly use instrumentation to determine the safest time to kill off the old feature. Or you could never kill it and commit to paying the cost in perpetuity. That's a design decision, and not a trivial one. But, odds are you're not the only feature being rolled out. You have to anticipate/deal with potential interactions with other features, rollbacks of other people's work, etc. I could go on, but I think I've already demonstrated why this is by no means a trivial accomplishment, even for a full time engineer. Add to this the fact that every intern has to race against the clock to get ramped up on their project, making something of this complexity and with this large an impact happen deserves applause. I should add, I'm speaking as myself here and not representing Google in any way.
- jedc 10y ago"Google Student Blog" // "Google news and updates especially for students" Important context for this blog post and the comments in this thread.