5 ms·
The reason they did that was so that they could build up data on the effect of each factor independently. That way, when the bigwigs get into a meeting and dis
by nostrademons 11y ago
The reason they did that was so that they could build up data on the effect of each factor independently. That way, when the bigwigs get into a meeting and discuss alternative designs, they can predict "Ok, this will have $X effect on revenue but will decrease search latency by Y ms, which itself will have +$X effect on revenue."
I was the first engineer on the first visual redesign that tried to change everything about the page [1]. One of our biggest problems was that since we tried to change everything at once, the metrics went haywire, and we couldn't know if a change was because we introduced a left-nav (okay, actually we kinda could, because we'd experimented with a left-nav on its own beforehand) or because we changed the line spacing or because we cleaned up the logo or because we added icons. There was effectively no way of "debugging" the user; when user behavior varied from expected, we didn't know why.
I've got plenty of other stories from that era, but they aren't really fit for a public forum. Happy to discuss privately. (Kinda ironic when the non-Googler says that to the Googler...)
[1] http://googleblog.blogspot.com/2010/05/spring-metamorphosis-googles-new-look.html http://googleblog.blogspot.com/2010/05/spring-metamorphosis-...
- pasbesoin 11y agoCan you offer any perspective on the whole sometimes-named "mystery meat" UI that's been rolled particularly but not only into mobile design over the past half decade? Further, more and more of this seems to have been "backported" into traditional desktop presentation design. While I will experiment around until I more or less "get it" (although I may miss finding some useful features for months), it particularly frustrates people like my parents. Mom is somewhat spatially challenged (although plenty bright in other ways), and I've had numerous calls with her where the controls for what she wants to accomplish have changed and simply bewilder her. AND, given her spatial orientation problems, sometimes we will have to have the same conversation several times -- because the rather arbitrary-appearing icons and patterns don't immediately "sink in". P.S. I mean this as a legitimate question, if it fits the parent's -- or others' -- field of experience. It seems that a lot of this design is expected to simply permeate and be picked up by osmosis. Is that really the case?
- nostrademons 11y agoI'm curious what you mean by "mystery meat"? I assume it's the use of icons and drawers to trigger functionality rather than labels and links? That's driven by screen size - there's simply not enough screen real estate on a phone to label every button with words. It does have a legit discoverability problem, which most folks in UX acknowledge but don't know how to fix, given other constraints of the media. I've seen a bunch of mobile apps with help overlays or tour videos the first time you run them, which can help a lot. It gets back-ported to desktop apps because once people do learn the functionality, it leads to a cleaner, less cluttered UI. I'll also point out that early PCs had the same problem. A typical PC from the early 1980s had 320x200 resolution, a little less than a smartwatch does today. For most of the 80s and early 90s, the standard was VGA (640x480), slightly smaller than most smartphones today. Icons were really common back then as well, they weren't all that discoverable, but at least people read manuals occasionally. Most folks don't bother now. Menus were another response to that problem, but they don't really work on mobile because the tap target is too small to efficiently hit it (the navigation drawer & hamburger menu is the modern-day equivalent of the menu). The web - with its inherent discoverability and all of its space for explanatory text - didn't really become popular until 1280x1024 displays became widespread. I remember designing webapps for 800x600 screens; it was painful, and they had discoverability problems too.
- pasbesoin 11y agoWell, a, um, "literal" example was the initial occurrence of what I've just learned is called the "hamburger" icon/menu. Which was "backported" to Google's desktop designs. A particular instance I recall was changes to the Calendar web page presentation. Mom initially struggled a fair amount with the controls in the left bar for selecting which calendars were displayed. This was in a Google Apps environment with 20-ish people participating. I remember when screen presentations could be somewhat cryptic -- per your description as well as other instances. But, as you point out, there was usually a manual. Now? Often it seems to be down to guessing and poking around. Or asking friends. But if you are a bit older and not constantly talking over such things with your friends, that means of edification is somewhat starved. P.S. Thanks for the response. And as I said, I'm curious about the perspective of someone on "the other side". I'm not trying to be unduly critical, here; just, this is an issue I've consistently bumped up against, from my own perspective.
- msellout 11y agoI'm surprised no one on the team was familiar with multivariate analysis. That's the wrong way to say it. Let me rephrase more carefully... I'm surprised no one used more sophisticated analysis techniques to allow an experiment that changes many explanatory factors simultaneously and estimates the effects of not only each factor independently but also their interactions. It's relatively straightforward statistics and would have been taught to clever undergrads or first-year grad students who specialized in math, economics, machine learning, industrial engineering, etc.
- nostrademons 11y agoThere were stats Ph.Ds on the team that were certainly familiar with it, and I knew at least one former Amazonian manager that held that we were too afraid of having multiple experiments running at once and too afraid of confounding factors. (Amazon's A/B testing is much more traditional split-testing, where they show one variant to half the population and one to another half and then pick the best one.) The biggest problem is that many of the folks involved had encountered very surprising and counter-intuitive interactions between features, and because of that, they didn't trust the textbook models. For example, at one point we had a very convenient embedded-Python experimentation framework that let us rapidly try things out. Our main data-scientist on the project decreed that we were not to use it, because a number of the externalities that framework introduced - higher latency, and potential error conditions - could confound the metrics in ways that were not well-understood at the time. Data is useless unless you can trust that it actually represents reality. Stats textbooks all make certain assumptions about reality when they present the models, and if reality doesn't conform to the assumptions, the models won't conform to reality. Certain factors were well-studied enough that we could build useful models from them - that was the point of all the experimentation around color, after all. (I wonder if those conclusions still hold, anyway: data has an expiration date, and I recall personally running an experiment that directly contradicted an experiment Marissa Mayer had run 6-8 years previously.) But that requires a lot of hypothesis testing and verification: it's never a matter of "well, the model says so, therefore it must be", but rather "well, the model says so, let's run an experiment to see if the model's predictions line up with reality, and if they do, we have a pretty strong - but not ironclad - indication that we can use it for future calculations." tl;dr: Largest and best-funded social science program I've ever seen.
- seanmcdirmid 11y ago> One of our biggest problems was that since we tried to change everything at once, the metrics went haywire Isn't that the problem with empirical methods in general? They are great for measuring the effect of small simple things (like small changes to a web page), but are much less effective in measuring bigger things (like say, programming language A vs. programming language B).
- nostrademons 11y agoAbsolutely. Business is an interesting field because you derive a fairly large advantage from being correct about reality, but that advantage counts for nothing if you're so slow that somebody else changes the reality while you're figuring it out.
- lukestevens 11y agoThat is a very succinct way of putting it, thank you.
- meatysnapper 11y agoDesign by AB testing, my favorite :( I worked at a major tech company where every decision needed to be justified by AB testing. Our design was a series of haphazard guesses made on data that depended on a very brittle data pipeline. Needless to say, nobody thought it was a very good product, but the incredible bureaucracy and people behind the approach made it hard to fight against.
- meerita 11y agoIt shouldn't be a problem. In science, everything must be tested and proved if you want to know the truth or the effectiveness of an theory. Design is a science, so it can be measured and developed by tests of all kinds, even emotionals.
- ulfw 11y agoThat's a pretty one-sided statement. One could easily argue the opposite, that good design is an art form, not a science.
- meerita 11y agoArt is a science too. One different thing is that some artists don't care about the facts. Everything can be explained with numbers.
- jhanschoo 11y agoWhereas a design studio with sufficient resources should fare better with a data-driven approach, a sole designer or small team with time and feedback constraints will probably do better with traditional methodology. Traditional methodologies might have certain quack dogmatic rules and processes, but it offers a battle-tested baseline.
- deleted 11y ago[deleted]
- jacquesm 11y agoI understand the urge to try to maximize on stuff like this, but in the end it loses sight of what's most important: the quality of the results. And sometimes I feel that a lot of valuable engineering time and effort is wasted on making things prettier and better monetizing in the short term when in the longer term the better search results are what would really move the needle. And most A/B testing focuses on the very short time effects of the change (and change all by itself, even change for the worse can have a short term positive effect just because something is new). On another note: A/B testing has another limitation, which is that you're optimizing for the bulk, please one person more and you more or less automatically annoy another.
- nostrademons 11y agoThere are plenty of talented people working on ranking and webspam too. There's a limit to how parallelizable those problems are. At some point, more people working on an algorithm doesn't make it faster or better, it just makes it more confusing and worse. Mythical Man Month applies as much to Google as it did to IBM. And so you might as well put people onto optimizing the rest of the page as well, because they'll do more good there than having yet another cook in the ranking algorithm kitchen. (I'd argue that there are too many people working at Google in general...I left, so I'm no longer contributing to that problem. But then, there were probably too many people when I joined, so I contributed to it for 5 years.)
- bad_user 11y agoThe worst limitation of A/B testing is that it's not producing anything better than a local maxima. Nature does indeed work like that, but it took millions of years along with random mutations and we don't have that much time. And if in our thinking process we would have relied on optimization algorithms favoring the local maxima, then Jazz wouldn't have been produced, amongst others.