11 ms·

Lots of folks/companies are going to lose long-term as a result of bad A/B testing driven decision making. I suspect the decline of the Facebook feed might be a

Lots of folks/companies are going to lose long-term as a result of bad A/B testing driven decision making. I suspect the decline of the Facebook feed might be attributed to this as well (maybe even the recent polls of Trump's twitter practices[1]). Generally these sorts of things don't really consider long term fatigue and other important factors, and take short term micro-benchmark wins as success. On the flip side, some attribute not A/B testing to the failure of the SnapChat redesign[2]. Is there anyway to do this properly? I can imagine the problem will get worse and worse as we reduce human evaluation of A/B test results and automatically make real-time decisions through software and ML. [1] https://www.businessinsider.com/poll-shows-trump-appears-to-be-losing-somewhat-supporters-with-antics-2018-12 https://www.businessinsider.com/poll-shows-trump-appears-to-... [2] https://www.engadget.com/2018/10/05/snap-evan-spiegel-app-redesign-rushed/ https://www.engadget.com/2018/10/05/snap-evan-spiegel-app-re...

johnday 8y ago
> Is there anyway to do this properly? The way to do it properly is to hire experienced, competent UX designers and human computer interaction specialists, who understand the problem space and how to keep people engaged without addictive malpractice / dark patterns. Metrics are useful but they are what they claim to be - measurements - and nothing more. You wouldn't choose which house to buy based on how large it was measured to be, but you would expect a surveyor to make those measurements. Speaking as a web and video game designer, I can't wait until I can hire someone to think about these things for me.
- TaylorAlexander 8y ago
  This makes sense. It’s like how clickbait headlines would, if they previously didn’t exist, improve click rates. But long term if your platform was all click bait it would fatigue the users. Or at least, one might think so...
  - alan_wade 8y ago
    Clickbait + low quality content is what fatigues users. Personally I don't mind clickbait if after doing whatever it takes to grab my attention, the creator delivers on an actually solid high quality educational/entertaining content.
    - WA 8y ago
      This concept is called: overpromise and underdeliver. The content should at least match the promise of the headline, or even overdeliver. Then readers are happy.
    - mikestew 8y ago
      Personally I don't mind clickbait if after doing whatever it takes to grab my attention, the creator delivers on an actually solid high quality Captain Pedantic, reporting for duty, and that’s not actually clickbait. If the masterful baiter of clicks actually delivers, then it wasn’t bait. I mean, what’s to complain about? Turns out there is a way to target stubborn belly fat, five minutes a day, who knew?
- pfschell 8y ago
  Designers don't cause terrible designs like this. MBAs and marketing droids do.
  - newppc 8y ago
    Pretty sure Facebook is not a company that is taken over by MBAs and marketers that are making this happen. Rather it's an engineering-first organization, run by an engineer. Probably a mix of everyone's fault.
    - dbt00 8y ago
      How is an MBA or marketer evil in a way that Zuck isn’t?
  - abakker 8y ago
    That’s a bit reductive. You may be right, but it would be less facile to say that poor designs can be made by anybody, but that a mismatch between design and business objectives led to bad results. Not all business people are morons, and not all designers are good at design.
    - ikfmpwdsoz 8y ago
      If the business objectives lead to a design so bad that it hurts the business, surely the people who came up with these objectives didn't do a very good job?
- kokokokoko 8y ago
  People are not hired for roles when a company can afford to. People are hired for roles when company cannot afford not to.
  - chaosite 8y ago
    That's an ideal. Most companies like to grow, and growth usually means hiring.
  - johnday 8y ago
    At the moment, I can't afford to and I can't afford not to. So I'm just muddling through on my own.
  - kamaal 8y ago
    >>People are hired for roles when company cannot afford not to. Only if people have a stake in the decline of a company, and have something to lose. Other wise in these days of executive networks, and people getting in their golf and dinner buddies- Execs mostly care about nothing at all. The only people to lose anything are a few low level employees and may be shareholders. Shareholders mostly don't care either. People who hold large block of shares, are generally diversified. And since nothing fails at all at once- They are covered too. So nobody loses any thing much basically.
- chongli 8y ago
  how to keep people engaged Once this becomes the goal, the game is already up. It doesn't matter whether you use A/B testing, focus groups, or brilliant designers who operate on intuition. By targeting engagement, you're treating your users as a means to an end and falling afoul of the categorical imperative [1]. You're focused not on meeting their needs as users but your own needs as an engagement-driven platform. The answer to this puzzle is to take engagement right off the table. An ethical business model is one in which you treat your customers as fully autonomous agents. As soon as you look past people's autonomy and see them as creatures of habit with psychological traits that can be gamed, you stray from the ethical path and find yourself searching for other rationalizations. [1] https://en.wikipedia.org/wiki/Categorical_imperative https://en.wikipedia.org/wiki/Categorical_imperative
  - vladbarash 8y ago
    Which metrics would one track to remain on the ethical path, if any?
    - deleted 8y ago
      [deleted]
    - tivert 8y ago
      Probably lots of things that require direct user feedback, like output from focus groups such as customer attitudes towards your product or customer impressions of utility and value.
    - chongli 8y ago
      I would say: talk to your customers, find out what their needs are and try to make your product better at fulfilling them. Focusing on metrics is going back in the direction of treating your customers as means to an end: like grain to be harvested or sheep to be shorn. Instead try to think of your customers the way an old barber would think of his clients: as friends with individual needs who trust your experience in a particular problem domain. Of course, this approach is extremely unlikely to lead to a billion+$ exit. But if your goal is to become a billionaire and you want to be as stridently ethical as possible, I don't know how to help you (and would invite you to further examine your priorities).
zzzcpan 8y ago
Usability testing is a way to do it properly. That and hiring actual UX professionals, not designers.
xiphias2 8y ago
At Google (while Amit was there) trained manual raters were very important part of determining whether a change should be launched or not (especially for relevance). I'm not sure how the process changed since he left, as I see much more wrong auto-corrections by Google.
annadane 8y ago
I mean one way to do it properly is to actually listen to your users. So many corporations claim to have that as a priority and when you look at real, actual feedback, they really don't. How hard is it to look at the masses of complaints about autoplay or notification spam or... any numbers of dark patterns you could name and decide that you shouldn't do that?
- jazzyjackson 8y ago
  Yes I remember this, every time, as Facebook, Twitter, and Instagram made it more and more difficult to keep your feed in chronological order. It is nakedly a psychological trick to give you a mixture of things you've seen before and new content to make you feel like you should try again more often - the slot machine strategy. No one asked for it. Of course it helps that ads are easier to integrate into a feed when you don't even know what to expect in your feed in the first place. So now, not only do these platforms not give a feed of pages and people I subscribe to, they litter my feed with ads and 'popular content' that I never subscribed to! otherwise how could they charge money to increase your organic reach? what a crock! It was a marketers' dream to have their latest news and coupons delivered to your inbox and have you actually look at them, but if you subscribe to businesses you want to follow, how can the ad network make money? So they turned everyone into a customer of their ad platform, and held your content hostage until you paid up. This has effected numerous friends that are small content producers that maxxed out facebook's 5,000 friend limit and all they wanted was to invite all of their fans to their next event. /rant
  - yoble 8y ago
    Other POV: I only check FB a few minutes everyday or so and that algo, maybe ironically, is what allows me to do so. You have 1000 friends each interacting once per day with FB in a news feed worthy way (posting, joining an event, commenting etc). Are you supposed to look at 500 daily news feed posts to get a 50% chance of knowing what that friend you really like is up to? (pushing the trait but you get the idea) Instead having to seep through endless activities I don't care about I can scan 10-15 posts per day and they are the relevant ones from the groups and friends I want to interact with. -- I'm fully with you on the point you make about business/content creator you volontarily subscribed to having to pay extra for you to see their stuff.
TeMPOraL 8y ago
> Is there anyway to do this properly? Others say "hire competent UX people", but I don't think it's necessary, and I don't think doing that will help. Even the best UX person won't help you if management - or processes they set up - push for microoptimizing profitability at all costs. If the competent UX person doesn't have morals and ability to push back based on that, they'll either leave, or - worse - implement the bad things that much more efficiently. The way I see it, companies must realize where their profit/growth optimization borders on user abuse, and then not cross that line (or be made to not cross that line).
- Mtinie 8y ago
  All the morals in the world are meaningless when there is no agency. I’ve yet to find a UX team I’ve worked on with enough political capital to win any conversation which results in a net-decrease in profits, regardless of the ultimate benefit. Too many quarter to quarter direction to overcome the profits-today-at-all-costs momentum which grips management.
  - TeMPOraL 8y ago
    Yup. I have my differences of opinion with most UI/UX teams I worked with (in particular, I don't buy the trend of making software as easy as possible at all costs; it leads to dumbing down and reduced utility). But I'm not blaming UX people for this. They have as much to say in this topic as developers about third-party JS bloat - i.e. we can all complain all we want, out of practicality or morality, and ultimately the decisionmakers don't care.
nonbel 8y ago
AB testing is just NHST. This destroys every area that adopts it.
- rmellow 8y ago
  I assume you mean Null Hypothesis Significance Testing. Can you elaborate or provide sources as to why this destroys adopters?
  - notafraudster 8y ago
    To be brief: The problems associated with NHST are well documented (three that are especially awful: multiple comparisons lead to uncontrolled rate of false positives; large sample sizes leading to statistically significant but substantively irrelevant effects being "accepted"; hard thresholds of statistical significance are arbitrary and the difference between a significant effect and insignificant effect is rarely significant). In addition, there are problems that emerge from using NHST as part of an A/B test: namely, using live, rolling samples to identify significant results rather than prespecifying sample sizes; and making atomic changes as though they are additive without re-testing joint hypotheses of multiple changes. But when you come down to it, the question you want to ask is "Is B better than A?" and the question you do ask is "If A were truly no better than B, how often would a sample of size n drawn using the same sampling procedure I think I'm using produce the impression than B is as much better than A as it is apparently in my observed data?", and two problems are that these aren't the same question and almost no one knows they're doing the latter. To be totally fair, one of the most common problems with NHST (the null hypothesis is patently absurd) isn't necessarily a problem in the A/B UX case. Not sure which of these in specific the grandparent is referring to, but I suspect they and I are on the same page in general.
    - nonbel 8y ago
      You may be interested in distinguishing between enumerative and analytic studies as described here: https://s3.amazonaws.com/wedi/www/Articles/b21d561f-9aea-4b8d-8757-d70964ae13b5.pdf https://s3.amazonaws.com/wedi/www/Articles/b21d561f-9aea-4b8...
tybit 8y ago
I find it interesting that everyone’s alternatives all involve much more subjective decision making progress than A/B testing. Not that it necessarily makes them worse but it partly explains why it’s so popular to A/B test.
- deleted 8y ago
  [deleted]
- sbov 8y ago
  It's popular because math and stats are always right. Even if you've hacked your A/B test to the point of where you should have zero confidence in the results, you can still point to them and say "math says this is the best decision, so we should take it".
  - darawk 8y ago
    That may be part of it, but maybe more importantly, it's also cheaper, which I think was the grandparent's point.
  - hinkley 8y ago
    Lies, damn lies, and statistics.
  - pbhjpbhj 8y ago
    Yes, in addition you need to define "right" -- people are fickle beasts. You can test, find something is objectively better, but then have your users dislike it for fashion or historic reasons, or whatever.
deleted 8y ago
[deleted]
apatters 8y ago
Any metric or KPI is going to get gamed in harmful ways if it's prioritized too highly. So the answer is to use A/B testing to make sure you're not regressing on the goals of a design change, and judiciously, to validate whether your change is moving things in the right direction. But beyond that you have to do the hard slog of mastering the discipline of UX, real, human-centric design, and accept that not everything important is measurable. We have plenty of examples of how doing real UX instead of playing a numbers game can differentiate your business, Apple has applied this philosophy consistently over many years. The root of the problem is a "fuck you, market share at all costs" culture that has come to permeate Silicon Valley. And you can argue (somewhat cynically) that this philosophy makes sense in a blue ocean where you have no competition and just need to gobble up people and turn them into cash before someone else does. But I think going forward this mentality may actually become a liability as more humane alternatives to heavily despised products emerge. Many of the current crop of giants seem to have forgotten that a company's most valuable asset is always its brand.
- amelius 8y ago
  The problem could be simply shareholders and their short term vision.
- sizzle 8y ago
  When did the practice of 'informed consent' on human test subjects become optional?
- protonfish 8y ago
  I agree with most of what your wrote, but "hire an expert" or "you just have to learn" are non-answers to the question that I am seeing a lot in these comments including your: > mastering the discipline of UX... There is no reason to be ambiguous about the practice of good UX design. The main mistake made by most A/B testing is to think that just because you do some testing, it will magically make your UI somehow better. You can make any tests you want, but how do you know they the right tests? And test results do not interpret themselves. There are no obvious conclusions to be drawn from the results of a test. The correct way to do user testing is to start with clear, articulated goals, then select tests to see if the goals are failing or succeeding. Anything else is just back-dating hypotheses - a known anti-science practice. The other misinterpretation of A/B testing is that it will make user interfaces "better" for the end user. Testing can only support or disprove the stated goal and it works just as well for goals that are foolish, short-sighted, self-destructive, and malevolent as it does for those that are ethical and beneficial.
nurettin 8y ago
From a software engineering point if view, the problem may be not A/B testing correctly. From a sociological and personal point, problem is the companies which effectively make money off of stealing people's time and data for profit by making them agree to short term gains through addictive content. What we as responsible adults need to do is to find out what addictive psychology is and put an end to these opportunistic constructs through law making. In my opinion, time is better spent on cultural skills and pursuits, not flocking to "the next big thing" you are spoonfed.
geofft 8y ago
Your comment made me realize that New Coke is essentially a cautionary tale about A/B testing.
LoSboccacc 8y ago
> Is there anyway to do this properly Cycle metrics so that you don't get stuck on local optimization minima. It's also helpful to have long term and short term optimizations to avoid burning fats, like measure both conversion and yearly conversion because it's more than likely that a discount for a week will not increase yearly conversion but will reduce yearly earning. This was learned hard by steam, they made the sale event so big the whole game industry changed their pricing practices dropping in time much much more slowly than what it used to be.
Rjevski 8y ago
Snapchat didn’t need A/B testing. Snapchat needed someone with common sense to take a look at the new design and say “this is bullshit!” Sadly this didn’t happen because the entire company is based on a lack of common sense by the looks of it.
- wpietri 8y ago
  This is perfectly useless advice. You've hidden all the magic in vagueness. Everybody thinks they have common sense, so of course they already think somebody with common sense looked at the design. I'd suggest either just giving an honest rant or giving actual advice. And for that you have to really unpack what you mean by "common sense" until somebody who doesn't have "common sense" can have a chance of understanding and fixing the problem.
  - Rjevski 8y ago
    I would ageee with you for cases where both sides had a strong argument, but in Snapchat’s case everyone was complaining about the new UI, so I’m not sure how to describe that sense - people who complained are from all different ages & backgrounds so it’s not like they had UI design experience or anything - “common sense” is probably the only thing they all had in common and that thing alone was enough for them to determine the UI sucked. In this case, simply doing some basic user-testing would’ve been considered common sense and would’ve brought up the problem immediately.
    - wpietri 8y ago
      In which case, they didn't need "someone with common sense to take a look at the new design". They needed basic user testing. Telling a company "do some basic user testing" is a useful recommendation. Telling them to have "common sense" isn't.
      - Rjevski 8y ago
        Basic user testing would’ve been recommended by anyone with even a tiny bit of common sense. Either the company truly didn’t have anyone do user testing, or they did and were ignored (which doesn’t surprise me either considering the child they appointed as “CEO”).
pmart123 8y ago
I keep being more and more impressed with Apple to be honest. Whether it is desktop UX, phone touch UI, laptop trackpads, or even how seamless Apple Pay/FaceId works, most of what they push stands up to the test of time relative to the seemingly more data-driven experiences.
the_clarence 8y ago
It always sounds like the people working on these redesigns are not heavy users of the product. And by that I mean legit heavy users, not dogfed users.
- Nasrudith 8y ago
  Personally I've said half-jokingly that the redesigns are for job security because admitting that the design is perfectly fine and changes have no guarantee of improvement greater than the adjustment cost would result in a pink slip. Although that does point to a common problem of 'design for purchasing manager' where software is designed for the sake of pleasing people who don't actually use it and merely adds a bunch of useless cruft that would rapidly become an irritating obstacle to users but looks impressive on first blush.
rjf72 8y ago
One of the best examples of this problem is something entirely outside of software. Coke and Pepsi. It's well known that people prefer Pepsi to Coke in blind taste tests by a very wide margin. Yet in the market Coke outsells Pepsi by a very wide margin. This anecdote was used for quite some time as an example of the power of advertising. But it turns out the truth is something much more simple. While people do prefer Pepsi in blind taste tests, that's because in blind taste tests you're only getting an isolated sample of the product. To cut out the software metaphor, people prefer Coke when it comes down to drinking an entire bottle of the product. It has nothing at all to do with advertising or false preference - Pepsi was just testing the wrong thing. The point here is that just because people think they want or like something doesn't necessarily really mean they do, especially when what they're 'consuming' is in some way different than the regular long term consumption of the product (as you hit on). I don't think it's much of a coincidence that some of the most famous marketers, most relevantly here probably being Steve Jobs, weren't particularly interested in what the consumer thought they wanted.
snarf21 8y ago
It is as you say. The key is that you always get what you optimize for. You can optimize for short term addiction or for long term sustainability. The issue is that large factors of these goals are incompatible. This reminds me of a few years ago when they did "cash for clunkers" to stimulate the economy. Tons of people took advantage of it and car dealerships has two amazing months........ followed by four months of almost no sales. All that happened was that people who would have bought one in the next six months took advantage of the incentive now. It didn't create any new demand for cars or stimulate anything. If you only measured success during the first two months you might have been convinced that it was a roaring success.
- anon_cow1111 8y ago
  >This reminds me of a few years ago when they did "cash for clunkers" to stimulate the economy. Psst.. Are you ready to feel old today? The Cash for clunkers program was from 2009. Yeah, really. (The federal program in the US at least, I don't know if states ran their own version since)
sizzle 8y ago
I think there needs to be some sort of human ethics oversight for running psychologically harmful research experiments under the guise of A/B testing. When did the practice of 'informed consent' on human test subjects become optional? The infamous Facebook user emotion study comes to mind: https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/ https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook... Anyone know why tech companies aren't held to the same ethical standards as academia?
- vageli 8y ago
  To what body would a corporate entity be accountable for ethics, beyond government? Which code of ethics would apply?
darawk 8y ago
This is exactly the way capitalism is supposed to work. Companies correctly prioritizing their long-term wellbeing over short term gains. Netflix is watching what's happening to Facebook, and deciding that that's not the path they want to take, even though it would likely be good for their profits in the near term.
- SN76477 8y ago
  I wish we could see more logical long term ideas. So much of it is short sighted I feel. I often describe stuff such as Facebook as immature technology that doesn’t understand what it is capable of, and doesn’t seem to care about the negative impact.
  - darawk 8y ago
    Ya, it is unfortunate. But I think it is the process working the way it's supposed to. It sucks that we have to touch the stove sometimes to learn that it's hot, but that seems to be how it works. Facebook touched the stove and their stock price got burnt. Now everyone else is learning from that so it doesn't happen to them. I think it's the system working as it should, and I just hope people don't over-react and try to throw the system out when it seems to me to be actually doing a pretty good job at healing itself here.

johnday 8y ago

> Is there anyway to do this properly? The way to do it properly is to hire experienced, competent UX designers and human computer interaction specialists, who understand the problem space and how to keep people engaged without addictive malpractice / dark patterns. Metrics are useful but they are what they claim to be - measurements - and nothing more. You wouldn't choose which house to buy based on how large it was measured to be, but you would expect a surveyor to make those measurements. Speaking as a web and video game designer, I can't wait until I can hire someone to think about these things for me.

TaylorAlexander 8y ago
This makes sense. It’s like how clickbait headlines would, if they previously didn’t exist, improve click rates. But long term if your platform was all click bait it would fatigue the users. Or at least, one might think so...
- alan_wade 8y ago
  Clickbait + low quality content is what fatigues users. Personally I don't mind clickbait if after doing whatever it takes to grab my attention, the creator delivers on an actually solid high quality educational/entertaining content.
  - WA 8y ago
    This concept is called: overpromise and underdeliver. The content should at least match the promise of the headline, or even overdeliver. Then readers are happy.
  - mikestew 8y ago
    Personally I don't mind clickbait if after doing whatever it takes to grab my attention, the creator delivers on an actually solid high quality Captain Pedantic, reporting for duty, and that’s not actually clickbait. If the masterful baiter of clicks actually delivers, then it wasn’t bait. I mean, what’s to complain about? Turns out there is a way to target stubborn belly fat, five minutes a day, who knew?
pfschell 8y ago
Designers don't cause terrible designs like this. MBAs and marketing droids do.
- newppc 8y ago
  Pretty sure Facebook is not a company that is taken over by MBAs and marketers that are making this happen. Rather it's an engineering-first organization, run by an engineer. Probably a mix of everyone's fault.
  - dbt00 8y ago
    How is an MBA or marketer evil in a way that Zuck isn’t?
- abakker 8y ago
  That’s a bit reductive. You may be right, but it would be less facile to say that poor designs can be made by anybody, but that a mismatch between design and business objectives led to bad results. Not all business people are morons, and not all designers are good at design.
  - ikfmpwdsoz 8y ago
    If the business objectives lead to a design so bad that it hurts the business, surely the people who came up with these objectives didn't do a very good job?
kokokokoko 8y ago
People are not hired for roles when a company can afford to. People are hired for roles when company cannot afford not to.
- chaosite 8y ago
  That's an ideal. Most companies like to grow, and growth usually means hiring.
- johnday 8y ago
  At the moment, I can't afford to and I can't afford not to. So I'm just muddling through on my own.
- kamaal 8y ago
  >>People are hired for roles when company cannot afford not to. Only if people have a stake in the decline of a company, and have something to lose. Other wise in these days of executive networks, and people getting in their golf and dinner buddies- Execs mostly care about nothing at all. The only people to lose anything are a few low level employees and may be shareholders. Shareholders mostly don't care either. People who hold large block of shares, are generally diversified. And since nothing fails at all at once- They are covered too. So nobody loses any thing much basically.
chongli 8y ago
how to keep people engaged Once this becomes the goal, the game is already up. It doesn't matter whether you use A/B testing, focus groups, or brilliant designers who operate on intuition. By targeting engagement, you're treating your users as a means to an end and falling afoul of the categorical imperative [1]. You're focused not on meeting their needs as users but your own needs as an engagement-driven platform. The answer to this puzzle is to take engagement right off the table. An ethical business model is one in which you treat your customers as fully autonomous agents. As soon as you look past people's autonomy and see them as creatures of habit with psychological traits that can be gamed, you stray from the ethical path and find yourself searching for other rationalizations. [1] https://en.wikipedia.org/wiki/Categorical_imperative https://en.wikipedia.org/wiki/Categorical_imperative
- vladbarash 8y ago
  Which metrics would one track to remain on the ethical path, if any?
  - deleted 8y ago
    [deleted]
  - tivert 8y ago
    Probably lots of things that require direct user feedback, like output from focus groups such as customer attitudes towards your product or customer impressions of utility and value.
  - chongli 8y ago
    I would say: talk to your customers, find out what their needs are and try to make your product better at fulfilling them. Focusing on metrics is going back in the direction of treating your customers as means to an end: like grain to be harvested or sheep to be shorn. Instead try to think of your customers the way an old barber would think of his clients: as friends with individual needs who trust your experience in a particular problem domain. Of course, this approach is extremely unlikely to lead to a billion+$ exit. But if your goal is to become a billionaire and you want to be as stridently ethical as possible, I don't know how to help you (and would invite you to further examine your priorities).

zzzcpan 8y ago

Usability testing is a way to do it properly. That and hiring actual UX professionals, not designers.

xiphias2 8y ago

At Google (while Amit was there) trained manual raters were very important part of determining whether a change should be launched or not (especially for relevance). I'm not sure how the process changed since he left, as I see much more wrong auto-corrections by Google.

annadane 8y ago

I mean one way to do it properly is to actually listen to your users. So many corporations claim to have that as a priority and when you look at real, actual feedback, they really don't. How hard is it to look at the masses of complaints about autoplay or notification spam or... any numbers of dark patterns you could name and decide that you shouldn't do that?

jazzyjackson 8y ago
Yes I remember this, every time, as Facebook, Twitter, and Instagram made it more and more difficult to keep your feed in chronological order. It is nakedly a psychological trick to give you a mixture of things you've seen before and new content to make you feel like you should try again more often - the slot machine strategy. No one asked for it. Of course it helps that ads are easier to integrate into a feed when you don't even know what to expect in your feed in the first place. So now, not only do these platforms not give a feed of pages and people I subscribe to, they litter my feed with ads and 'popular content' that I never subscribed to! otherwise how could they charge money to increase your organic reach? what a crock! It was a marketers' dream to have their latest news and coupons delivered to your inbox and have you actually look at them, but if you subscribe to businesses you want to follow, how can the ad network make money? So they turned everyone into a customer of their ad platform, and held your content hostage until you paid up. This has effected numerous friends that are small content producers that maxxed out facebook's 5,000 friend limit and all they wanted was to invite all of their fans to their next event. /rant
- yoble 8y ago
  Other POV: I only check FB a few minutes everyday or so and that algo, maybe ironically, is what allows me to do so. You have 1000 friends each interacting once per day with FB in a news feed worthy way (posting, joining an event, commenting etc). Are you supposed to look at 500 daily news feed posts to get a 50% chance of knowing what that friend you really like is up to? (pushing the trait but you get the idea) Instead having to seep through endless activities I don't care about I can scan 10-15 posts per day and they are the relevant ones from the groups and friends I want to interact with. -- I'm fully with you on the point you make about business/content creator you volontarily subscribed to having to pay extra for you to see their stuff.

TeMPOraL 8y ago

> Is there anyway to do this properly? Others say "hire competent UX people", but I don't think it's necessary, and I don't think doing that will help. Even the best UX person won't help you if management - or processes they set up - push for microoptimizing profitability at all costs. If the competent UX person doesn't have morals and ability to push back based on that, they'll either leave, or - worse - implement the bad things that much more efficiently. The way I see it, companies must realize where their profit/growth optimization borders on user abuse, and then not cross that line (or be made to not cross that line).

Mtinie 8y ago
All the morals in the world are meaningless when there is no agency. I’ve yet to find a UX team I’ve worked on with enough political capital to win any conversation which results in a net-decrease in profits, regardless of the ultimate benefit. Too many quarter to quarter direction to overcome the profits-today-at-all-costs momentum which grips management.
- TeMPOraL 8y ago
  Yup. I have my differences of opinion with most UI/UX teams I worked with (in particular, I don't buy the trend of making software as easy as possible at all costs; it leads to dumbing down and reduced utility). But I'm not blaming UX people for this. They have as much to say in this topic as developers about third-party JS bloat - i.e. we can all complain all we want, out of practicality or morality, and ultimately the decisionmakers don't care.

nonbel 8y ago

AB testing is just NHST. This destroys every area that adopts it.

rmellow 8y ago
I assume you mean Null Hypothesis Significance Testing. Can you elaborate or provide sources as to why this destroys adopters?
- notafraudster 8y ago
  To be brief: The problems associated with NHST are well documented (three that are especially awful: multiple comparisons lead to uncontrolled rate of false positives; large sample sizes leading to statistically significant but substantively irrelevant effects being "accepted"; hard thresholds of statistical significance are arbitrary and the difference between a significant effect and insignificant effect is rarely significant). In addition, there are problems that emerge from using NHST as part of an A/B test: namely, using live, rolling samples to identify significant results rather than prespecifying sample sizes; and making atomic changes as though they are additive without re-testing joint hypotheses of multiple changes. But when you come down to it, the question you want to ask is "Is B better than A?" and the question you do ask is "If A were truly no better than B, how often would a sample of size n drawn using the same sampling procedure I think I'm using produce the impression than B is as much better than A as it is apparently in my observed data?", and two problems are that these aren't the same question and almost no one knows they're doing the latter. To be totally fair, one of the most common problems with NHST (the null hypothesis is patently absurd) isn't necessarily a problem in the A/B UX case. Not sure which of these in specific the grandparent is referring to, but I suspect they and I are on the same page in general.
  - nonbel 8y ago
    You may be interested in distinguishing between enumerative and analytic studies as described here: https://s3.amazonaws.com/wedi/www/Articles/b21d561f-9aea-4b8d-8757-d70964ae13b5.pdf https://s3.amazonaws.com/wedi/www/Articles/b21d561f-9aea-4b8...

tybit 8y ago

I find it interesting that everyone’s alternatives all involve much more subjective decision making progress than A/B testing. Not that it necessarily makes them worse but it partly explains why it’s so popular to A/B test.

deleted 8y ago
[deleted]
sbov 8y ago
It's popular because math and stats are always right. Even if you've hacked your A/B test to the point of where you should have zero confidence in the results, you can still point to them and say "math says this is the best decision, so we should take it".
- darawk 8y ago
  That may be part of it, but maybe more importantly, it's also cheaper, which I think was the grandparent's point.
- hinkley 8y ago
  Lies, damn lies, and statistics.
- pbhjpbhj 8y ago
  Yes, in addition you need to define "right" -- people are fickle beasts. You can test, find something is objectively better, but then have your users dislike it for fashion or historic reasons, or whatever.

deleted 8y ago

[deleted]

apatters 8y ago

Any metric or KPI is going to get gamed in harmful ways if it's prioritized too highly. So the answer is to use A/B testing to make sure you're not regressing on the goals of a design change, and judiciously, to validate whether your change is moving things in the right direction. But beyond that you have to do the hard slog of mastering the discipline of UX, real, human-centric design, and accept that not everything important is measurable. We have plenty of examples of how doing real UX instead of playing a numbers game can differentiate your business, Apple has applied this philosophy consistently over many years. The root of the problem is a "fuck you, market share at all costs" culture that has come to permeate Silicon Valley. And you can argue (somewhat cynically) that this philosophy makes sense in a blue ocean where you have no competition and just need to gobble up people and turn them into cash before someone else does. But I think going forward this mentality may actually become a liability as more humane alternatives to heavily despised products emerge. Many of the current crop of giants seem to have forgotten that a company's most valuable asset is always its brand.

amelius 8y ago
The problem could be simply shareholders and their short term vision.
sizzle 8y ago
When did the practice of 'informed consent' on human test subjects become optional?
protonfish 8y ago
I agree with most of what your wrote, but "hire an expert" or "you just have to learn" are non-answers to the question that I am seeing a lot in these comments including your: > mastering the discipline of UX... There is no reason to be ambiguous about the practice of good UX design. The main mistake made by most A/B testing is to think that just because you do some testing, it will magically make your UI somehow better. You can make any tests you want, but how do you know they the right tests? And test results do not interpret themselves. There are no obvious conclusions to be drawn from the results of a test. The correct way to do user testing is to start with clear, articulated goals, then select tests to see if the goals are failing or succeeding. Anything else is just back-dating hypotheses - a known anti-science practice. The other misinterpretation of A/B testing is that it will make user interfaces "better" for the end user. Testing can only support or disprove the stated goal and it works just as well for goals that are foolish, short-sighted, self-destructive, and malevolent as it does for those that are ethical and beneficial.

nurettin 8y ago

From a software engineering point if view, the problem may be not A/B testing correctly. From a sociological and personal point, problem is the companies which effectively make money off of stealing people's time and data for profit by making them agree to short term gains through addictive content. What we as responsible adults need to do is to find out what addictive psychology is and put an end to these opportunistic constructs through law making. In my opinion, time is better spent on cultural skills and pursuits, not flocking to "the next big thing" you are spoonfed.

geofft 8y ago

Your comment made me realize that New Coke is essentially a cautionary tale about A/B testing.

LoSboccacc 8y ago

> Is there anyway to do this properly Cycle metrics so that you don't get stuck on local optimization minima. It's also helpful to have long term and short term optimizations to avoid burning fats, like measure both conversion and yearly conversion because it's more than likely that a discount for a week will not increase yearly conversion but will reduce yearly earning. This was learned hard by steam, they made the sale event so big the whole game industry changed their pricing practices dropping in time much much more slowly than what it used to be.

Rjevski 8y ago

Snapchat didn’t need A/B testing. Snapchat needed someone with common sense to take a look at the new design and say “this is bullshit!” Sadly this didn’t happen because the entire company is based on a lack of common sense by the looks of it.

wpietri 8y ago
This is perfectly useless advice. You've hidden all the magic in vagueness. Everybody thinks they have common sense, so of course they already think somebody with common sense looked at the design. I'd suggest either just giving an honest rant or giving actual advice. And for that you have to really unpack what you mean by "common sense" until somebody who doesn't have "common sense" can have a chance of understanding and fixing the problem.
- Rjevski 8y ago
  I would ageee with you for cases where both sides had a strong argument, but in Snapchat’s case everyone was complaining about the new UI, so I’m not sure how to describe that sense - people who complained are from all different ages & backgrounds so it’s not like they had UI design experience or anything - “common sense” is probably the only thing they all had in common and that thing alone was enough for them to determine the UI sucked. In this case, simply doing some basic user-testing would’ve been considered common sense and would’ve brought up the problem immediately.
  - wpietri 8y ago
    In which case, they didn't need "someone with common sense to take a look at the new design". They needed basic user testing. Telling a company "do some basic user testing" is a useful recommendation. Telling them to have "common sense" isn't.
    - Rjevski 8y ago
      Basic user testing would’ve been recommended by anyone with even a tiny bit of common sense. Either the company truly didn’t have anyone do user testing, or they did and were ignored (which doesn’t surprise me either considering the child they appointed as “CEO”).

pmart123 8y ago

I keep being more and more impressed with Apple to be honest. Whether it is desktop UX, phone touch UI, laptop trackpads, or even how seamless Apple Pay/FaceId works, most of what they push stands up to the test of time relative to the seemingly more data-driven experiences.

the_clarence 8y ago

It always sounds like the people working on these redesigns are not heavy users of the product. And by that I mean legit heavy users, not dogfed users.

Nasrudith 8y ago
Personally I've said half-jokingly that the redesigns are for job security because admitting that the design is perfectly fine and changes have no guarantee of improvement greater than the adjustment cost would result in a pink slip. Although that does point to a common problem of 'design for purchasing manager' where software is designed for the sake of pleasing people who don't actually use it and merely adds a bunch of useless cruft that would rapidly become an irritating obstacle to users but looks impressive on first blush.

rjf72 8y ago

One of the best examples of this problem is something entirely outside of software. Coke and Pepsi. It's well known that people prefer Pepsi to Coke in blind taste tests by a very wide margin. Yet in the market Coke outsells Pepsi by a very wide margin. This anecdote was used for quite some time as an example of the power of advertising. But it turns out the truth is something much more simple. While people do prefer Pepsi in blind taste tests, that's because in blind taste tests you're only getting an isolated sample of the product. To cut out the software metaphor, people prefer Coke when it comes down to drinking an entire bottle of the product. It has nothing at all to do with advertising or false preference - Pepsi was just testing the wrong thing. The point here is that just because people think they want or like something doesn't necessarily really mean they do, especially when what they're 'consuming' is in some way different than the regular long term consumption of the product (as you hit on). I don't think it's much of a coincidence that some of the most famous marketers, most relevantly here probably being Steve Jobs, weren't particularly interested in what the consumer thought they wanted.

snarf21 8y ago

It is as you say. The key is that you always get what you optimize for. You can optimize for short term addiction or for long term sustainability. The issue is that large factors of these goals are incompatible. This reminds me of a few years ago when they did "cash for clunkers" to stimulate the economy. Tons of people took advantage of it and car dealerships has two amazing months........ followed by four months of almost no sales. All that happened was that people who would have bought one in the next six months took advantage of the incentive now. It didn't create any new demand for cars or stimulate anything. If you only measured success during the first two months you might have been convinced that it was a roaring success.

anon_cow1111 8y ago
>This reminds me of a few years ago when they did "cash for clunkers" to stimulate the economy. Psst.. Are you ready to feel old today? The Cash for clunkers program was from 2009. Yeah, really. (The federal program in the US at least, I don't know if states ran their own version since)

sizzle 8y ago

I think there needs to be some sort of human ethics oversight for running psychologically harmful research experiments under the guise of A/B testing. When did the practice of 'informed consent' on human test subjects become optional? The infamous Facebook user emotion study comes to mind: https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/ https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook... Anyone know why tech companies aren't held to the same ethical standards as academia?

vageli 8y ago
To what body would a corporate entity be accountable for ethics, beyond government? Which code of ethics would apply?

darawk 8y ago

This is exactly the way capitalism is supposed to work. Companies correctly prioritizing their long-term wellbeing over short term gains. Netflix is watching what's happening to Facebook, and deciding that that's not the path they want to take, even though it would likely be good for their profits in the near term.

SN76477 8y ago
I wish we could see more logical long term ideas. So much of it is short sighted I feel. I often describe stuff such as Facebook as immature technology that doesn’t understand what it is capable of, and doesn’t seem to care about the negative impact.
- darawk 8y ago
  Ya, it is unfortunate. But I think it is the process working the way it's supposed to. It sucks that we have to touch the stove sometimes to learn that it's hot, but that seems to be how it works. Facebook touched the stove and their stock price got burnt. Now everyone else is learning from that so it doesn't happen to them. I think it's the system working as it should, and I just hope people don't over-react and try to throw the system out when it seems to me to be actually doing a pretty good job at healing itself here.