10 ms·

This is why I sometimes hate a/b testing. I'm sure someone at booking a/b tested these things and saw an increase in revenue. The thing that these tests don't m

by jvans 3y ago

This is why I sometimes hate a/b testing. I'm sure someone at booking a/b tested these things and saw an increase in revenue. The thing that these tests don't measure are very long term effects where people either start to hate your product and look for alternatives, or become so numb to the changes that the initial novelty effect wears off. The person who ran the test gets a promotion for increasing revenue during the quarter but the net result is a massive negative for the longevity of the product.

dyno12345 3y ago
see also: facebook
wpietri 3y ago
That is not a problem with A/B testing. It's a problem with the values of the company. I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." People who value revenue metrics over all else will still do shitty things for users even if they don't A/B test.
- lylejantzi3rd 3y ago
  The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.
  - eropple 3y ago
    > Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values. If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making. Which means the company does not have those "good values" in the first place. They have lip service. Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.
    - hectormalot 3y ago
      I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white: - almost any change will have have people arguing for and against it - if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change - as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns) - as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact. Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.
      - thrashh 3y ago
        Still, that’s what sets a good manager from an average, no? No different with how there are relatively few good engineers but most are average or bad? I mean there are companies that put out good products but most are average at best. I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.
    - grogenaut 3y ago
      You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.
    - dahart 3y ago
      You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.
  - projectazorian 3y ago
    This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.
    - lylejantzi3rd 3y ago
      > They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience. And how do you know the short term changes will degrade the brand and the user experience?
      - mitthrowaway2 3y ago
        Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!
    - chefandy 3y ago
      In my experience founders are often more than happy to toe the line between nudges and shoves.
  - MarkSweep 3y ago
    This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.
    - dotancohen 3y ago
      You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
      - smachiz 3y ago
        isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
        dotancohen 3y ago
        I suppose that is a way of looking at it.
      - bootsmann 3y ago
        Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
    - rdca 3y ago
      Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
      - esafak 3y ago
        And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
      - 3y ago
  - HPsquared 3y ago
    "Seeing like a State" in action.
  - wpietri 3y ago
    Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects. I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.
- deleted 3y ago
  [deleted]
- antman 3y ago
  There is also a mathematical problem of assigning events and actions to long term effects. The usual IT crowd unfamiliar with the respective literature will try to ab test and grid search out of the actual scientific part of data science. I had also fallen to that trap.
  - computerfriend 3y ago
    What would you recommend reading on this?
- jiggywiggy 3y ago
  There is a grey line. I think only one room left is actually useful info. But yeah they push it too far. But most people in a company will be able to argue for themselves the info is useful and truthful so in their minds it's morally ok.
  - toast0 3y ago
    Only one room left would be useful information if it were accurate. Most of the time, booking agents don't really have an accurate picture of inventory though. It's more like only one room available for booking.com to book right now. It could be that the hotel is holding back rooms for other channels or because they like to not be fully booked so far ahead of time or perhaps the hotel has found listing only one room at a time gets them a better look to book ratio (in part because of anxiety inspiring features like this). Without an understanding of the industry though, it's not really useful information.
- chambers 3y ago
  DHH, founder of Ruby on Rails and Basecamp, drew a line between core values and A/B testing https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5f562 https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5... > This is the tyranny of easy metrics. It's easy to measure how much money is saved by preventing cancelations, it's much harder to measure how much long-term business is lost by poisoning your reputation with the 99.9% of customers who had to jump hoops and dodge sleazeballs to get out of the subscription. But the latter could well be orders of magnitude money more over the long run.
- jvans 3y ago
  I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.
- canadianfella 3y ago
  [dead]
- chefandy 3y ago
  Yeah-- I think blaming dark patterns on A/B testing is like blaming thermostats for chilly houses. A/B tests just show if one thing does more of something you're testing for than something else. If you're just testing for "conversions" then you're going to make websites like booking.com. If you're testing to see if users are more stressed out by one situation or another, or testing to see if expert users are stymied by some interface abstraction designed to make things easier for less sophisticated users, then that's totally different.
  - tspike 3y ago
    Sure, but it’s a lot harder to test for things like stress or being stymied than it is to test for conversion. Easy implementation + results that make money graphs go up will win every time without C-level involvement.
    - chefandy 3y ago
      It's a lot easier to break something with a hammer than bang in some nails. Using a marker to deface property is a lot easier than using it to draw a picture.
- SilasX 3y ago
  >I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." Then why were they testing it, if they already knew other concerns would veto that alternative?
  - wpietri 3y ago
    I suspect if you were to think a bit, you could come up with some ideas yourself. If you're really stuck, let me know what you've got so far and I'll fill in the rest.
    - SilasX 3y ago
      I think I'll just wait for you to give the substantive reply you have in mind. Accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.
      - wpietri 3y ago
        You'll be waiting quite awhile. If there's anything HN has taught me, it's the meaning of "pearls before swine". And the futility in providing free labor for people who seem to think they're entitled to it.
        SilasX 3y ago
        It's a discussion forum, amigo. You're supposed to share your insights with those who haven't yet achieved them. If you're not comfortable doing that, you can just not post instead of posting merely to whine and insult. Bookmarking for later.
        wpietri 3y ago
        I share with people who seem like they'll benefit from it. That includes people who have done at least a little work to understanding what's going on. It definitely doesn't include HN's legion of querulous jerks who mainly look for nitpicks, gotchas, and other opportunities to feel smart, even when that comes at the expense of the good discussion this place is nominally for. You get to influence which group people see you as part of.
- dhosek 3y ago
  When I worked at eHarmony there were two things that we were aware of above all else: 1. Users hated all the advertising even if they were signed in with a paid account and 2. eHarmony had no intention of stopping this because they valued the revenue from the advertising over the user experience. I noticed that Redfin has started adding ads to their site which really annoys me. Folks you’ll get a sizable commission for connecting me with the listing agent. The ad revenue has to be a rounding error compared to that. Why are you damaging your brand by showing me ads for luxury vacations next to houses that I’m looking at buying?
  - wpietri 3y ago
    Great examples. To this list, I'd add Amazon's ads. It's providing them with very obvious revenue growth. It's parasitic on their ecommerce business, though, in ways that for me and many others, harm the things that made Amazon popular in the first place. That'll never change, though, because that revenue has enough people attached to it that they'll fight tooth and nail to keep it no matter how much it harms the rest of Amazon.
- phendrenad2 3y ago
  Companies need to balance between metrics and vision. It's easy to chase metrics and lose track of why your company exists. If you're just chasing metrics, what differentiates you from anyone else? What big bets are you making? Every company I've worked at has fallen into this trap, and it's gotten worse due to the tech slump.
- smueller1234 3y ago
  I think the problem underneath is quantitative fallacy: https://en.m.wikipedia.org/wiki/McNamara_fallacy https://en.m.wikipedia.org/wiki/McNamara_fallacy
GuB-42 3y ago
It is also one of these things where the first to do it gets the advantage. If you are the first to do the "only 2 rooms left" trick for example, you will get the full results before people get desensitized. But people will get desensitized everywhere, not just on your website, so if a competitor tries to pull the same trick, he won't get the same effect as you did. If fact, it may be time for you to roll back, to make competitors look bad for using the now well known and ineffective trick you invented. And if it works long term, then you get a head start.
- mschuster91 3y ago
  Yup, this "poisoning the well" effect is real, and it's blatantly uncompetitive - which is why it's so important for regulatory agencies to step in and act fast and hard so that this "first mover advantage" is eliminated. The problem is, regulatory agencies are slow as molasses and courts are overloaded with crap, which means by the time the process is done years later, the companies have long since switched to yet another sleazebag tactic.
  - flappyeagle 3y ago
    Why? The tactics burn themselves out.
  - Rufbdbskrufb473 3y ago
    I don't want government agencies involved in dictating the minutia of acceptable UI design for a private website. Businesses have brand reputations. Just let people vote with their wallets. That won't discourage all dark patterns, but is preferable to a world of over-regulation.
    - mschuster91 3y ago
      Businesses, particularly in oligopoly/platform scenarios, don't care about brand reputation. Just look at Meta - they don't even care about billions of euros worth of GDPR fines, it's all priced in.
    - 3y ago
listenallyall 3y ago
I don't disagree with you, certainly some "improvements" only have short-term positive effects. However, at least the A/B test was performed and data collected. Your assumption that it will eventually be "a massive negative" is pure speculation with no data behind it.
- RussianCow 3y ago
  The value of a "brand" is really, really difficult to measure objectively, but certainly has an effect on your revenue. It's hard to know in advance what's going to tarnish your reputation in the long run, but by the time you can measure it, it's far too late.
Freak_NL 3y ago
One of the effects this has on me is that I will use booking.com to find places (amongst other tools), but book directly with the accommodation. Only if the accommodation doesn't do its own booking will I use booking.com to book (about once every twenty places).
- EVa5I7bHFq9mnYK 3y ago
  I also had great results just walking up to the reception desk and booking there. Some deals are too good to be put online.
  - vshade 3y ago
    But when you are taking vacations with your family you don't want the stress and the risk of not finding a good accommodation.
- distances 3y ago
  Every time I've checked this, direct booking was more expensive than Booking.com or Hotels.com.
  - criddell 3y ago
    It often is, but I still book direct because I want to be a customer of the hotel and not of some third party. When things go wrong, I don't want to be be told to take it up with the third party (and this has happened to me).
  - gramie 3y ago
    I book directly. If there is a problem with the reservation, hotels have much more flexibility to accommodate changes. You can usually cancel on the arrival date with no penalty, change room type, etc. I've heard others say that if you quote an online price, the front desk can match it (better to get 100% of the discounted price than 60%), although I haven't tried it myself.
wbobeirne 3y ago
We had someone from Booking.com come and speak at a company I was working at a good few years ago to talk about their testing process. They were using the multi-armed bandit approach[1] of just throwing dozens of changes at the wall and seeing which worked best. It definitely reflected in the UX. 1: https://en.m.wikipedia.org/wiki/Multi-armed_bandit https://en.m.wikipedia.org/wiki/Multi-armed_bandit
- digitalengineer 3y ago
  “Dozens of changes” but without values who gets to decide WHAT changes are tested? Sure there is UX/CRO research but without a moral compass it is exactly how you end up with dark patterns.
  - wbobeirne 3y ago
    Agreed, and my impression was there wasn't a lot of scrutiny there. It was purely numbers driven, if it could be tested and measured it would, and if it won it was adopted. Very little emphasis on UX, product cohesion, or any specific design principles.
- sizzle 3y ago
  Can’t believe they are giving talks on this like it’s innovative or something when admiring. I’d honestly be embarrassed saying that my design process was to make shit up and see what happens instead of investing in UX design/ research professionals can run generative studies to better understand their consumer base and sentiments towards booking trips online, like Airbnb heavily invested and can be seen as having a world class user experience next to booking.com I digress…
marcinzm 3y ago
Many companies in my experience do test for these things. They remove old features and measure impact periodically, or run a long term hold out bucket, or some other such approach. The deep dark secret of the web is that there often there isn't a negative impact that can be measured no matter how much people try to measure one.
newaccount74 3y ago
Booking.com lost me as a customer for life after I fell victim to their sleazy tactics a few times. I have refused to book anything on booking for the last few years because I didn't want to be mislead into booking a crappy hotel by their algorithms again. I guess my decision doesn't matter much in the grand scheme of things, because they are still around and judging by the screenshots it's as bad as ever, and people still use the site...
- FireBeyond 3y ago
  Yup. Go now to booking.com and pick a remote roadside motel in Wyoming for a midweek stay sometime in March of 2024. And watch booking.com try to tell you that "19 people have booked this today" or some such bullshit.
  - sizzle 3y ago
    I’m surprised California hasn’t outlawed this tactic as deceptive marketing or such? Any idea why companies can falsify data for the sole purpose of deceiving consumers to increase their revenue?
    - iruoy 3y ago
      There are no laws against it in the US. In the EU there are some[0][1] now. [0]: https://www.acm.nl/en/publications/bookingcom-commits-adjusting-its-website-following-action-european-consumer-authorities https://www.acm.nl/en/publications/bookingcom-commits-adjust... [1]: https://www.acm.nl/en/publications/booking-and-expedia-inform-consumers-more-clearly-about-their-offers-following-action-european-consumer-authorities https://www.acm.nl/en/publications/booking-and-expedia-infor...
  - woobar 3y ago
    I've just tried this and now they show me identical room for the same price with two different nags. Not sure if they have only 2 or 5 (or 7?) "Queen Studios" left. https://imgur.com/a/oqOTYlz https://imgur.com/a/oqOTYlz
  - 3y ago
shswkna 3y ago
I have a personal experience where booking.com’s nudging caused me to reconsider my trip. I was trying to find something suitable to stay in Paris. Maybe it was the exaggeration of booking.com or maybe there was some truth, but at some point I shut down and made a 180 on my plans. I had realised that I don’t want to go somewhere where I have to compete against this avalanche of other visitors who were or were not snapping my accommodation options away. I am now visiting friends in another European city.
- patneedham 3y ago
  But did you still use booking.com for that other European city trip, or another platform?
  - deleted 3y ago
    [deleted]
  - shswkna 3y ago
    I am staying with friends, who have a spare room available.
- schneems 3y ago
  I had a similar reaction to Lyft’s “you have 2 minutes to accept this faster trip” prompt and emphasizing the faster, more expensive option first. I saw that, balked a bit at the interaction and ended up taking a train instead. Not only was it $6.25 instead of $46 it got me there faster than Lyft’s fastest option. Including time walking to and from the station. I wasn’t in a hurry but the in your face “look how much money people are willing to spend to save 5 min” helped me rethink my priorities.
  - OJFord 3y ago
    Similarly (though also exhibiting teenage American language taking over the world symptoms) is Netflix's new (or new-ish, I just noticed it yesterday as a fairly frequent user) 'everone is watching' category. I have absolutely zero interest in that, it's literally repulsive, trivially incorrect, and just plain stupid even apart from the language - who cares what others have chosen to watch, how does it compare to what I've given thumbs up/down?
andix 3y ago
I really startet to hate using Booking.com, especially because every time after using it, they start bombarding me with emails. I could probably turn them off. Another thing that makes me laugh now: I often go to the same hotel, and Booking.com provides better rates then booking directly (no idea why, I asked multiple times for the same discounts). And for my favorite category the hotel has only one room. So booking.com constantly warns me "only 1 room left!". Yeah, I know, there is only one ;)
- vkou 3y ago
  > (no idea why, I asked multiple times for the same discounts) Hotels sell a fraction of their inventory cheaply to resellers. Also, they oversell, because people cancel. And when the hotel is oversold, and has to cancel a reservation, guess which customers get their booking declined?
- nroets 3y ago
  When you are on booking.com you have the most choices. Not only between all the inventory they have, but you can also browse Airbnb. When you're at the hotel and you ask their rate, your basically committed. They know how much time and money it cost you to drive to a different hotel. So they can set their rates accordingly. And hotels hate reservations made over the phone or by email because too many guests never pay a deposit and never show up.
  - quantum2022 3y ago
    "And hotels hate reservations made over the phone or by email because too many guests never pay a deposit and never show up." I'm thinking you're from a non-western country, otherwise that would be false; the only wiggle room you gave yourself was 'booking through email' which I don't think even really exists, unless you meant booking through a hotel's website. Hotels take your deposit when you make a reservation over the phone or on their website, and if you book through booking vs the hotel the hotel will lose 15-30% of the room sale. You have just as many choices when looking for hotels, just look at google or bing. You don't need booking for anything. People rarely just drive up to hotels without checking hotels in the area beforehand on their phone. I'm not familiar with it in developing countries.
philjohn 3y ago
Part of the reason you should always have a long term holdout, so you can see how the win degrades over time, which with dark patterns, it can do.
dan-robertson 3y ago
In theory, the solution in situations like this is to continue the test over a longer time. Typically with ‘holdbacks’ – subsets of users who don’t get a feature for a long time. This is easier if you have an app that everyone uses because with a website, it’s harder to reliably find holdbacks who are also a representative sample (eg it won’t work so well to hold back everyone using the site in a certain language as those people may be statistically different from the general population in other ways). There are still a bunch of problems – higher maintenance burden, harder to iterate on a site quickly. Though I think you identify what I would consider the bigger problem which is that they cause political difficulties as a holdback can only really turn around and say that the positive impact people claimed wasn’t really borne out in the long term. So even at places that do holdbacks, the results may be silenced or ignored. If a holdback shows something continuing to work, that’s hard to get excitement about even though I think one should expect many of these a/b test results to not have long lasting effects.
- jvans 3y ago
  The types of effects you want to measure would take months or years to show and are the combination of many different small decisions. Teams need to apply common sense thinking, empirical data, and the willingness to wrestle with uncertainty. All of that is hard, blindly following a/b test results relieves people of that cognitive burden
- squirtlebonflow 3y ago
  Am I the only one who thinks THAT is a dark pattern? It would at a minimum, confuse me, if I had a different UI / feature set than my friend(s) - confused for being unaware why my experience looks different, and you better bet that 90%+ of users do not know what an A/B test is. At worst, I'm angry because they are getting a better experience and I randomly got shafted with no ability to upgrade.
  - esafak 3y ago
    That's the same kind of thinking that would criticize pharmaceutical companies for giving placebos. They do it because they don't know with confidence, and they want to. They are collecting evidence.
tetrisgm 3y ago
While I agree, the problem is in the data-driven decision-making. There is simply no good way to get good data for this long term type of effect. It's either going to be user interviews, some arbitrary score like customer satisfaction, or maybe a really convoluted split test (long ago cohorts vs new ones... but let's try to remove criteria so it's not apples and oranges). If a company culture only values data in its product decisions, this is always going to happen. Good luck fighting an argument against someone who has a supporting data point, and a revenue increase!
halo 3y ago
Real-life stores do this as well - make individually profitable changes (local maximums) that eventually make the whole experience so miserable that you no-longer want to shop there. Direct upselling is the most obvious, along with constant messing about with pricing and offers, as well as constantly pushing lower quality expensive own-brand goods, but the one I notice most is various cynically high pricing of non-core impulse items around checkouts - it tends to stick in my mind as the shop being expensive.
scott_w 3y ago
> I'm sure someone at booking a/b tested these things and saw an increase in revenue. I know people who used to work at Booking.com. That's exactly what happened. From what I can gather, you're not allowed to make changes to the site without running a test.
dr_kiszonka 3y ago
Spot on! I think we have a similar problem with TV / video ads. Many of them are funny or intriguing the first time you see them (so they perform well in "one-shot" tests and focus groups), but after five views they start getting exceedingly annoying. A counter example is a State Farm's ad with a guy sitting quietly on a couch and watching TV for a few seconds; I find it very effective.
j45 3y ago
I find this kind of feels like a squeeze on one's attention. Where every app or signup has this false premise seeded in a design that it alone is at the center of your digital existence and therefore you can do anything it takes to vie for and keep attention and engagement because the product may experience too much attrition otherwise. 1) Take color away from sites who abuse it and watch your attention and focus go through the roof. Grayscale the Web is a handy extension. https://chrome.google.com/webstore/detail/grayscale-the-web-save-si/mblmpdpfppogibmoobibfannckeeleag https://chrome.google.com/webstore/detail/grayscale-the-web-... It has a setting to just turn off all color, or just a site. 2) Add a firewall layer when signing up. I recently registered a voip number and an extra email account to check out for things. It stops explosions of emails and text messages on my space of the same try to chase people into engaging in a pipeline. It's surprising how few of these well oiled machines can do without being able to interrupt you constantly. 3) Notifications off, and checking them scheduled. A good way to test any product beyond it is to silence all notifications. Maybe I have a habit to keep going back, or not.

by jvans 3y ago

dyno12345 3y ago

see also: facebook

wpietri 3y ago

That is not a problem with A/B testing. It's a problem with the values of the company. I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." People who value revenue metrics over all else will still do shitty things for users even if they don't A/B test.

lylejantzi3rd 3y ago
The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.
- eropple 3y ago
  > Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values. If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making. Which means the company does not have those "good values" in the first place. They have lip service. Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.
  - hectormalot 3y ago
    I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white: - almost any change will have have people arguing for and against it - if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change - as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns) - as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact. Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.
    - thrashh 3y ago
      Still, that’s what sets a good manager from an average, no? No different with how there are relatively few good engineers but most are average or bad? I mean there are companies that put out good products but most are average at best. I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.
  - grogenaut 3y ago
    You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.
  - dahart 3y ago
    You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.
- projectazorian 3y ago
  This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.
  - lylejantzi3rd 3y ago
    > They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience. And how do you know the short term changes will degrade the brand and the user experience?
    - mitthrowaway2 3y ago
      Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!
  - chefandy 3y ago
    In my experience founders are often more than happy to toe the line between nudges and shoves.
- MarkSweep 3y ago
  This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.
  - dotancohen 3y ago
    You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
    - smachiz 3y ago
      isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
      - dotancohen 3y ago
        I suppose that is a way of looking at it.
    - bootsmann 3y ago
      Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
  - rdca 3y ago
    Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
    - esafak 3y ago
      And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
    - 3y ago
- HPsquared 3y ago
  "Seeing like a State" in action.
- wpietri 3y ago
  Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects. I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.
deleted 3y ago
[deleted]
antman 3y ago
There is also a mathematical problem of assigning events and actions to long term effects. The usual IT crowd unfamiliar with the respective literature will try to ab test and grid search out of the actual scientific part of data science. I had also fallen to that trap.
- computerfriend 3y ago
  What would you recommend reading on this?
jiggywiggy 3y ago
There is a grey line. I think only one room left is actually useful info. But yeah they push it too far. But most people in a company will be able to argue for themselves the info is useful and truthful so in their minds it's morally ok.
- toast0 3y ago
  Only one room left would be useful information if it were accurate. Most of the time, booking agents don't really have an accurate picture of inventory though. It's more like only one room available for booking.com to book right now. It could be that the hotel is holding back rooms for other channels or because they like to not be fully booked so far ahead of time or perhaps the hotel has found listing only one room at a time gets them a better look to book ratio (in part because of anxiety inspiring features like this). Without an understanding of the industry though, it's not really useful information.
chambers 3y ago
DHH, founder of Ruby on Rails and Basecamp, drew a line between core values and A/B testing https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5f562 https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5... > This is the tyranny of easy metrics. It's easy to measure how much money is saved by preventing cancelations, it's much harder to measure how much long-term business is lost by poisoning your reputation with the 99.9% of customers who had to jump hoops and dodge sleazeballs to get out of the subscription. But the latter could well be orders of magnitude money more over the long run.
jvans 3y ago
I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.
canadianfella 3y ago
[dead]
chefandy 3y ago
Yeah-- I think blaming dark patterns on A/B testing is like blaming thermostats for chilly houses. A/B tests just show if one thing does more of something you're testing for than something else. If you're just testing for "conversions" then you're going to make websites like booking.com. If you're testing to see if users are more stressed out by one situation or another, or testing to see if expert users are stymied by some interface abstraction designed to make things easier for less sophisticated users, then that's totally different.
- tspike 3y ago
  Sure, but it’s a lot harder to test for things like stress or being stymied than it is to test for conversion. Easy implementation + results that make money graphs go up will win every time without C-level involvement.
  - chefandy 3y ago
    It's a lot easier to break something with a hammer than bang in some nails. Using a marker to deface property is a lot easier than using it to draw a picture.
SilasX 3y ago
>I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." Then why were they testing it, if they already knew other concerns would veto that alternative?
- wpietri 3y ago
  I suspect if you were to think a bit, you could come up with some ideas yourself. If you're really stuck, let me know what you've got so far and I'll fill in the rest.
  - SilasX 3y ago
    I think I'll just wait for you to give the substantive reply you have in mind. Accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.
    - wpietri 3y ago
      You'll be waiting quite awhile. If there's anything HN has taught me, it's the meaning of "pearls before swine". And the futility in providing free labor for people who seem to think they're entitled to it.
      - SilasX 3y ago
        It's a discussion forum, amigo. You're supposed to share your insights with those who haven't yet achieved them. If you're not comfortable doing that, you can just not post instead of posting merely to whine and insult. Bookmarking for later.
        wpietri 3y ago
        I share with people who seem like they'll benefit from it. That includes people who have done at least a little work to understanding what's going on. It definitely doesn't include HN's legion of querulous jerks who mainly look for nitpicks, gotchas, and other opportunities to feel smart, even when that comes at the expense of the good discussion this place is nominally for. You get to influence which group people see you as part of.
dhosek 3y ago
When I worked at eHarmony there were two things that we were aware of above all else: 1. Users hated all the advertising even if they were signed in with a paid account and 2. eHarmony had no intention of stopping this because they valued the revenue from the advertising over the user experience. I noticed that Redfin has started adding ads to their site which really annoys me. Folks you’ll get a sizable commission for connecting me with the listing agent. The ad revenue has to be a rounding error compared to that. Why are you damaging your brand by showing me ads for luxury vacations next to houses that I’m looking at buying?
- wpietri 3y ago
  Great examples. To this list, I'd add Amazon's ads. It's providing them with very obvious revenue growth. It's parasitic on their ecommerce business, though, in ways that for me and many others, harm the things that made Amazon popular in the first place. That'll never change, though, because that revenue has enough people attached to it that they'll fight tooth and nail to keep it no matter how much it harms the rest of Amazon.
phendrenad2 3y ago
Companies need to balance between metrics and vision. It's easy to chase metrics and lose track of why your company exists. If you're just chasing metrics, what differentiates you from anyone else? What big bets are you making? Every company I've worked at has fallen into this trap, and it's gotten worse due to the tech slump.
smueller1234 3y ago
I think the problem underneath is quantitative fallacy: https://en.m.wikipedia.org/wiki/McNamara_fallacy https://en.m.wikipedia.org/wiki/McNamara_fallacy

GuB-42 3y ago

It is also one of these things where the first to do it gets the advantage. If you are the first to do the "only 2 rooms left" trick for example, you will get the full results before people get desensitized. But people will get desensitized everywhere, not just on your website, so if a competitor tries to pull the same trick, he won't get the same effect as you did. If fact, it may be time for you to roll back, to make competitors look bad for using the now well known and ineffective trick you invented. And if it works long term, then you get a head start.

mschuster91 3y ago
Yup, this "poisoning the well" effect is real, and it's blatantly uncompetitive - which is why it's so important for regulatory agencies to step in and act fast and hard so that this "first mover advantage" is eliminated. The problem is, regulatory agencies are slow as molasses and courts are overloaded with crap, which means by the time the process is done years later, the companies have long since switched to yet another sleazebag tactic.
- flappyeagle 3y ago
  Why? The tactics burn themselves out.
- Rufbdbskrufb473 3y ago
  I don't want government agencies involved in dictating the minutia of acceptable UI design for a private website. Businesses have brand reputations. Just let people vote with their wallets. That won't discourage all dark patterns, but is preferable to a world of over-regulation.
  - mschuster91 3y ago
    Businesses, particularly in oligopoly/platform scenarios, don't care about brand reputation. Just look at Meta - they don't even care about billions of euros worth of GDPR fines, it's all priced in.
  - 3y ago

listenallyall 3y ago

I don't disagree with you, certainly some "improvements" only have short-term positive effects. However, at least the A/B test was performed and data collected. Your assumption that it will eventually be "a massive negative" is pure speculation with no data behind it.

RussianCow 3y ago
The value of a "brand" is really, really difficult to measure objectively, but certainly has an effect on your revenue. It's hard to know in advance what's going to tarnish your reputation in the long run, but by the time you can measure it, it's far too late.

Freak_NL 3y ago

One of the effects this has on me is that I will use booking.com to find places (amongst other tools), but book directly with the accommodation. Only if the accommodation doesn't do its own booking will I use booking.com to book (about once every twenty places).

EVa5I7bHFq9mnYK 3y ago
I also had great results just walking up to the reception desk and booking there. Some deals are too good to be put online.
- vshade 3y ago
  But when you are taking vacations with your family you don't want the stress and the risk of not finding a good accommodation.
distances 3y ago
Every time I've checked this, direct booking was more expensive than Booking.com or Hotels.com.
- criddell 3y ago
  It often is, but I still book direct because I want to be a customer of the hotel and not of some third party. When things go wrong, I don't want to be be told to take it up with the third party (and this has happened to me).
- gramie 3y ago
  I book directly. If there is a problem with the reservation, hotels have much more flexibility to accommodate changes. You can usually cancel on the arrival date with no penalty, change room type, etc. I've heard others say that if you quote an online price, the front desk can match it (better to get 100% of the discounted price than 60%), although I haven't tried it myself.

wbobeirne 3y ago

We had someone from Booking.com come and speak at a company I was working at a good few years ago to talk about their testing process. They were using the multi-armed bandit approach[1] of just throwing dozens of changes at the wall and seeing which worked best. It definitely reflected in the UX. 1: https://en.m.wikipedia.org/wiki/Multi-armed_bandit https://en.m.wikipedia.org/wiki/Multi-armed_bandit

digitalengineer 3y ago
“Dozens of changes” but without values who gets to decide WHAT changes are tested? Sure there is UX/CRO research but without a moral compass it is exactly how you end up with dark patterns.
- wbobeirne 3y ago
  Agreed, and my impression was there wasn't a lot of scrutiny there. It was purely numbers driven, if it could be tested and measured it would, and if it won it was adopted. Very little emphasis on UX, product cohesion, or any specific design principles.
sizzle 3y ago
Can’t believe they are giving talks on this like it’s innovative or something when admiring. I’d honestly be embarrassed saying that my design process was to make shit up and see what happens instead of investing in UX design/ research professionals can run generative studies to better understand their consumer base and sentiments towards booking trips online, like Airbnb heavily invested and can be seen as having a world class user experience next to booking.com I digress…

marcinzm 3y ago

Many companies in my experience do test for these things. They remove old features and measure impact periodically, or run a long term hold out bucket, or some other such approach. The deep dark secret of the web is that there often there isn't a negative impact that can be measured no matter how much people try to measure one.

newaccount74 3y ago

Booking.com lost me as a customer for life after I fell victim to their sleazy tactics a few times. I have refused to book anything on booking for the last few years because I didn't want to be mislead into booking a crappy hotel by their algorithms again. I guess my decision doesn't matter much in the grand scheme of things, because they are still around and judging by the screenshots it's as bad as ever, and people still use the site...

FireBeyond 3y ago
Yup. Go now to booking.com and pick a remote roadside motel in Wyoming for a midweek stay sometime in March of 2024. And watch booking.com try to tell you that "19 people have booked this today" or some such bullshit.
- sizzle 3y ago
  I’m surprised California hasn’t outlawed this tactic as deceptive marketing or such? Any idea why companies can falsify data for the sole purpose of deceiving consumers to increase their revenue?
  - iruoy 3y ago
    There are no laws against it in the US. In the EU there are some[0][1] now. [0]: https://www.acm.nl/en/publications/bookingcom-commits-adjusting-its-website-following-action-european-consumer-authorities https://www.acm.nl/en/publications/bookingcom-commits-adjust... [1]: https://www.acm.nl/en/publications/booking-and-expedia-inform-consumers-more-clearly-about-their-offers-following-action-european-consumer-authorities https://www.acm.nl/en/publications/booking-and-expedia-infor...
- woobar 3y ago
  I've just tried this and now they show me identical room for the same price with two different nags. Not sure if they have only 2 or 5 (or 7?) "Queen Studios" left. https://imgur.com/a/oqOTYlz https://imgur.com/a/oqOTYlz
- 3y ago

shswkna 3y ago

I have a personal experience where booking.com’s nudging caused me to reconsider my trip. I was trying to find something suitable to stay in Paris. Maybe it was the exaggeration of booking.com or maybe there was some truth, but at some point I shut down and made a 180 on my plans. I had realised that I don’t want to go somewhere where I have to compete against this avalanche of other visitors who were or were not snapping my accommodation options away. I am now visiting friends in another European city.

patneedham 3y ago
But did you still use booking.com for that other European city trip, or another platform?
- deleted 3y ago
  [deleted]
- shswkna 3y ago
  I am staying with friends, who have a spare room available.
schneems 3y ago
I had a similar reaction to Lyft’s “you have 2 minutes to accept this faster trip” prompt and emphasizing the faster, more expensive option first. I saw that, balked a bit at the interaction and ended up taking a train instead. Not only was it $6.25 instead of $46 it got me there faster than Lyft’s fastest option. Including time walking to and from the station. I wasn’t in a hurry but the in your face “look how much money people are willing to spend to save 5 min” helped me rethink my priorities.
- OJFord 3y ago
  Similarly (though also exhibiting teenage American language taking over the world symptoms) is Netflix's new (or new-ish, I just noticed it yesterday as a fairly frequent user) 'everone is watching' category. I have absolutely zero interest in that, it's literally repulsive, trivially incorrect, and just plain stupid even apart from the language - who cares what others have chosen to watch, how does it compare to what I've given thumbs up/down?

andix 3y ago

I really startet to hate using Booking.com, especially because every time after using it, they start bombarding me with emails. I could probably turn them off. Another thing that makes me laugh now: I often go to the same hotel, and Booking.com provides better rates then booking directly (no idea why, I asked multiple times for the same discounts). And for my favorite category the hotel has only one room. So booking.com constantly warns me "only 1 room left!". Yeah, I know, there is only one ;)

vkou 3y ago
> (no idea why, I asked multiple times for the same discounts) Hotels sell a fraction of their inventory cheaply to resellers. Also, they oversell, because people cancel. And when the hotel is oversold, and has to cancel a reservation, guess which customers get their booking declined?
nroets 3y ago
When you are on booking.com you have the most choices. Not only between all the inventory they have, but you can also browse Airbnb. When you're at the hotel and you ask their rate, your basically committed. They know how much time and money it cost you to drive to a different hotel. So they can set their rates accordingly. And hotels hate reservations made over the phone or by email because too many guests never pay a deposit and never show up.
- quantum2022 3y ago
  "And hotels hate reservations made over the phone or by email because too many guests never pay a deposit and never show up." I'm thinking you're from a non-western country, otherwise that would be false; the only wiggle room you gave yourself was 'booking through email' which I don't think even really exists, unless you meant booking through a hotel's website. Hotels take your deposit when you make a reservation over the phone or on their website, and if you book through booking vs the hotel the hotel will lose 15-30% of the room sale. You have just as many choices when looking for hotels, just look at google or bing. You don't need booking for anything. People rarely just drive up to hotels without checking hotels in the area beforehand on their phone. I'm not familiar with it in developing countries.

philjohn 3y ago

Part of the reason you should always have a long term holdout, so you can see how the win degrades over time, which with dark patterns, it can do.

dan-robertson 3y ago

In theory, the solution in situations like this is to continue the test over a longer time. Typically with ‘holdbacks’ – subsets of users who don’t get a feature for a long time. This is easier if you have an app that everyone uses because with a website, it’s harder to reliably find holdbacks who are also a representative sample (eg it won’t work so well to hold back everyone using the site in a certain language as those people may be statistically different from the general population in other ways). There are still a bunch of problems – higher maintenance burden, harder to iterate on a site quickly. Though I think you identify what I would consider the bigger problem which is that they cause political difficulties as a holdback can only really turn around and say that the positive impact people claimed wasn’t really borne out in the long term. So even at places that do holdbacks, the results may be silenced or ignored. If a holdback shows something continuing to work, that’s hard to get excitement about even though I think one should expect many of these a/b test results to not have long lasting effects.

jvans 3y ago
The types of effects you want to measure would take months or years to show and are the combination of many different small decisions. Teams need to apply common sense thinking, empirical data, and the willingness to wrestle with uncertainty. All of that is hard, blindly following a/b test results relieves people of that cognitive burden
squirtlebonflow 3y ago
Am I the only one who thinks THAT is a dark pattern? It would at a minimum, confuse me, if I had a different UI / feature set than my friend(s) - confused for being unaware why my experience looks different, and you better bet that 90%+ of users do not know what an A/B test is. At worst, I'm angry because they are getting a better experience and I randomly got shafted with no ability to upgrade.
- esafak 3y ago
  That's the same kind of thinking that would criticize pharmaceutical companies for giving placebos. They do it because they don't know with confidence, and they want to. They are collecting evidence.

tetrisgm 3y ago

While I agree, the problem is in the data-driven decision-making. There is simply no good way to get good data for this long term type of effect. It's either going to be user interviews, some arbitrary score like customer satisfaction, or maybe a really convoluted split test (long ago cohorts vs new ones... but let's try to remove criteria so it's not apples and oranges). If a company culture only values data in its product decisions, this is always going to happen. Good luck fighting an argument against someone who has a supporting data point, and a revenue increase!

halo 3y ago

Real-life stores do this as well - make individually profitable changes (local maximums) that eventually make the whole experience so miserable that you no-longer want to shop there. Direct upselling is the most obvious, along with constant messing about with pricing and offers, as well as constantly pushing lower quality expensive own-brand goods, but the one I notice most is various cynically high pricing of non-core impulse items around checkouts - it tends to stick in my mind as the shop being expensive.

scott_w 3y ago

> I'm sure someone at booking a/b tested these things and saw an increase in revenue. I know people who used to work at Booking.com. That's exactly what happened. From what I can gather, you're not allowed to make changes to the site without running a test.

dr_kiszonka 3y ago

Spot on! I think we have a similar problem with TV / video ads. Many of them are funny or intriguing the first time you see them (so they perform well in "one-shot" tests and focus groups), but after five views they start getting exceedingly annoying. A counter example is a State Farm's ad with a guy sitting quietly on a couch and watching TV for a few seconds; I find it very effective.

j45 3y ago

I find this kind of feels like a squeeze on one's attention. Where every app or signup has this false premise seeded in a design that it alone is at the center of your digital existence and therefore you can do anything it takes to vie for and keep attention and engagement because the product may experience too much attrition otherwise. 1) Take color away from sites who abuse it and watch your attention and focus go through the roof. Grayscale the Web is a handy extension. https://chrome.google.com/webstore/detail/grayscale-the-web-save-si/mblmpdpfppogibmoobibfannckeeleag https://chrome.google.com/webstore/detail/grayscale-the-web-... It has a setting to just turn off all color, or just a site. 2) Add a firewall layer when signing up. I recently registered a voip number and an extra email account to check out for things. It stops explosions of emails and text messages on my space of the same try to chase people into engaging in a pipeline. It's surprising how few of these well oiled machines can do without being able to interrupt you constantly. 3) Notifications off, and checking them scheduled. A good way to test any product beyond it is to silence all notifications. Maybe I have a habit to keep going back, or not.