11 ms·

TLDR: this post is entirely to calm advertisers, it can be boiled down to "don't worry, we still prioritize brand safety like before". It does have this gem in

TLDR: this post is entirely to calm advertisers, it can be boiled down to "don't worry, we still prioritize brand safety like before". It does have this gem in it: "What has changed, however, is our approach to experimentation. As you’ve seen over the past several weeks, Twitter is embracing public testing. We believe that this open and transparent approach to innovation is healthy, as it enables us to move faster and gather user feedback in real-time. We believe that a service of this importance will benefit from feedback at scale, and that there is value in being open about our experiments and what we are learning. We do all of this work with one goal in mind: to improve Twitter for our customers, partners, and the people who use it across the world." What a weird thing to say... A/B tests are a thing, does anyone buy that experimenting with new things by rolling out new features to all users at once is a good strategy?

jzelinskie 4y ago
Twitter was previously doing A/B tests. Spaces, communities, downvotes, circles, and even moderation were applied only to a subset of accounts while they actively experimented. The new policy is an objectively less scientific approach to testing functionality than what was occurring before.
- generalizations 4y ago
  It's not like A/B tests approach anything like the rigor that we expect from actual science, though.
  - squaredot 4y ago
    It's about marketing, the A/B testing they do, not science.
    - sithlord 4y ago
      what is not scientific about it? maybe its not good science, but it is science. really almost anything is science... Science is just observation and experimentation. Science doesn't dictate how you do the above. Now, someone would find it impossible to reproduce your findings, but - that would just suggest bad science
      - Sohcahtoa82 4y ago
        Incorrect conclusions are frequently drawn from A/B tests. To some, this makes it unscientific, to others, it just means it's bad science. I think the argument is more semantic than objective. For example, if your metric is "time spent interacting on the platform", then a testing of a rollout of a feature ends up with longer page load times, so users spend more time there because they're waiting for pages to load would increase that metric, and management decides it's a good idea.
      - ska 4y ago
        > Science is just observation and experimentation. That's not enough. If you don't include some sense of both 'systematic' and 'rigorous' (and yes, these terms are slippery), you aren't doing science.
        nvrspyx 4y ago
        But both of those qualifiers exist on continuums and where an experiment lies on them is subject to opinion, so there's no singular threshold. It means it would depend on the person reviewing the experiment whether it's "systematic" and "rigorous" enough to (personally) be considered science. What you're describing is the quality of the science, not whether or not it is science. Until you can objectively measure how "systematic" and "rigorous" an experiment is, your definition of science only applies to you.
        ska 4y ago
        "Experiment and observation" is something we know humans have been doing for literally thousands of years (and without written evidence, probably 10s-100s). At least for loose versions of "experiment" which ties back to rigor. We haven't been doing science for very long. The primary difference is the desire and effort to add rigor and systematic thinking. The difference in efficacy is hard to understate.
        nvrspyx 4y ago
        > We haven't been doing science for very long. Again, this comes back to your definition. Many would disagree in that science has existed for, at least, as long as recorded human history because an actual tenable definition of science is something along the lines of "the endeavor to build knowledge by experimenting and observing the results". The rigor of the experiment is part of the quality of the science, not whether it's science itself. No one is arguing that rigor isn't important to good science. It is important because rigor lends to reliable and valid results. What we ultimately want is results that are reproducible and can be used to predict. If you observe bad results, it's because you did a bad experiment, thus bad science, not that you didn't do science at all. As an analogy: if I took notes during a meeting that no one can understand or use, that doesn't mean I didn't take notes. It just means that I took bad notes.
  - radicaldreamer 4y ago
    It wasn't "scientific" to begin with but what's happening right now is pretty clearly panicked throwing stuff at the wall based on Elon's intuition and day to day demands... while that may have worked for him in the past, I don't think it's going to work as well in this domain where you're working with a complex system dependent on millions of people's behaviors and incentives.
    - dragonwriter 4y ago
      > while that may have worked for him in the past Did it? The nature of Tesla and his other current businesses buffer that a bit even if it has been his approach, and it seems to have gotten him thrown out as CEO at X.com twice; among the things going on Twitter seems to be Musk trying to relitigate his failure at X.com without other investors being in a position to kick him out, but he seems to be piling up existential threats without resolving them.
      - radicaldreamer 4y ago
        It does kind of seem like he's addicted to dancing on the edge of cliffs
  - thewataccount 4y ago
    > actual science I'm not sure what you mean by this. Many quality studies are A/B tests. A/B just refers to the two IV states you're testing, which you're then observing a DV - sales, engagement, errors, etc. A/B tests can be double blinded (don't tell the error monitoring people which results are from a trial), and have high number of samples, far beyond even most pharmaceutical trials. They can also be really crappy, changing too many variables at once, etc. But they are certainly "real science". EDIT: an example, Drug vs placebo - is an A/B test.
    - compiskey 4y ago
      Advertisers and marketers use the equations but measure contemporary trends. Science is more “what’s true if humans didn’t exist.” Marketing is more “what widget generates more revenue?”
      - typest 4y ago
        This isn’t true. Science is ultimately the scientific method — make a hypothesis, test a change, observe the results, repeat. It’s an algorithm for learning and broadly gaining information about reality. It can equally be applied to things having to do with humans and things not having to do with humans.
        sdrinf 4y ago
        Not gp, but there's a significant kink when this applies to humans; namely, that humans have the ability to reflect on publicly known outcomes, and change their behavior en-masse in light of information so gained. I put this earler in the phrase "reflection completeness": https://sdrinf.com/reflection-completeness https://sdrinf.com/reflection-completeness ie there are things which stops working when people know about it. In particular with A/B testing, this means that the initial A/B test is intermingled from at least 3 effects: specifically it measures how the naive population's behavior changes as a function of new functionality being made available. This is heavily, heavily time-dependent; specifically there's a "novelty effect" (early data collection will not be representative to long-term usage patterns); and there's "reflection effect" (once the outcome of the test is widely known, people can change their behavior based on that). Controlling for the first is difficult, but possible; controlling for the second, beyond just "keeping everything secret", is significantly more so, as the timelines for that might be years in length. I strongly suspect GP was pointing at this timeline factor, and specifically that market engineering, as currently, generally, widely practiced, is grounded on the immediately available signal of "does it increases sales in 2 weeks of A/B test running". Which, given novelty effects, is heavily biased towards "yes"; and these people aren't incentivized (nor have the time/energy) to measure _very_ long-term effects beyond novelty, and reflection period.
  - arrrg 4y ago
    That’s not at all an argument for dropping AB testing, it’s an argument for being more rigorous, especially since in principle the circumstances under which services with a large active user base can test are downright luxurious. Sociologists will frequently not have as good access to such a large participant pool under near ideal experimental conditions with such good ways to observe behavior. And the stuff you have to keep in mind when running experiments is not terribly complex. A bit of statistics, a few things you absolutely have to get right, that’s it. Obviously there are reasons why AB tests are often not run rigorously (statistical illiteracy and pressure to get things done quick as well as to produce tangible results as often as possible – all three of which might lead you to run underpowered experiments with too few participants and to stop testing early which will lead to too many false positives). However, stopping to do experiments (and instead just releasing new stuff and observing the reaction) isn’t really an improvement that leads to better outcomes compared to that.
  - joshuamorton 4y ago
    Most scientists likely wish they could get the rigor of a company like Twitter's a/b testing suite.
    - HWR_14 4y ago
      Why wouldn't Twitter (or similarly large companies) have open-sourced their a/b testing suite? It's not like the math there is proprietary. I mean, I'm sure the parameters to the math are proprietary. But the basic math seems simple enough.
      - joshuamorton 4y ago
        If you're not using an off the shelf one, chances are it's tightly integrated to your application framework. Trying to tease out the pieces that aren't coupled to Twitter's User class is probably more effort than it's worth
        HWR_14 4y ago
        That makes sense. Thanks. I guess the comment I read implied Twitter had an amazing A/B test suite, as opposed to a tightly specialized A/B test suite.
        simeonf 4y ago
        It's called DuckDuckGoose - see https://blog.twitter.com/engineering/en_us/a/2015/twitter-experimentation-technical-overview https://blog.twitter.com/engineering/en_us/a/2015/twitter-ex...
        cornel_io 4y ago
        And this is sadly true about almost every good A/B testing system out there. They always involve tooling that has to exactly fit your stack, and it's really tough to make a general purpose product that would offer anywhere near the same value. To some extent, you grow your company and codebase around your A/B testing system, not the other way around, because it has to slot into so many places: deployment, testing, front-end, back-end, analytics, monitoring, etc. Almost no two companies share the same stacks across all of those dimensions, and an A/B testing system that doesn't hook in tightly to every one of those systems is not complete. This is why I always get a bit scared when people think they can just use one of those "drop-in" services and be done with it: yeah, great, you can now fiddle your JavaScript from some third-party website, but you're going to have a big project ahead of you passing group assignments out to all of the different systems that will need to know about them, and almost guaranteed some part of your stack will not have a library available from the service you picked so you're going to be writing your own REST wrapper, and dammit they don't document that API very well and it seems to be responding differently since the last update, and man it'd be a lot easier if I could just pipe results straight from my own service into Big query rather than running a daily user export, and damn, their dashboard doesn't let me set exclusion criteria on my own metrics, I have to send activation events now, and etc, etc. By the end you've basically built your own A/B test system from scratch, you've just paid someone else to do the "int myGroup = Random.next()" call, which is the easiest part to build.
  - root_axis 4y ago
    Based on what reasoning? Data quality with respect to computer systems approaches perfection, it is far more reliable than any data you can hope to approximate in the real world. There is also plenty of "actual science" with terrible data quality, hence the replication crises spanning large sections of the scientific landscape.
  - fullshark 4y ago
    You are likely putting "actual science" on a pedestal here
  - Waterluvian 4y ago
    If "actual science" is a real thing, then absolutely nothing humanity has ever done in the name of scientific endeavour is "actual science," given every experiment exists on a continuum of trade-offs. A case could be made that A/B testing is insufficiently rigorous given specific goals, resources, limitations, context, etc. But that case isn't being made here.
    - williamstein 4y ago
      Some pure mathematics is actually rigorous.
      - Waterluvian 4y ago
        Hah! You're fast. I was about to comment on that: "actual math" probably exists. "actual science" probably doesn't. I'm not sure any experiment can ever be free from error, uncertainty, trade-offs. (And now I'm super curious about this...) Thanks though. I've narrowed my original comment to more accurately represent the scope I am referring to.
        philippejara 4y ago
        to be fair whether mathematics is a science or not is debated[0], so your initial claim might very well stand true depending on which side one takes in the argument. [0]:https://philosophy.stackexchange.com/questions/10178/is-mathematics-an-art-or-a-science https://philosophy.stackexchange.com/questions/10178/is-math...
        Waterluvian 4y ago
        That's fair. Though that gets into semantics which I don't find interesting or fun. In this context, I think it's fair to say we're talking about "things that can be experimented on to learn more things."
  - cornel_io 4y ago
    If they're done right, they are exactly as rigorous as we'd expect from science, since they are literally the same as randomized control trials. You can do even better with A/B testing because you have much tighter control over inclusion criteria, treatment compliance, and outcome analysis.
  - dredmorbius 4y ago
    Others have addressed the ways in which A/B testing does hew quite closely to the standard of empirical observation under controlled circumstances, with which I largely agree. Where A/B studies may go wrong in my view is a few other elements: - A/B studies have difficulty in determining differences based on multiple interacting characteristics. In fairness, so does empirical science, and the principle of "holding all else constant" is a frequent assumption of scientific processes. - A/B studies face an inherent self-selection / exclusion bias: the participants in this round of A/B testing are those who've not been driven off the project/product from past experiments and design changes. Given that many Web 2.0 companies eventually dance with pushing people right up to the border of tolerance, it's quite possible that A/B testing has a long-term effect of pushing those participants whose tolerance has been exceeded out of the study population entirely. I don't know how large a factor this is, though loud / rage quitters are certainly a prominent (if not necessarily large) cohort. Whether or not they're also influential, or perhaps more importantly when they become influential is another question. Again, this is a fairly common problem with any social experiment, including natural social experiments, see various forms of brain-drain and social flight. - A/B testing tends to focus on short term changes and behaviours, which may mask longer-term outcomes. This has some overlap with the above, but also with subjects' general response to change. See the classic case of this in the Hawthorne Effect (<https://en.wikipedia.org/wiki/Hawthorne_effect https://en.wikipedia.org/wiki/Hawthorne_effect>). The upshot is that A/B testing can be valid and useful, but that experimental design, particularly in the case of social and psychological experiments, where subject feed back into the study and its methodology itself, is exceptionally thorny.
    - TeMPOraL 4y ago
      > it's quite possible that A/B testing has a long-term effect of pushing those participants whose tolerance has been exceeded out of the study population entirely. I'd compare this to how evaporation rate increases with temperature, as more particles find themselves with enough energy to escape the liquid. From my personal experience, even if I can tolerate a lot of UX abuse, each such "optimization" lowers my threshold of switching to a competitor. Software in general, and SaaS specifically, resists commoditization, but every now and then an actual alternative to a product/service I'm using shows up - and whether or not I switch (and when) is correlated with how much I resent the incumbent for their UX "improvements". I'd add one bullet point to your list: - Unlike regular scientific experimentation, A/B testing is a methodology primarily spread in business circles using regular hype channels. That is, the average practitioner is not qualified to execute it correctly, which is one of the reasons I see A/B testing more as tools to launder arbitrary decisions. Because consequences of doing it wrong are typically not immediately apparent or obvious, both companies and customers suffer (and a vast space for fraudsters is created). I'm in a charitable mood, so I'm not passing judgement on people for not having PhD-level understanding of statistics - just pointing out that, to the degree much larger than in sciences (even soft ones, which suffer some of the same structural problems), there's little pressure to do such tests correctly (and there's lot of ways to make money or status by doing them without regards for correctness). From what I hear, a common way of executing A/B test badly and getting bullshit results, is by terminating the test early when it shows the relevant metrics improving for the test group - vs. running it longer if no big improvements are observed (or the metrics start getting worse for the test group). This biases the experiment towards giving false positives. This problem was big enough that there was a debacle around Optimizely few years ago, whose UI was accused to promote this early termination of tests. The cynical take I'm still somewhat partial to is that it wasn't an accident (if not done on purpose, then possibly... a result of an A/B test!) - false positives make the (statistically naive) users feel they're getting more value from Optimizely than they actually are.
- sayrer 4y ago
  A/B tests are pretty problematic for social networks, since anything effective can also influence the control group. They are excellent for single-player activities like shopping carts and signup flows. "Network Experimentation at Scale" from Facebook describes how difficult this problem is. Most A/B test frameworks don't reach this level of sophistication. It does make some sense to just ship things if you don't have time to build out something like that. (disclosure: I worked at Twitter long ago)
  - throwawaymaths 4y ago
    They're also problematic because you'll likely optimize for whatever metric you pick with no regard to the ethics of the optimization. Sometimes a double blind test is not the right thing to do.
    - sayrer 4y ago
      I've never really bought this argument. There might be unethical growth hackers in the company, but A/B tests are not the problem in that situation. In my experience, they mostly just catch bugs. Stuff like "hmm, our much better looking signup flow underperforms... oh, the form is broken on Safari." That kind of thing.
      - solarkraft 4y ago
        Think of the AI optimizing for a stated goal while ignoring implied constraints (e.g. eradicating humanity to stop wars). This can be the case for A/B testing. Sure, you can increase ad clicks by 30% ... if you trick the user into clicking it through a carefully timed layout jump. I think GP's argumentation may go in this direction. I'd probably not say A/B testing is the problem itself, it is a tool after all, but I could imagine it's sometimes not used very well. Another point: Spotify's core flow changes so much (feels like almost daily) that I've lost all confidence in using it.
      - P5fRxh5kUvp2th 4y ago
        To make it easier for everyone to understand the other posters point... a double blind test can help you determine the most effective way to cause pain to a monkey, but it will never answer the question of whether you should be doing so.
- ilyt 4y ago
  I feel that we can hardly call that random changes "policy"...
- inquirerGeneral 4y ago
  You are suggesting that A/B testing is going away rather than this being simply an addition to widespread testing.
- racked 4y ago
  They've probably fired so many people that they wouldn't even know _how_ to run A/B tests even if they wanted to.
aaroninsf 4y ago
It's only weird, if you attempt to parse it with a straight face. This is some B-grade best-effort spin on what has been uncontrolled chaos with predictably awful effects. The only thing keeping Twitter rolling is the majority percentage of the casual niche user in non-political and non-technical niches, who haven't been paying attention to the chaos; and who by virtue of being casual users are not notably surprised at everything that has broken both culturally, wrt safety and content, and technically. Unfortunately for Leon the money comes from corners who HAVE been paying attention and not only see what's happened, and ongoing—they see through this kind of comedic college-try at handwaving around it.
- sf_rob 4y ago
  ̶u̶n̶s̶t̶a̶g̶e̶d̶ ̶r̶o̶l̶l̶o̶u̶t̶s̶ ̶o̶f̶ ̶f̶e̶a̶t̶u̶r̶e̶s̶ ̶t̶h̶e̶ ̶C̶E̶O̶ ̶t̶h̶i̶n̶k̶s̶ ̶a̶r̶e̶ ̶c̶o̶o̶l̶ ̶ public testing/experiments
- phpisthebest 4y ago
  Twitter has never been more fun, useful, and engaging for me. I signed up for an account after years of refusing to do so after elon took over
  - bvasilis 4y ago
    Then, if I may ask, how do you have an idea of what it was before?
    - phpisthebest 4y ago
      I was a lurker before, I read twitter all the time with out an account. It as a pain because they kept trying to force you to create an account so I kept having to find ways around their account signup walls which was fun...
      - baking 4y ago
        What's funny is they no longer try to force you to create an account. I used to use nitter.net to browse someone's history, but now I can do it on Twitter apparently with no restrictions.
janoc 4y ago
How is that different from Tesla essentially using Tesla car owners as unpaid betatesters for their autonomous driving (and other) software while disclaiming all responsibility when anything happens? This "learn by doing" is going to end up well ... not. Especially when trying to learn by repeating mistakes with completely foreseable consequences, like that blue "verified" badge being available for anyone who pays without any verification.
- w0m 4y ago
  > using Tesla car owners as unpaid betatesters Almost right. Charging Tesla owners to beta test their autonomous driving software.
  - dexterdog 4y ago
    Almost fully right. Charging and risking Tesla subscribers to beta test their autonomous driving software.
    - lancesells 4y ago
      So close. Charging and risking Tesla subscribers along with everyone else on the road to beta test their autonomous driving software.
      - LanceJones 4y ago
        I'm one of those people. Running FSD Beta in my Model S Plaid in BC. And your negative connotation doesn't apply to me or many/most of the other beta testers with whom I'm regularly communicating.
  - yreg 4y ago
    It's curious how many people are unhappy that someone choses to pay for an early-access product that's in beta.
    - 4y ago
14u2c 4y ago
Maybe they are referring to Elon's recent unhinged polls as testing.
- giantrobot 4y ago
  Hey: vox populi, vox dei! It's Latin so it must be smart. Apparently Twitter had some huge problem with bots when Musk was trying to get out of the purchase. Thankfully he solved the bot problem after the purchase so he could run polls and really get the will of the people and not, you know, all those previously problematic bots.
  - bryananderson 4y ago
    Except when a poll gets a lot of votes that he doesn’t like, in which case he says “looks like the bots are out in force today!”
    - giantrobot 4y ago
      Good ol' Schrödinger's bots.
  - edavis 4y ago
    > It's Latin so it must be smart. It's perfect, in a way, that the full quote is "Nec audiendi qui solent dicere, Vox populi, vox Dei, quum tumultuositas vulgi semper insaniae proxima sit." which translates to "And those people should not be listened to who keep saying the voice of the people is the voice of God, since the riotousness of the crowd is always very close to madness." Refs: https://twitter.com/cagrimmett/status/1595967787339743232 https://twitter.com/cagrimmett/status/1595967787339743232
vineyardmike 4y ago
> What a weird thing to say... They need to explain musks antics somehow. More polite wording than idve used.
Toxide 4y ago
Hey remember all those terrible things we banned? Lets experiment by letting them free again! Weeeeeeeeeeee.
- s2radhak 4y ago
  for our next experiment, we should re-introduce lead into paint
  - fortuna86 4y ago
    looking into this
    - ss108 4y ago
      The science on it is unsetttled and ideologically biased because I don't like the policies people justify on its basis.
nicce 4y ago
It might be just an escape plan when Musk uses platform for his own needs. Banning users he does not like or allowing certain speech when filtering others and so on….
davesque 4y ago
I think this new idea of "public testing" is really just a post hoc recasting of Elon's failed Twitter Blue rollout. Calling it by those words is...creative? But the failure was less an experiment as it was a lesson in humility.
- dragonwriter 4y ago
  > I think this new idea of "public testing" is really just a post hoc recasting of Elon's failed Twitter Blue rollout. Calling it by those words is...creative? Its just a diplomatic rephrasing of Elon’s “do lots of dumb things” tweet: https://twitter.com/elonmusk/status/1590384919829962752?t=ccBHJovjIC1jFCyGrh5gfw&s=19 https://twitter.com/elonmusk/status/1590384919829962752?t=cc... But advertisers who have pulled out because of distrust and lack of stability aren’t likely to be reassured by rationalizations for the policy instability, they’ll just be confirmed in their decisions to wait to see how things shake out. And regulators concerned about noncompliance with binding rules aren't going to care about a PR rationalization at all, except insofar as it provides evidence that the failures were intentional rather than inadvertent.
- unsui 4y ago
  Exactly. It's just a CYA statement, basically saying "oops, I meant to do that...", and leaving the door open to make more oopsies as intentional "experiments".
- renewiltord 4y ago
  A typical HN comment on a SpaceX post might have once looked like: > Unfortunately, the "move fast and break things" attitude doesn't work with rockets. You will likely find this comment verbatim from the last 10 years. Given that is so, I can't see how someone could claim that Elon Musk is new to visible experimentation.
  - palata 4y ago
    Well here it sounds more like "break things and be an asshole", to be honest. I wonder what would have happened, had he fired 75% of the engineers when he took control of Tesla /s.
sharkjacobs 4y ago
It sort of makes sense if the experiment is about judging not just individual user reactions but the wider reaction of the media, advertisers, "the conversation" to a change. I don't think that makes it a good idea though, seems like each failed "experiment" poisons the pool and certainly makes it possible to do experiments in isolation.
memish 4y ago
It's just referencing the fact that the new Twitter is more open and transparent. I can't believe how many people are spinning this as a bad thing. We heard almost nothing from Parag and little from Jack when he was running it and experiments were opaque from the outside. Now we're hearing from the CEO and employees like George Hotz about what they are doing and planning, and they're involving the community, asking for feedback directly.
- JumpCrisscross 4y ago
  > previous Twitter wasn't as transparent as it is now Reinstating accounts on the basis of a poll, on a platform you have spent months railing for having too many bots, is a good example of CYA transparency.
  - spamuel 4y ago
    Elon specifically mentioned that his polls were actually a great way to obviate bots, as reflected by the bot activity seen on them. He was going to reinstate the accounts anyway.
    - ilyt 4y ago
      That sounds like excuse after the fact someone called him on his bullshit
  - h3rsko 4y ago
    And are all the bots expected to vote one way?
    - tptacek 4y ago
      That's your response? That polling is transparent because both sides will just turn it into a game of CoreWars, and may the best argument win (at CoreWars)?
JumpCrisscross 4y ago
> this post is entirely to calm advertisers Regulators, too [1]. [1] https://www.ft.com/content/a07ca1ae-9f9a-46ee-9457-27bb30e18ed2 https://www.ft.com/content/a07ca1ae-9f9a-46ee-9457-27bb30e18...
- dev_tty01 4y ago
  Yes, but "[w]e believe that a service of this importance will benefit from feedback at scale" will just make advertisers unhappy. To state the obvious, I don't think they are reading the room very well...
zzzeek 4y ago
the marketing team writes a Grown Up , Adult blog post to try to look normal, then commander Elon will come out tomorrow high fiving more nazis [1] and banning more non-nazi accounts for supposedly belonging to "antifa" and saying things he doesn't personally like [2]. This is the same "trapped enabler" pattern we saw with Trump in the early days, with his staff constantly coming out to try to paper over whatever horrible thing he did. They had no shame, and neither does the Twitter staff that wrote this blog post. [1] https://archive.ph/xYeYY https://archive.ph/xYeYY - TPM article without paywall [2] https://theintercept.com/2022/11/29/elon-musk-twitter-andy-ngo-antifascist/ https://theintercept.com/2022/11/29/elon-musk-twitter-andy-n...
everdrive 4y ago
I largely agree with your point, but I have to say as a user I generally don’t enjoy A/B testing at all. It’s invisible to me, and some part of the app or website I’m using is now either broken or annoying in some way, but still works normally for the people I mention it to, since they’re not in my group. Additionally, the changes are sometimes totally transient, so things are disrupted for me briefly, only to be distrusted again.
- bombcar 4y ago
  A/B testing when it's like the color of something (did you know you can change the hacker news banner color?) doesn't bother me much, but when it's more invasive and I can't turn it off or switch teams, it starts to get annoying. Especially when you're trying to help someone and they are seeing something different from what you see.
s2radhak 4y ago
that blurb you quoted is just twitter rationalizing for Elon's red-bull fueled tweets to "improve twitter"
- superfrank 4y ago
  Lol, red bull... Elon's late night tweets always make me think of that scene in The Office where the new CEO (James Spader) decides to close one of the branches without telling anyone and when asked about it he goes "I got into a case of Australian Reds... and... How should I say this, Colombian whites"
cpr 4y ago
Where does it say or even imply "rolling out new features to all users at once"?
lesuorac 4y ago
> TLDR: this post is entirely to calm advertisers, How though, Twitter has had week(s) to prepare this post and its so bare. > First, none of our policies have changed. ... The team remains strong and well-resourced. If you're Eli Lily, why would you re-advertise on twitter. Nothings changed from when you stopped!
- fortuna86 4y ago
  > If you're Eli Lily, why would you re-advertise on twitter. Nothings changed from when you stopped! Not only would I not buy ads, i'd rethinking even having a corporate presence considering the lack of protections against fraud.
  - insin 4y ago
    Looking forward to a verified @RealEliLilleyCEO account appearing 5 minutes after Twitter Blue 2.0 3.0 comes out, which only allows you to trivially impersonate people, after which Musk will completely independently invent Verification 2.0 3.0 which will either just be the old system again or a separate Twitter Blue badge. Visionary genius at work.
- andreyk 4y ago
  Oh I agree, this post won't change anything at this point. But I guess that's why it was released (and as others noted, also for regulators / to save face after weeks of chaos).
NaturalPhallacy 4y ago
Musk literally said "we're going to dumb things" and keep the things that aren't dumb. This "going to do dumb things" is completely on brand for him with his 5 step manufacturing improvement process, step 1 of which is "make your requirements less dumb". This is actually how Nike works, having worked there. They try all kinds of weird things aren't aren't afraid to, and then drop the ones that don't work with no regrets, keeping what works. Try new things, fail fast isn't a bad strategy for innovation. Whether or not it'll work out for twitter remains to be seen. Especially with the rest of biased tech still upset that they lost their monopoly on the narrative arrayed against him.
- ilyt 4y ago
  Right, but cost to that is your company losing some lunch money on prototypes. The costs to "fail fast" (if we're being generous) strategy Twitter employed so far far exceed that
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
tootie 4y ago
I'm guessing that some product people finally managed to explain what an AB test is to Elon and why they use them to validate product ideas before rushing implementations out the door based on gut instinct. Aside from trying to assuage advertisers about his capricious product decisions he's also trying to act like he is now an expert on digital product development.
awestley 4y ago
I read the experimentation bit as Twitter (Elon) trying to push the boundaries of what Twitter can get away with from a PR perspective.
alfalfasprout 4y ago
More like Twitter now lacks staffing of its ML and experimentation infra to actually do experiments carefully.
choppaface 4y ago
The thing is Twitter already A/B tested a LOT and Twitter’s new owner is naive to those prior findings as well as the long-term effects of them. Any advertiser is going to want to know about attrition rates, and those were previously best reported through now-fired employees as well as earnings releases. This is Twitter’s new owner learning the hard way that there are dumb advertisers out there who do want quick lift but longer term he’ll retain only spam. No way Twitter ever gets better than Doubleclick now.
eternalban 4y ago
> What a weird thing to say... A/B tests are a thing, does anyone buy that experimenting with new things by rolling out new features to all users at once is a good strategy? What it means: 'Elon looked bad when the blue checkmark fiasco happened. And more boo boos are on the way. Now Elon doesn't like to look bad. Solution? Everything we do is now covered by "it was just a test" disclaimer. Problem solved.'
guywithahat 4y ago
It could make sense in the context of twitter. They're a "public square" so to speak, so if everyone experiences the same change, they can discuss it at the same time instead of small subsets experiencing it and being confused (or getting back data from the wrong people)
inquirerGeneral 4y ago
I do not think it means A/B testing and such is going away, but that public testing is going to be more widespread about things most people will probably be interested in.
gloosx 4y ago
He should unban Trump as an A/B test, like ones who answered Yes in that poll should be in B group, ones answered No in A group got him banned. It would be great even to separate these people completely into the A/B Twitter (2.0) so they don't see each other's tweets. The main metric should be better poll distribution. Like instead of stupid 51/49 polls better miniority-suppressing 85/15 polls, and these 15 go into C group then eventually getting us to fair 100/0 polls and poll-communism, preventing any pesky problems with different opinions, we all know they are easily solved by polls. I wonder why they don't have 7/12 polls in court, surely 9/12 is enough to decide a sentence already, and all americans think it's fair, why twitter doesn't freakin comply with 75% poll rule?
ladyattis 4y ago
It's kind of hard to calm your advertisers when the boss has basically cut most of the marketing staff responsible for keeping contact with them. That's been the biggest issue so far, aside from the right-wing favoritism that the boss (Musk) has online (he's banned Crimethinc on a whim of Andy Ngo's hearsay). Basically, I wouldn't bother advertising on Twitter since the new boss/owner has no one to filter his nonsense out of the daily decision making. It's one thing to spew platitudes for free speech and it's another to actually lay out the rules of free speech on the platform (i.e. trying not to trigger regulations or reprisals of big govt agencies).

jzelinskie 4y ago

Twitter was previously doing A/B tests. Spaces, communities, downvotes, circles, and even moderation were applied only to a subset of accounts while they actively experimented. The new policy is an objectively less scientific approach to testing functionality than what was occurring before.

generalizations 4y ago
It's not like A/B tests approach anything like the rigor that we expect from actual science, though.
- squaredot 4y ago
  It's about marketing, the A/B testing they do, not science.
  - sithlord 4y ago
    what is not scientific about it? maybe its not good science, but it is science. really almost anything is science... Science is just observation and experimentation. Science doesn't dictate how you do the above. Now, someone would find it impossible to reproduce your findings, but - that would just suggest bad science
    - Sohcahtoa82 4y ago
      Incorrect conclusions are frequently drawn from A/B tests. To some, this makes it unscientific, to others, it just means it's bad science. I think the argument is more semantic than objective. For example, if your metric is "time spent interacting on the platform", then a testing of a rollout of a feature ends up with longer page load times, so users spend more time there because they're waiting for pages to load would increase that metric, and management decides it's a good idea.
    - ska 4y ago
      > Science is just observation and experimentation. That's not enough. If you don't include some sense of both 'systematic' and 'rigorous' (and yes, these terms are slippery), you aren't doing science.
      - nvrspyx 4y ago
        But both of those qualifiers exist on continuums and where an experiment lies on them is subject to opinion, so there's no singular threshold. It means it would depend on the person reviewing the experiment whether it's "systematic" and "rigorous" enough to (personally) be considered science. What you're describing is the quality of the science, not whether or not it is science. Until you can objectively measure how "systematic" and "rigorous" an experiment is, your definition of science only applies to you.
        ska 4y ago
        "Experiment and observation" is something we know humans have been doing for literally thousands of years (and without written evidence, probably 10s-100s). At least for loose versions of "experiment" which ties back to rigor. We haven't been doing science for very long. The primary difference is the desire and effort to add rigor and systematic thinking. The difference in efficacy is hard to understate.
        nvrspyx 4y ago
        > We haven't been doing science for very long. Again, this comes back to your definition. Many would disagree in that science has existed for, at least, as long as recorded human history because an actual tenable definition of science is something along the lines of "the endeavor to build knowledge by experimenting and observing the results". The rigor of the experiment is part of the quality of the science, not whether it's science itself. No one is arguing that rigor isn't important to good science. It is important because rigor lends to reliable and valid results. What we ultimately want is results that are reproducible and can be used to predict. If you observe bad results, it's because you did a bad experiment, thus bad science, not that you didn't do science at all. As an analogy: if I took notes during a meeting that no one can understand or use, that doesn't mean I didn't take notes. It just means that I took bad notes.
- radicaldreamer 4y ago
  It wasn't "scientific" to begin with but what's happening right now is pretty clearly panicked throwing stuff at the wall based on Elon's intuition and day to day demands... while that may have worked for him in the past, I don't think it's going to work as well in this domain where you're working with a complex system dependent on millions of people's behaviors and incentives.
  - dragonwriter 4y ago
    > while that may have worked for him in the past Did it? The nature of Tesla and his other current businesses buffer that a bit even if it has been his approach, and it seems to have gotten him thrown out as CEO at X.com twice; among the things going on Twitter seems to be Musk trying to relitigate his failure at X.com without other investors being in a position to kick him out, but he seems to be piling up existential threats without resolving them.
    - radicaldreamer 4y ago
      It does kind of seem like he's addicted to dancing on the edge of cliffs
- thewataccount 4y ago
  > actual science I'm not sure what you mean by this. Many quality studies are A/B tests. A/B just refers to the two IV states you're testing, which you're then observing a DV - sales, engagement, errors, etc. A/B tests can be double blinded (don't tell the error monitoring people which results are from a trial), and have high number of samples, far beyond even most pharmaceutical trials. They can also be really crappy, changing too many variables at once, etc. But they are certainly "real science". EDIT: an example, Drug vs placebo - is an A/B test.
  - compiskey 4y ago
    Advertisers and marketers use the equations but measure contemporary trends. Science is more “what’s true if humans didn’t exist.” Marketing is more “what widget generates more revenue?”
    - typest 4y ago
      This isn’t true. Science is ultimately the scientific method — make a hypothesis, test a change, observe the results, repeat. It’s an algorithm for learning and broadly gaining information about reality. It can equally be applied to things having to do with humans and things not having to do with humans.
      - sdrinf 4y ago
        Not gp, but there's a significant kink when this applies to humans; namely, that humans have the ability to reflect on publicly known outcomes, and change their behavior en-masse in light of information so gained. I put this earler in the phrase "reflection completeness": https://sdrinf.com/reflection-completeness https://sdrinf.com/reflection-completeness ie there are things which stops working when people know about it. In particular with A/B testing, this means that the initial A/B test is intermingled from at least 3 effects: specifically it measures how the naive population's behavior changes as a function of new functionality being made available. This is heavily, heavily time-dependent; specifically there's a "novelty effect" (early data collection will not be representative to long-term usage patterns); and there's "reflection effect" (once the outcome of the test is widely known, people can change their behavior based on that). Controlling for the first is difficult, but possible; controlling for the second, beyond just "keeping everything secret", is significantly more so, as the timelines for that might be years in length. I strongly suspect GP was pointing at this timeline factor, and specifically that market engineering, as currently, generally, widely practiced, is grounded on the immediately available signal of "does it increases sales in 2 weeks of A/B test running". Which, given novelty effects, is heavily biased towards "yes"; and these people aren't incentivized (nor have the time/energy) to measure _very_ long-term effects beyond novelty, and reflection period.
- arrrg 4y ago
  That’s not at all an argument for dropping AB testing, it’s an argument for being more rigorous, especially since in principle the circumstances under which services with a large active user base can test are downright luxurious. Sociologists will frequently not have as good access to such a large participant pool under near ideal experimental conditions with such good ways to observe behavior. And the stuff you have to keep in mind when running experiments is not terribly complex. A bit of statistics, a few things you absolutely have to get right, that’s it. Obviously there are reasons why AB tests are often not run rigorously (statistical illiteracy and pressure to get things done quick as well as to produce tangible results as often as possible – all three of which might lead you to run underpowered experiments with too few participants and to stop testing early which will lead to too many false positives). However, stopping to do experiments (and instead just releasing new stuff and observing the reaction) isn’t really an improvement that leads to better outcomes compared to that.
- joshuamorton 4y ago
  Most scientists likely wish they could get the rigor of a company like Twitter's a/b testing suite.
  - HWR_14 4y ago
    Why wouldn't Twitter (or similarly large companies) have open-sourced their a/b testing suite? It's not like the math there is proprietary. I mean, I'm sure the parameters to the math are proprietary. But the basic math seems simple enough.
    - joshuamorton 4y ago
      If you're not using an off the shelf one, chances are it's tightly integrated to your application framework. Trying to tease out the pieces that aren't coupled to Twitter's User class is probably more effort than it's worth
      - HWR_14 4y ago
        That makes sense. Thanks. I guess the comment I read implied Twitter had an amazing A/B test suite, as opposed to a tightly specialized A/B test suite.
        simeonf 4y ago
        It's called DuckDuckGoose - see https://blog.twitter.com/engineering/en_us/a/2015/twitter-experimentation-technical-overview https://blog.twitter.com/engineering/en_us/a/2015/twitter-ex...
      - cornel_io 4y ago
        And this is sadly true about almost every good A/B testing system out there. They always involve tooling that has to exactly fit your stack, and it's really tough to make a general purpose product that would offer anywhere near the same value. To some extent, you grow your company and codebase around your A/B testing system, not the other way around, because it has to slot into so many places: deployment, testing, front-end, back-end, analytics, monitoring, etc. Almost no two companies share the same stacks across all of those dimensions, and an A/B testing system that doesn't hook in tightly to every one of those systems is not complete. This is why I always get a bit scared when people think they can just use one of those "drop-in" services and be done with it: yeah, great, you can now fiddle your JavaScript from some third-party website, but you're going to have a big project ahead of you passing group assignments out to all of the different systems that will need to know about them, and almost guaranteed some part of your stack will not have a library available from the service you picked so you're going to be writing your own REST wrapper, and dammit they don't document that API very well and it seems to be responding differently since the last update, and man it'd be a lot easier if I could just pipe results straight from my own service into Big query rather than running a daily user export, and damn, their dashboard doesn't let me set exclusion criteria on my own metrics, I have to send activation events now, and etc, etc. By the end you've basically built your own A/B test system from scratch, you've just paid someone else to do the "int myGroup = Random.next()" call, which is the easiest part to build.
- root_axis 4y ago
  Based on what reasoning? Data quality with respect to computer systems approaches perfection, it is far more reliable than any data you can hope to approximate in the real world. There is also plenty of "actual science" with terrible data quality, hence the replication crises spanning large sections of the scientific landscape.
- fullshark 4y ago
  You are likely putting "actual science" on a pedestal here
- Waterluvian 4y ago
  If "actual science" is a real thing, then absolutely nothing humanity has ever done in the name of scientific endeavour is "actual science," given every experiment exists on a continuum of trade-offs. A case could be made that A/B testing is insufficiently rigorous given specific goals, resources, limitations, context, etc. But that case isn't being made here.
  - williamstein 4y ago
    Some pure mathematics is actually rigorous.
    - Waterluvian 4y ago
      Hah! You're fast. I was about to comment on that: "actual math" probably exists. "actual science" probably doesn't. I'm not sure any experiment can ever be free from error, uncertainty, trade-offs. (And now I'm super curious about this...) Thanks though. I've narrowed my original comment to more accurately represent the scope I am referring to.
      - philippejara 4y ago
        to be fair whether mathematics is a science or not is debated[0], so your initial claim might very well stand true depending on which side one takes in the argument. [0]:https://philosophy.stackexchange.com/questions/10178/is-mathematics-an-art-or-a-science https://philosophy.stackexchange.com/questions/10178/is-math...
        Waterluvian 4y ago
        That's fair. Though that gets into semantics which I don't find interesting or fun. In this context, I think it's fair to say we're talking about "things that can be experimented on to learn more things."
- cornel_io 4y ago
  If they're done right, they are exactly as rigorous as we'd expect from science, since they are literally the same as randomized control trials. You can do even better with A/B testing because you have much tighter control over inclusion criteria, treatment compliance, and outcome analysis.
- dredmorbius 4y ago
  Others have addressed the ways in which A/B testing does hew quite closely to the standard of empirical observation under controlled circumstances, with which I largely agree. Where A/B studies may go wrong in my view is a few other elements: - A/B studies have difficulty in determining differences based on multiple interacting characteristics. In fairness, so does empirical science, and the principle of "holding all else constant" is a frequent assumption of scientific processes. - A/B studies face an inherent self-selection / exclusion bias: the participants in this round of A/B testing are those who've not been driven off the project/product from past experiments and design changes. Given that many Web 2.0 companies eventually dance with pushing people right up to the border of tolerance, it's quite possible that A/B testing has a long-term effect of pushing those participants whose tolerance has been exceeded out of the study population entirely. I don't know how large a factor this is, though loud / rage quitters are certainly a prominent (if not necessarily large) cohort. Whether or not they're also influential, or perhaps more importantly when they become influential is another question. Again, this is a fairly common problem with any social experiment, including natural social experiments, see various forms of brain-drain and social flight. - A/B testing tends to focus on short term changes and behaviours, which may mask longer-term outcomes. This has some overlap with the above, but also with subjects' general response to change. See the classic case of this in the Hawthorne Effect (<https://en.wikipedia.org/wiki/Hawthorne_effect https://en.wikipedia.org/wiki/Hawthorne_effect>). The upshot is that A/B testing can be valid and useful, but that experimental design, particularly in the case of social and psychological experiments, where subject feed back into the study and its methodology itself, is exceptionally thorny.
  - TeMPOraL 4y ago
    > it's quite possible that A/B testing has a long-term effect of pushing those participants whose tolerance has been exceeded out of the study population entirely. I'd compare this to how evaporation rate increases with temperature, as more particles find themselves with enough energy to escape the liquid. From my personal experience, even if I can tolerate a lot of UX abuse, each such "optimization" lowers my threshold of switching to a competitor. Software in general, and SaaS specifically, resists commoditization, but every now and then an actual alternative to a product/service I'm using shows up - and whether or not I switch (and when) is correlated with how much I resent the incumbent for their UX "improvements". I'd add one bullet point to your list: - Unlike regular scientific experimentation, A/B testing is a methodology primarily spread in business circles using regular hype channels. That is, the average practitioner is not qualified to execute it correctly, which is one of the reasons I see A/B testing more as tools to launder arbitrary decisions. Because consequences of doing it wrong are typically not immediately apparent or obvious, both companies and customers suffer (and a vast space for fraudsters is created). I'm in a charitable mood, so I'm not passing judgement on people for not having PhD-level understanding of statistics - just pointing out that, to the degree much larger than in sciences (even soft ones, which suffer some of the same structural problems), there's little pressure to do such tests correctly (and there's lot of ways to make money or status by doing them without regards for correctness). From what I hear, a common way of executing A/B test badly and getting bullshit results, is by terminating the test early when it shows the relevant metrics improving for the test group - vs. running it longer if no big improvements are observed (or the metrics start getting worse for the test group). This biases the experiment towards giving false positives. This problem was big enough that there was a debacle around Optimizely few years ago, whose UI was accused to promote this early termination of tests. The cynical take I'm still somewhat partial to is that it wasn't an accident (if not done on purpose, then possibly... a result of an A/B test!) - false positives make the (statistically naive) users feel they're getting more value from Optimizely than they actually are.
sayrer 4y ago
A/B tests are pretty problematic for social networks, since anything effective can also influence the control group. They are excellent for single-player activities like shopping carts and signup flows. "Network Experimentation at Scale" from Facebook describes how difficult this problem is. Most A/B test frameworks don't reach this level of sophistication. It does make some sense to just ship things if you don't have time to build out something like that. (disclosure: I worked at Twitter long ago)
- throwawaymaths 4y ago
  They're also problematic because you'll likely optimize for whatever metric you pick with no regard to the ethics of the optimization. Sometimes a double blind test is not the right thing to do.
  - sayrer 4y ago
    I've never really bought this argument. There might be unethical growth hackers in the company, but A/B tests are not the problem in that situation. In my experience, they mostly just catch bugs. Stuff like "hmm, our much better looking signup flow underperforms... oh, the form is broken on Safari." That kind of thing.
    - solarkraft 4y ago
      Think of the AI optimizing for a stated goal while ignoring implied constraints (e.g. eradicating humanity to stop wars). This can be the case for A/B testing. Sure, you can increase ad clicks by 30% ... if you trick the user into clicking it through a carefully timed layout jump. I think GP's argumentation may go in this direction. I'd probably not say A/B testing is the problem itself, it is a tool after all, but I could imagine it's sometimes not used very well. Another point: Spotify's core flow changes so much (feels like almost daily) that I've lost all confidence in using it.
    - P5fRxh5kUvp2th 4y ago
      To make it easier for everyone to understand the other posters point... a double blind test can help you determine the most effective way to cause pain to a monkey, but it will never answer the question of whether you should be doing so.
ilyt 4y ago
I feel that we can hardly call that random changes "policy"...
inquirerGeneral 4y ago
You are suggesting that A/B testing is going away rather than this being simply an addition to widespread testing.
racked 4y ago
They've probably fired so many people that they wouldn't even know _how_ to run A/B tests even if they wanted to.

aaroninsf 4y ago

It's only weird, if you attempt to parse it with a straight face. This is some B-grade best-effort spin on what has been uncontrolled chaos with predictably awful effects. The only thing keeping Twitter rolling is the majority percentage of the casual niche user in non-political and non-technical niches, who haven't been paying attention to the chaos; and who by virtue of being casual users are not notably surprised at everything that has broken both culturally, wrt safety and content, and technically. Unfortunately for Leon the money comes from corners who HAVE been paying attention and not only see what's happened, and ongoing—they see through this kind of comedic college-try at handwaving around it.

sf_rob 4y ago
̶u̶n̶s̶t̶a̶g̶e̶d̶ ̶r̶o̶l̶l̶o̶u̶t̶s̶ ̶o̶f̶ ̶f̶e̶a̶t̶u̶r̶e̶s̶ ̶t̶h̶e̶ ̶C̶E̶O̶ ̶t̶h̶i̶n̶k̶s̶ ̶a̶r̶e̶ ̶c̶o̶o̶l̶ ̶ public testing/experiments
phpisthebest 4y ago
Twitter has never been more fun, useful, and engaging for me. I signed up for an account after years of refusing to do so after elon took over
- bvasilis 4y ago
  Then, if I may ask, how do you have an idea of what it was before?
  - phpisthebest 4y ago
    I was a lurker before, I read twitter all the time with out an account. It as a pain because they kept trying to force you to create an account so I kept having to find ways around their account signup walls which was fun...
    - baking 4y ago
      What's funny is they no longer try to force you to create an account. I used to use nitter.net to browse someone's history, but now I can do it on Twitter apparently with no restrictions.

janoc 4y ago

How is that different from Tesla essentially using Tesla car owners as unpaid betatesters for their autonomous driving (and other) software while disclaiming all responsibility when anything happens? This "learn by doing" is going to end up well ... not. Especially when trying to learn by repeating mistakes with completely foreseable consequences, like that blue "verified" badge being available for anyone who pays without any verification.

w0m 4y ago
> using Tesla car owners as unpaid betatesters Almost right. Charging Tesla owners to beta test their autonomous driving software.
- dexterdog 4y ago
  Almost fully right. Charging and risking Tesla subscribers to beta test their autonomous driving software.
  - lancesells 4y ago
    So close. Charging and risking Tesla subscribers along with everyone else on the road to beta test their autonomous driving software.
    - LanceJones 4y ago
      I'm one of those people. Running FSD Beta in my Model S Plaid in BC. And your negative connotation doesn't apply to me or many/most of the other beta testers with whom I'm regularly communicating.
- yreg 4y ago
  It's curious how many people are unhappy that someone choses to pay for an early-access product that's in beta.
  - 4y ago

14u2c 4y ago

Maybe they are referring to Elon's recent unhinged polls as testing.

giantrobot 4y ago
Hey: vox populi, vox dei! It's Latin so it must be smart. Apparently Twitter had some huge problem with bots when Musk was trying to get out of the purchase. Thankfully he solved the bot problem after the purchase so he could run polls and really get the will of the people and not, you know, all those previously problematic bots.
- bryananderson 4y ago
  Except when a poll gets a lot of votes that he doesn’t like, in which case he says “looks like the bots are out in force today!”
  - giantrobot 4y ago
    Good ol' Schrödinger's bots.
- edavis 4y ago
  > It's Latin so it must be smart. It's perfect, in a way, that the full quote is "Nec audiendi qui solent dicere, Vox populi, vox Dei, quum tumultuositas vulgi semper insaniae proxima sit." which translates to "And those people should not be listened to who keep saying the voice of the people is the voice of God, since the riotousness of the crowd is always very close to madness." Refs: https://twitter.com/cagrimmett/status/1595967787339743232 https://twitter.com/cagrimmett/status/1595967787339743232

vineyardmike 4y ago

> What a weird thing to say... They need to explain musks antics somehow. More polite wording than idve used.

Toxide 4y ago

Hey remember all those terrible things we banned? Lets experiment by letting them free again! Weeeeeeeeeeee.

s2radhak 4y ago
for our next experiment, we should re-introduce lead into paint
- fortuna86 4y ago
  looking into this
  - ss108 4y ago
    The science on it is unsetttled and ideologically biased because I don't like the policies people justify on its basis.

nicce 4y ago

It might be just an escape plan when Musk uses platform for his own needs. Banning users he does not like or allowing certain speech when filtering others and so on….

davesque 4y ago

I think this new idea of "public testing" is really just a post hoc recasting of Elon's failed Twitter Blue rollout. Calling it by those words is...creative? But the failure was less an experiment as it was a lesson in humility.

dragonwriter 4y ago
> I think this new idea of "public testing" is really just a post hoc recasting of Elon's failed Twitter Blue rollout. Calling it by those words is...creative? Its just a diplomatic rephrasing of Elon’s “do lots of dumb things” tweet: https://twitter.com/elonmusk/status/1590384919829962752?t=ccBHJovjIC1jFCyGrh5gfw&s=19 https://twitter.com/elonmusk/status/1590384919829962752?t=cc... But advertisers who have pulled out because of distrust and lack of stability aren’t likely to be reassured by rationalizations for the policy instability, they’ll just be confirmed in their decisions to wait to see how things shake out. And regulators concerned about noncompliance with binding rules aren't going to care about a PR rationalization at all, except insofar as it provides evidence that the failures were intentional rather than inadvertent.
unsui 4y ago
Exactly. It's just a CYA statement, basically saying "oops, I meant to do that...", and leaving the door open to make more oopsies as intentional "experiments".
renewiltord 4y ago
A typical HN comment on a SpaceX post might have once looked like: > Unfortunately, the "move fast and break things" attitude doesn't work with rockets. You will likely find this comment verbatim from the last 10 years. Given that is so, I can't see how someone could claim that Elon Musk is new to visible experimentation.
- palata 4y ago
  Well here it sounds more like "break things and be an asshole", to be honest. I wonder what would have happened, had he fired 75% of the engineers when he took control of Tesla /s.

sharkjacobs 4y ago

It sort of makes sense if the experiment is about judging not just individual user reactions but the wider reaction of the media, advertisers, "the conversation" to a change. I don't think that makes it a good idea though, seems like each failed "experiment" poisons the pool and certainly makes it possible to do experiments in isolation.

memish 4y ago

It's just referencing the fact that the new Twitter is more open and transparent. I can't believe how many people are spinning this as a bad thing. We heard almost nothing from Parag and little from Jack when he was running it and experiments were opaque from the outside. Now we're hearing from the CEO and employees like George Hotz about what they are doing and planning, and they're involving the community, asking for feedback directly.

JumpCrisscross 4y ago
> previous Twitter wasn't as transparent as it is now Reinstating accounts on the basis of a poll, on a platform you have spent months railing for having too many bots, is a good example of CYA transparency.
- spamuel 4y ago
  Elon specifically mentioned that his polls were actually a great way to obviate bots, as reflected by the bot activity seen on them. He was going to reinstate the accounts anyway.
  - ilyt 4y ago
    That sounds like excuse after the fact someone called him on his bullshit
- h3rsko 4y ago
  And are all the bots expected to vote one way?
  - tptacek 4y ago
    That's your response? That polling is transparent because both sides will just turn it into a game of CoreWars, and may the best argument win (at CoreWars)?

JumpCrisscross 4y ago

> this post is entirely to calm advertisers Regulators, too [1]. [1] https://www.ft.com/content/a07ca1ae-9f9a-46ee-9457-27bb30e18ed2 https://www.ft.com/content/a07ca1ae-9f9a-46ee-9457-27bb30e18...

dev_tty01 4y ago
Yes, but "[w]e believe that a service of this importance will benefit from feedback at scale" will just make advertisers unhappy. To state the obvious, I don't think they are reading the room very well...

zzzeek 4y ago

the marketing team writes a Grown Up , Adult blog post to try to look normal, then commander Elon will come out tomorrow high fiving more nazis [1] and banning more non-nazi accounts for supposedly belonging to "antifa" and saying things he doesn't personally like [2]. This is the same "trapped enabler" pattern we saw with Trump in the early days, with his staff constantly coming out to try to paper over whatever horrible thing he did. They had no shame, and neither does the Twitter staff that wrote this blog post. [1] https://archive.ph/xYeYY https://archive.ph/xYeYY - TPM article without paywall [2] https://theintercept.com/2022/11/29/elon-musk-twitter-andy-ngo-antifascist/ https://theintercept.com/2022/11/29/elon-musk-twitter-andy-n...

everdrive 4y ago

I largely agree with your point, but I have to say as a user I generally don’t enjoy A/B testing at all. It’s invisible to me, and some part of the app or website I’m using is now either broken or annoying in some way, but still works normally for the people I mention it to, since they’re not in my group. Additionally, the changes are sometimes totally transient, so things are disrupted for me briefly, only to be distrusted again.

bombcar 4y ago
A/B testing when it's like the color of something (did you know you can change the hacker news banner color?) doesn't bother me much, but when it's more invasive and I can't turn it off or switch teams, it starts to get annoying. Especially when you're trying to help someone and they are seeing something different from what you see.

s2radhak 4y ago

that blurb you quoted is just twitter rationalizing for Elon's red-bull fueled tweets to "improve twitter"

superfrank 4y ago
Lol, red bull... Elon's late night tweets always make me think of that scene in The Office where the new CEO (James Spader) decides to close one of the branches without telling anyone and when asked about it he goes "I got into a case of Australian Reds... and... How should I say this, Colombian whites"

cpr 4y ago

Where does it say or even imply "rolling out new features to all users at once"?

lesuorac 4y ago

> TLDR: this post is entirely to calm advertisers, How though, Twitter has had week(s) to prepare this post and its so bare. > First, none of our policies have changed. ... The team remains strong and well-resourced. If you're Eli Lily, why would you re-advertise on twitter. Nothings changed from when you stopped!

fortuna86 4y ago
> If you're Eli Lily, why would you re-advertise on twitter. Nothings changed from when you stopped! Not only would I not buy ads, i'd rethinking even having a corporate presence considering the lack of protections against fraud.
- insin 4y ago
  Looking forward to a verified @RealEliLilleyCEO account appearing 5 minutes after Twitter Blue 2.0 3.0 comes out, which only allows you to trivially impersonate people, after which Musk will completely independently invent Verification 2.0 3.0 which will either just be the old system again or a separate Twitter Blue badge. Visionary genius at work.
andreyk 4y ago
Oh I agree, this post won't change anything at this point. But I guess that's why it was released (and as others noted, also for regulators / to save face after weeks of chaos).

NaturalPhallacy 4y ago

Musk literally said "we're going to dumb things" and keep the things that aren't dumb. This "going to do dumb things" is completely on brand for him with his 5 step manufacturing improvement process, step 1 of which is "make your requirements less dumb". This is actually how Nike works, having worked there. They try all kinds of weird things aren't aren't afraid to, and then drop the ones that don't work with no regrets, keeping what works. Try new things, fail fast isn't a bad strategy for innovation. Whether or not it'll work out for twitter remains to be seen. Especially with the rest of biased tech still upset that they lost their monopoly on the narrative arrayed against him.

ilyt 4y ago
Right, but cost to that is your company losing some lunch money on prototypes. The costs to "fail fast" (if we're being generous) strategy Twitter employed so far far exceed that

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

tootie 4y ago

I'm guessing that some product people finally managed to explain what an AB test is to Elon and why they use them to validate product ideas before rushing implementations out the door based on gut instinct. Aside from trying to assuage advertisers about his capricious product decisions he's also trying to act like he is now an expert on digital product development.

awestley 4y ago

I read the experimentation bit as Twitter (Elon) trying to push the boundaries of what Twitter can get away with from a PR perspective.

alfalfasprout 4y ago

More like Twitter now lacks staffing of its ML and experimentation infra to actually do experiments carefully.

choppaface 4y ago

The thing is Twitter already A/B tested a LOT and Twitter’s new owner is naive to those prior findings as well as the long-term effects of them. Any advertiser is going to want to know about attrition rates, and those were previously best reported through now-fired employees as well as earnings releases. This is Twitter’s new owner learning the hard way that there are dumb advertisers out there who do want quick lift but longer term he’ll retain only spam. No way Twitter ever gets better than Doubleclick now.

eternalban 4y ago

> What a weird thing to say... A/B tests are a thing, does anyone buy that experimenting with new things by rolling out new features to all users at once is a good strategy? What it means: 'Elon looked bad when the blue checkmark fiasco happened. And more boo boos are on the way. Now Elon doesn't like to look bad. Solution? Everything we do is now covered by "it was just a test" disclaimer. Problem solved.'

guywithahat 4y ago

It could make sense in the context of twitter. They're a "public square" so to speak, so if everyone experiences the same change, they can discuss it at the same time instead of small subsets experiencing it and being confused (or getting back data from the wrong people)

inquirerGeneral 4y ago

I do not think it means A/B testing and such is going away, but that public testing is going to be more widespread about things most people will probably be interested in.

gloosx 4y ago

He should unban Trump as an A/B test, like ones who answered Yes in that poll should be in B group, ones answered No in A group got him banned. It would be great even to separate these people completely into the A/B Twitter (2.0) so they don't see each other's tweets. The main metric should be better poll distribution. Like instead of stupid 51/49 polls better miniority-suppressing 85/15 polls, and these 15 go into C group then eventually getting us to fair 100/0 polls and poll-communism, preventing any pesky problems with different opinions, we all know they are easily solved by polls. I wonder why they don't have 7/12 polls in court, surely 9/12 is enough to decide a sentence already, and all americans think it's fair, why twitter doesn't freakin comply with 75% poll rule?

ladyattis 4y ago

It's kind of hard to calm your advertisers when the boss has basically cut most of the marketing staff responsible for keeping contact with them. That's been the biggest issue so far, aside from the right-wing favoritism that the boss (Musk) has online (he's banned Crimethinc on a whim of Andy Ngo's hearsay). Basically, I wouldn't bother advertising on Twitter since the new boss/owner has no one to filter his nonsense out of the daily decision making. It's one thing to spew platitudes for free speech and it's another to actually lay out the rules of free speech on the platform (i.e. trying not to trigger regulations or reprisals of big govt agencies).