5 ms·

Every few years, I read an article critical of testing. It's always written by a designer. It's always the same arguments, and the "solutions" are always qualit

by madrox 8y ago

Every few years, I read an article critical of testing. It's always written by a designer. It's always the same arguments, and the "solutions" are always qualitative. They rely a lot on rationalizing after the fact. For example, "this redesign failed because you didn't build your website around the EXACT words your customers might describe their problems and their solution" (italics a quote from this article). Saying these things don't actually help you arrive at better process. At this point, all these articles get me thinking about is why they keep getting written. Were these designers abused by bad product managers? Are they ego-driven and don't like having their creativity reduced to quantitative values? I don't know, but I do know that designers who talk this way tend to be toxic to a productive culture. I've experienced that firsthand.

nostrademons 8y ago
Nah, it's because they're consultants, which means they need to constantly drum up business, which means that they need potential customers to believe they have unique knowledge that'll help improve their business and all the alternatives are shit. It's a sales pitch. And because they're consultants, they don't need 80% or even 10% of website viewers to convert: they just need a handful of customers who will each pay them tens of thousands of dollars for services. Controversy is great advertising: they get all the folks who hate them to help spread the word about their services. It's the same thing with software methodologists, gurus, and architects. Roughly daily there's a new blog post about how [common practice] is now considered harmful, and you need [proprietary expertise held by consultant] to implement some other replacement instead. These posts may or may not be helpful to your software engineering efforts, but they are certainly helpful to the poster's bottom line.
mobjack 8y ago
Testing is not always the right tool for the job. For many types of businesses, once a website design is good enough, it is really hard to gain meaningful improvements from further changes. Chances are that changing the color or position of button is not going to make a difference. If you have 50 customers a month, you are not going to have enough data measure 5% improvement in conversion even if you let the test run for a year. If a business has 50,000 customers are month, then you can measure a 5% improvement in conversion in a few days and testing is a valuable tool. Until you are at that scale where testing becomes feasible, you are better off just following best practices in design and focus on other areas to drive more business. If you are going to test something, it has to be big bold changes so you are able to measure the significance with a small sample size.
hnzix 8y ago
It's the same principle as programmers who want to build a 16 layer microservice vortex using alpha-version frameworks when the product is a simple CRUD form. Let's call these people "architecture astronauts". In the same way, some designers want to sculpt an ultra-minimalist flat typographic fantasyland with no user cues when the product is a straightforward ecommerce shopfront. Let's call these people "design wankers". Fortunately, split testing can prove that the "wankers" are wrong. This makes them write angry blog posts about how testing is bullshit and Steve Jobs and skeuomorphism and building a better horse. Can someone now please solve the "astronaut" problem for me?
jonathankoren 8y ago
There are plenty of things not worth A/B testing, and even if you did, it’s unclear what was the mechanism that made one succeed over the other. (It’s probably just chance.) Marissa Mayer and her 40 shades of blue[0] is a perfect example. In this case, the A/B teat wasn’t science, but rather scientism. [0] https://www.theguardian.com/technology/2012/jul/16/marissa-mayer-appointment-mean-yahoo https://www.theguardian.com/technology/2012/jul/16/marissa-m...

by madrox 8y ago

nostrademons 8y ago

Nah, it's because they're consultants, which means they need to constantly drum up business, which means that they need potential customers to believe they have unique knowledge that'll help improve their business and all the alternatives are shit. It's a sales pitch. And because they're consultants, they don't need 80% or even 10% of website viewers to convert: they just need a handful of customers who will each pay them tens of thousands of dollars for services. Controversy is great advertising: they get all the folks who hate them to help spread the word about their services. It's the same thing with software methodologists, gurus, and architects. Roughly daily there's a new blog post about how [common practice] is now considered harmful, and you need [proprietary expertise held by consultant] to implement some other replacement instead. These posts may or may not be helpful to your software engineering efforts, but they are certainly helpful to the poster's bottom line.

mobjack 8y ago

Testing is not always the right tool for the job. For many types of businesses, once a website design is good enough, it is really hard to gain meaningful improvements from further changes. Chances are that changing the color or position of button is not going to make a difference. If you have 50 customers a month, you are not going to have enough data measure 5% improvement in conversion even if you let the test run for a year. If a business has 50,000 customers are month, then you can measure a 5% improvement in conversion in a few days and testing is a valuable tool. Until you are at that scale where testing becomes feasible, you are better off just following best practices in design and focus on other areas to drive more business. If you are going to test something, it has to be big bold changes so you are able to measure the significance with a small sample size.

hnzix 8y ago

It's the same principle as programmers who want to build a 16 layer microservice vortex using alpha-version frameworks when the product is a simple CRUD form. Let's call these people "architecture astronauts". In the same way, some designers want to sculpt an ultra-minimalist flat typographic fantasyland with no user cues when the product is a straightforward ecommerce shopfront. Let's call these people "design wankers". Fortunately, split testing can prove that the "wankers" are wrong. This makes them write angry blog posts about how testing is bullshit and Steve Jobs and skeuomorphism and building a better horse. Can someone now please solve the "astronaut" problem for me?

jonathankoren 8y ago

There are plenty of things not worth A/B testing, and even if you did, it’s unclear what was the mechanism that made one succeed over the other. (It’s probably just chance.) Marissa Mayer and her 40 shades of blue[0] is a perfect example. In this case, the A/B teat wasn’t science, but rather scientism. [0] https://www.theguardian.com/technology/2012/jul/16/marissa-mayer-appointment-mean-yahoo https://www.theguardian.com/technology/2012/jul/16/marissa-m...