11 ms·

I used to work at a musical instruments startup and we tried blind tests of prototypes to assess which combination of materials recipes/sensor design/firmware a

I used to work at a musical instruments startup and we tried blind tests of prototypes to assess which combination of materials recipes/sensor design/firmware algorithms etc were 'the best'. Ultimately these tests only reveal the biases of the players. They will prefer – and actually perform better with – that which they believe to be configured in a certain way, regardless of whether it is in reality or not. We had one prototype that us engineers hated to death because the only difference between it and the rest was that it was a different colour, but ALL the players hailed it as the golden standard. The same goes for the famous 'Does Fuck All' button on recording mixing desks, which you can use to tell players that you've made the change that they wanted to satisfy them. And also when you're buying speakers it's very common to be shown the same set of speakers three times and have three opinions. I just recently tour managed a friend and would set up her mic sound every night, and each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same.

jerf 10y ago
"And also when you're buying speakers it's very common to be shown the same set of speakers three times and have three opinions." I've noticed with sensory inputs in general that the things that impress you immediately, and the things that impress you once your brain has adjusted to the input and has settled in, are different. For the visual example, observe the difference between the settings on the TVs in the store vs. what the community will generally agree is the optimal long-term setting for your TV. Also, while the first ten minutes of a 3D movie may be pretty whiz-bang, I've generally stopped noticing it's in 3D by the end. I think in both cases, our initial preferences are for something sharper and exaggerating the differences, but over the long term that becomes quite tiring. A case could probably be made even for smell; in the store the strong, unusual perfume may be very impressive but if you are going to use it all day long, I prefer to be around people who use something a lot more subtle.
- splawn 10y ago
  This reminds me of the Pepsi challenge. They blindfold you and have you take a sip of Pepsi then a sip of Coke. The results usually have Pepsi wiping the floor with Coke, despite Coke outselling them everywhere. It turns out the reason for the disparity it that they only test the initial sip. Pepsi is sweeter than Coke which seems better to most people when just sipping rather than drinking a whole can.
  - Avshalom 10y ago
    No Pepsi just tastes better. The whole people prefer coke when drinking a whole can comes from Blink where a Pepsi scientist throws out three or four different reasons why kiosk testing might not reflected by consumer purchasing. It ends with both the scientist and gladwell saying "brands are powerful".
    - Declanomous 10y ago
      Coke has a slight vanilla flavor that Pepsi doesn't have. I generally prefer Pepsi to Coke, but sometimes I have a taste for Coke because of that slight vanilla flavor. On the other hand, I tend not to drink pop for long periods on end. My first pop in a long while always tastes incredibly syrupy, to the point where it's borderline unpalatable. If I have more than a couple of pops in a short period it starts tasting like what I 'remember' it tasting like.
    - splawn 10y ago
      Whoops.. thank you for pointing that out. I have in fact read Blink, so that is probably where I picked up this dubious factoid. I need to read it again to refresh my memory so I don't find myself unknowingly quoting it.
      - Avshalom 10y ago
        It's one of those things I tracked down after seeing one to many people say it as if it was a settled fact (never mind of course that there isn't really reason to assume people wouldn't prefer an entire can of the sweeter cola)
- jarmitage 10y ago
  That's exactly it - we notice and respond to differences. See https://en.wikipedia.org/wiki/Just-noticeable_difference https://en.wikipedia.org/wiki/Just-noticeable_difference Once those differences have been normalised we stop feeling the same way about them, and become attracted to other things just because they are novel. This is why it's often better to call a mix 'done' before you start believing it's completely wrong and tearing it apart again, only to prefer the old version.
- rz2k 10y ago
  Interesting. I find the same thing with music itself. I often hate the albums I end up liking the most the first few times I listen to them. Possibly it's because those artists I make the effort to listen to when I don't like their new albums at first are evolving their sound rather than trying to rehash old successes. Anyway, my first impression of new music often doesn't match what my later opinion will be after listening more.
  - RUG3Y 10y ago
    This is the same for me. I've also learned how to recognize an album that I WILL like after more listening, which feels really weird, but it works. Also, very catchy albums that immediately grab me rarely prove to have long-term listenability for me. An example would be the Postal Service album, I remember thinking that was really catchy and great, but after a few listens I hated it.
  - wnevets 10y ago
    This has happen to me with some of my favorite albums/bands. I'll be completely unimpressed the first go around but months later I'll fall in love.
- nostrademons 10y ago
  I've also noticed that there's a difference between products that demo well and products that serve well. Oftentimes the products I most rely on in daily life are really unimpressive when first introduced, and conversely, the products that make a huge splash in a demo are kinda annoying to use. I suspect this is behind many hyped-up, venture-funded startups that fail to get lasting traction in the marketplace. Besides the novelty bias you've identified, I also think there's a context bias. Products that make a lasting impact on our lives do so by fitting into the context of our lives - they are well-adapted to the situation in which they will eventually be used. Products that demo well fit into the context of a demo - they look splashy and exciting when shown to a set of observers who are specifically judging the product. The set of qualities optimized for each of these situations differs.
  - stcredzero 10y ago
    This is why lots of games are optimized for the game media more than they are for players.
    - emp_zealoth 10y ago
      As an avid gamer: that's why you keep your distance from the "AAAAAAA GIGAHITZ" currently steaming on the hype train ^^ There are tons of amazing, enthralling games out there so full of substance - Factorio for the latest example I came across
  - eric_h 10y ago
    I agree, in general, but I'd like to present one counterpoint. Twilio's demos have always been fantastic and their product in actual use is as well.
  - speeder 10y ago
    After some years using flat panels, I decided to grab old CRTs from my storage, and I am loving them. I realised later, that most flat panels are optimized to look good on the brightly lighted stores, they have high brightness and saturation, and look brilliant. Even at home at first they look better than the old CRTs... until on daily usage I get annoyed with their poor contrast and colour reproduction. Yes, on the store a red looking extra red looks good... but when you see on the screen a photo of a relative and their shamed pink face became a monstrous mess of bright yellow and red, then you realised it doesn't look so good... EDIT: Fixed ortography
- z3t4 10y ago
  I can relate to food (supplement) products. That taste panel only test the product once, while the user eats it every day for years. So the test panel will prefer sweet tasty while the user would want a more natural taste.
slig 10y ago
> The same goes for the famous 'Does Fuck All' button on recording mixing desks, which you can use to tell players that you've made the change that they wanted to satisfy them. This is classic. From [0]: > When Florence’s mayor, Piero Sorderini told Michelangelo that the nose on his sculpture that he had been commissioned to do of a young David with sling in hand was too big, he did not argue that the nose was just right, Michelangelo merely took the mayor onto the scaffolding and gave the appearance of changing the nose. Sorderini was pleased and no one was offended. http://48laws-of-power.blogspot.com/2011/05/law-9-win-through-your-actions-never.html http://48laws-of-power.blogspot.com/2011/05/law-9-win-throug...
- wnevets 10y ago
  Reminds me of just remove the duck https://rachelbythebay.com/w/2013/06/05/duck/ https://rachelbythebay.com/w/2013/06/05/duck/
bbarn 10y ago
>They will prefer – and actually perform better with... That's an important distinction. There is something to valuing a tool that makes you use it better. My first version of Visual Studio I paid for out of pocket and I made sure to get my money's worth. My bicycles, probably don't actually make me that much faster, but when I enjoy riding them and like them, I am faster.
- jarmitage 10y ago
  There's an important qualifier here, which is that the tools in question have to be nearly as good as each other, for their user's arbitrary decision making to become more important than the actual differences themselves. I.e. as long as the bikes have two wheels etc, you will choose the one that makes you feel good.
- rconti 10y ago
  I spent double my budget on my last road bike when I decided to go all-carbon, and my rationale was I'd be more likely to ride it if I spent way too much money on it. That was CERTAINLY the case, at least for the first few years.
  - SixSigma 10y ago
    Did you actually like it? I prefer my metal bikes, steel most of all. Carbon is so harsh.
    - grmarcil 10y ago
      Depends on what you like. I ride a Cannondale Supersix for sport and love it. I used to race and still enjoy training hard, so my priorities are lightness, rigidity, and handling. A carbon frame beats steel in all of those. I think the reputation for harshness is undeserved. I rode an aluminum frame (the gold standard of harsh and rigid) for a long time and IMO the difference between frame materials is way smaller than say 23 vs 25mm tires. That said, I also have a 70s era steel frame as my commuter. It's a fun, springy ride that is respectably quick, and I enjoy riding it to work. I would not want to do thousands of feet of climbing on it though.
moultano 10y ago
>I just recently tour managed a friend and would set up her mic sound every night, and each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same. Could she have been responding to the acoustics in the new room?
- jarmitage 10y ago
  During soundcheck there is no crowd around, so the acoustics sound super boomy which is not good for the vocalist. So one trick is to almost have a soundcheck sound that accounts for that, and then switch it up later once the venue packs out.
daveguy 10y ago
> each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same. Very curious of about this one weird trick that you did. What did you do for it to sound right to her? Are you sure you weren't just tuning parameters in a way that allowed her to best identify resonance in a venue?
- jarmitage 10y ago
  The main variables were volume (onstage and in the venue, separately), delay and reverb. We would start with a dry (no effects) sound and get the volume right first. This would already be kinda stressful for her as a dry sound is raw and unkind! We would then bring up the effects to a normal level which would sound way too extreme in soundcheck, as there is no crowd to absorb the sound - so again it wouldn't sound good to her. Then we would bring the effects down in order for it to sound right for an empty room, which felt more comfortable to her. Then for the performance the volume and effects levels would be higher to account for the crowd absorbing the sound. So for soundcheck it was important for it to sound right for an empty room, whereas we were also somewhat covertly tuning parameters in anticipation of the crowd's effect on the acoustics. So the key point is that we were building two sounds; one to fit her psychoacoustic perception in an empty room to make for a comfortable soundcheck, and another for the actual acoustic scenario of the performance. The latter was offputting for the performer despite being 'correct'.
  - daveguy 10y ago
    Very interesting! Thank you for the detailed reply. I have always wondered about the detail that went into soundcheck. Sound often seems too loud in a bar venue. I don't know if that's where you are more likely to have an amateur sound tech or it is just difficult to get a loud enough sound in a small space. Or maybe I'm just a little baby when it comes to sound.
    - jarmitage 10y ago
      To get a good sound, or particularly a good loudness, in a small place is a lot to do with the proportionality of the speaker power output relative to the room size. You can have huge speakers sound awful in a small room and vice versa. Levelling the playing field in 'amateur sound tech' (mostly good craftspeople who are just doing what they can with limited time/resources) is an active area of research in acoustics and DSP. I recommend Dr Josh Reiss of QMUL (where I'm based) for more on this. Check out his work on automatic audio engineering. Machine learning is very appropriate to this area of work: https://scholar.google.co.uk/citations?hl=en&user=fVlS_EgAAAAJ&view_op=list_works&sortby=pubdate https://scholar.google.co.uk/citations?hl=en&user=fVlS_EgAAA...
B1FF_PSUVM 10y ago
> No amount of 'rational tweaking' could achieve We're all gonna be killed by the machines. Unless they grow a sense of humor, and I very much doubt the machine learning guys are doing diddly squat on that front. We're scrod.
etangent 10y ago
> We had one prototype that us engineers hated to death because the only difference between it and the rest was that it was a different colour, but ALL the players hailed it as the golden standard. Out of curiosity, which colour was it?
- jarmitage 10y ago
  The products were black, this one was grey to trial alternative materials processes. So it was even more infamous for being a one-off, and we tried to hide it as much as possible as it confounded our usability test variables. No one could adequately explain why it was rated the best, so we dubbed it The Phantom.

jerf 10y ago

"And also when you're buying speakers it's very common to be shown the same set of speakers three times and have three opinions." I've noticed with sensory inputs in general that the things that impress you immediately, and the things that impress you once your brain has adjusted to the input and has settled in, are different. For the visual example, observe the difference between the settings on the TVs in the store vs. what the community will generally agree is the optimal long-term setting for your TV. Also, while the first ten minutes of a 3D movie may be pretty whiz-bang, I've generally stopped noticing it's in 3D by the end. I think in both cases, our initial preferences are for something sharper and exaggerating the differences, but over the long term that becomes quite tiring. A case could probably be made even for smell; in the store the strong, unusual perfume may be very impressive but if you are going to use it all day long, I prefer to be around people who use something a lot more subtle.

splawn 10y ago
This reminds me of the Pepsi challenge. They blindfold you and have you take a sip of Pepsi then a sip of Coke. The results usually have Pepsi wiping the floor with Coke, despite Coke outselling them everywhere. It turns out the reason for the disparity it that they only test the initial sip. Pepsi is sweeter than Coke which seems better to most people when just sipping rather than drinking a whole can.
- Avshalom 10y ago
  No Pepsi just tastes better. The whole people prefer coke when drinking a whole can comes from Blink where a Pepsi scientist throws out three or four different reasons why kiosk testing might not reflected by consumer purchasing. It ends with both the scientist and gladwell saying "brands are powerful".
  - Declanomous 10y ago
    Coke has a slight vanilla flavor that Pepsi doesn't have. I generally prefer Pepsi to Coke, but sometimes I have a taste for Coke because of that slight vanilla flavor. On the other hand, I tend not to drink pop for long periods on end. My first pop in a long while always tastes incredibly syrupy, to the point where it's borderline unpalatable. If I have more than a couple of pops in a short period it starts tasting like what I 'remember' it tasting like.
  - splawn 10y ago
    Whoops.. thank you for pointing that out. I have in fact read Blink, so that is probably where I picked up this dubious factoid. I need to read it again to refresh my memory so I don't find myself unknowingly quoting it.
    - Avshalom 10y ago
      It's one of those things I tracked down after seeing one to many people say it as if it was a settled fact (never mind of course that there isn't really reason to assume people wouldn't prefer an entire can of the sweeter cola)
jarmitage 10y ago
That's exactly it - we notice and respond to differences. See https://en.wikipedia.org/wiki/Just-noticeable_difference https://en.wikipedia.org/wiki/Just-noticeable_difference Once those differences have been normalised we stop feeling the same way about them, and become attracted to other things just because they are novel. This is why it's often better to call a mix 'done' before you start believing it's completely wrong and tearing it apart again, only to prefer the old version.
rz2k 10y ago
Interesting. I find the same thing with music itself. I often hate the albums I end up liking the most the first few times I listen to them. Possibly it's because those artists I make the effort to listen to when I don't like their new albums at first are evolving their sound rather than trying to rehash old successes. Anyway, my first impression of new music often doesn't match what my later opinion will be after listening more.
- RUG3Y 10y ago
  This is the same for me. I've also learned how to recognize an album that I WILL like after more listening, which feels really weird, but it works. Also, very catchy albums that immediately grab me rarely prove to have long-term listenability for me. An example would be the Postal Service album, I remember thinking that was really catchy and great, but after a few listens I hated it.
- wnevets 10y ago
  This has happen to me with some of my favorite albums/bands. I'll be completely unimpressed the first go around but months later I'll fall in love.
nostrademons 10y ago
I've also noticed that there's a difference between products that demo well and products that serve well. Oftentimes the products I most rely on in daily life are really unimpressive when first introduced, and conversely, the products that make a huge splash in a demo are kinda annoying to use. I suspect this is behind many hyped-up, venture-funded startups that fail to get lasting traction in the marketplace. Besides the novelty bias you've identified, I also think there's a context bias. Products that make a lasting impact on our lives do so by fitting into the context of our lives - they are well-adapted to the situation in which they will eventually be used. Products that demo well fit into the context of a demo - they look splashy and exciting when shown to a set of observers who are specifically judging the product. The set of qualities optimized for each of these situations differs.
- stcredzero 10y ago
  This is why lots of games are optimized for the game media more than they are for players.
  - emp_zealoth 10y ago
    As an avid gamer: that's why you keep your distance from the "AAAAAAA GIGAHITZ" currently steaming on the hype train ^^ There are tons of amazing, enthralling games out there so full of substance - Factorio for the latest example I came across
- eric_h 10y ago
  I agree, in general, but I'd like to present one counterpoint. Twilio's demos have always been fantastic and their product in actual use is as well.
- speeder 10y ago
  After some years using flat panels, I decided to grab old CRTs from my storage, and I am loving them. I realised later, that most flat panels are optimized to look good on the brightly lighted stores, they have high brightness and saturation, and look brilliant. Even at home at first they look better than the old CRTs... until on daily usage I get annoyed with their poor contrast and colour reproduction. Yes, on the store a red looking extra red looks good... but when you see on the screen a photo of a relative and their shamed pink face became a monstrous mess of bright yellow and red, then you realised it doesn't look so good... EDIT: Fixed ortography
z3t4 10y ago
I can relate to food (supplement) products. That taste panel only test the product once, while the user eats it every day for years. So the test panel will prefer sweet tasty while the user would want a more natural taste.

slig 10y ago

> The same goes for the famous 'Does Fuck All' button on recording mixing desks, which you can use to tell players that you've made the change that they wanted to satisfy them. This is classic. From [0]: > When Florence’s mayor, Piero Sorderini told Michelangelo that the nose on his sculpture that he had been commissioned to do of a young David with sling in hand was too big, he did not argue that the nose was just right, Michelangelo merely took the mayor onto the scaffolding and gave the appearance of changing the nose. Sorderini was pleased and no one was offended. http://48laws-of-power.blogspot.com/2011/05/law-9-win-through-your-actions-never.html http://48laws-of-power.blogspot.com/2011/05/law-9-win-throug...

wnevets 10y ago
Reminds me of just remove the duck https://rachelbythebay.com/w/2013/06/05/duck/ https://rachelbythebay.com/w/2013/06/05/duck/

bbarn 10y ago

>They will prefer – and actually perform better with... That's an important distinction. There is something to valuing a tool that makes you use it better. My first version of Visual Studio I paid for out of pocket and I made sure to get my money's worth. My bicycles, probably don't actually make me that much faster, but when I enjoy riding them and like them, I am faster.

jarmitage 10y ago
There's an important qualifier here, which is that the tools in question have to be nearly as good as each other, for their user's arbitrary decision making to become more important than the actual differences themselves. I.e. as long as the bikes have two wheels etc, you will choose the one that makes you feel good.
rconti 10y ago
I spent double my budget on my last road bike when I decided to go all-carbon, and my rationale was I'd be more likely to ride it if I spent way too much money on it. That was CERTAINLY the case, at least for the first few years.
- SixSigma 10y ago
  Did you actually like it? I prefer my metal bikes, steel most of all. Carbon is so harsh.
  - grmarcil 10y ago
    Depends on what you like. I ride a Cannondale Supersix for sport and love it. I used to race and still enjoy training hard, so my priorities are lightness, rigidity, and handling. A carbon frame beats steel in all of those. I think the reputation for harshness is undeserved. I rode an aluminum frame (the gold standard of harsh and rigid) for a long time and IMO the difference between frame materials is way smaller than say 23 vs 25mm tires. That said, I also have a 70s era steel frame as my commuter. It's a fun, springy ride that is respectably quick, and I enjoy riding it to work. I would not want to do thousands of feet of climbing on it though.

moultano 10y ago

>I just recently tour managed a friend and would set up her mic sound every night, and each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same. Could she have been responding to the acoustics in the new room?

jarmitage 10y ago
During soundcheck there is no crowd around, so the acoustics sound super boomy which is not good for the vocalist. So one trick is to almost have a soundcheck sound that accounts for that, and then switch it up later once the venue packs out.

daveguy 10y ago

> each time I would need to play some sort of subtle trick with her so that she felt things were just right. No amount of 'rational tweaking' could achieve the same. Very curious of about this one weird trick that you did. What did you do for it to sound right to her? Are you sure you weren't just tuning parameters in a way that allowed her to best identify resonance in a venue?

jarmitage 10y ago
The main variables were volume (onstage and in the venue, separately), delay and reverb. We would start with a dry (no effects) sound and get the volume right first. This would already be kinda stressful for her as a dry sound is raw and unkind! We would then bring up the effects to a normal level which would sound way too extreme in soundcheck, as there is no crowd to absorb the sound - so again it wouldn't sound good to her. Then we would bring the effects down in order for it to sound right for an empty room, which felt more comfortable to her. Then for the performance the volume and effects levels would be higher to account for the crowd absorbing the sound. So for soundcheck it was important for it to sound right for an empty room, whereas we were also somewhat covertly tuning parameters in anticipation of the crowd's effect on the acoustics. So the key point is that we were building two sounds; one to fit her psychoacoustic perception in an empty room to make for a comfortable soundcheck, and another for the actual acoustic scenario of the performance. The latter was offputting for the performer despite being 'correct'.
- daveguy 10y ago
  Very interesting! Thank you for the detailed reply. I have always wondered about the detail that went into soundcheck. Sound often seems too loud in a bar venue. I don't know if that's where you are more likely to have an amateur sound tech or it is just difficult to get a loud enough sound in a small space. Or maybe I'm just a little baby when it comes to sound.
  - jarmitage 10y ago
    To get a good sound, or particularly a good loudness, in a small place is a lot to do with the proportionality of the speaker power output relative to the room size. You can have huge speakers sound awful in a small room and vice versa. Levelling the playing field in 'amateur sound tech' (mostly good craftspeople who are just doing what they can with limited time/resources) is an active area of research in acoustics and DSP. I recommend Dr Josh Reiss of QMUL (where I'm based) for more on this. Check out his work on automatic audio engineering. Machine learning is very appropriate to this area of work: https://scholar.google.co.uk/citations?hl=en&user=fVlS_EgAAAAJ&view_op=list_works&sortby=pubdate https://scholar.google.co.uk/citations?hl=en&user=fVlS_EgAAA...

B1FF_PSUVM 10y ago

> No amount of 'rational tweaking' could achieve We're all gonna be killed by the machines. Unless they grow a sense of humor, and I very much doubt the machine learning guys are doing diddly squat on that front. We're scrod.

etangent 10y ago

> We had one prototype that us engineers hated to death because the only difference between it and the rest was that it was a different colour, but ALL the players hailed it as the golden standard. Out of curiosity, which colour was it?

jarmitage 10y ago
The products were black, this one was grey to trial alternative materials processes. So it was even more infamous for being a one-off, and we tried to hide it as much as possible as it confounded our usability test variables. No one could adequately explain why it was rated the best, so we dubbed it The Phantom.