3 ms·

How can you have great results but not be able to prove it? I see this perception coming up again and again, and it has, in my experience, always been due to 2

How can you have great results but not be able to prove it? I see this perception coming up again and again, and it has, in my experience, always been due to 2 factors: the time has not been taken to determine a study design; or the improvement doesn't exist. I'm not saying your method doesn't work, but surely if it is your business you could take the time and money to prove it (i'm in a similar situation with mine, although in the health space no-one trusts you unless you can publish, which raises a whole other set of problems - ie shitty studies)

austenallred 8y ago
We have had a huge swing in results, we can show that, but we’re simultaneously improving and testing a lot of things across the entire company (we’re 18 months old and are iterating in everything quickly), so it’s not like we have a well-designed study that isolates this as the only cause. We could do it now I suppose, but you begin to run into ethical questions that don’t mesh with our vision: “let’s give n students a subpar experience to prove a point” would be a tough sell.
- robbiep 8y ago
  have you thought of doing a crossover study and reducing fees for a portion of a cohort? If your method has results, then you should be able to catch them up, and then you can go to market with real numbers (alternatively, it might hint towards class sizes or other changes you can make to maximise efficiency)
  - austenallred 8y ago
    I’m sure you could find a way to prove it’s better; I’d rather just do a great job for everyone and let the results speak for themselves
    - robbiep 8y ago
      Not to harp on about this too much, but that is exactly the problem: results that speak for themselves but don’t have any rigour are sham results and leave you open to accusations that you are simply using marketing flash or selective statistics. Now, a real study can open you to this as wel, but it is harder when you have some ostensibly objective results. There is no such thing as achieving results unless you can prove it, it doesn’t make sense
      - austenallred 8y ago
        > there is no such thing as achieving results unless you can prove it That’s not true. Things happen regardless of whether or not they happen within the guidelines of a scientifically designed/valid study. Students are getting better. Was that correlation or causation? I’m not going to force students to have a mediocre experience to prove it.
        robbiep 8y ago
        In other words, ‘just trust me’. Your business is not self evident

austenallred 8y ago

We have had a huge swing in results, we can show that, but we’re simultaneously improving and testing a lot of things across the entire company (we’re 18 months old and are iterating in everything quickly), so it’s not like we have a well-designed study that isolates this as the only cause. We could do it now I suppose, but you begin to run into ethical questions that don’t mesh with our vision: “let’s give n students a subpar experience to prove a point” would be a tough sell.

robbiep 8y ago
have you thought of doing a crossover study and reducing fees for a portion of a cohort? If your method has results, then you should be able to catch them up, and then you can go to market with real numbers (alternatively, it might hint towards class sizes or other changes you can make to maximise efficiency)
- austenallred 8y ago
  I’m sure you could find a way to prove it’s better; I’d rather just do a great job for everyone and let the results speak for themselves
  - robbiep 8y ago
    Not to harp on about this too much, but that is exactly the problem: results that speak for themselves but don’t have any rigour are sham results and leave you open to accusations that you are simply using marketing flash or selective statistics. Now, a real study can open you to this as wel, but it is harder when you have some ostensibly objective results. There is no such thing as achieving results unless you can prove it, it doesn’t make sense
    - austenallred 8y ago
      > there is no such thing as achieving results unless you can prove it That’s not true. Things happen regardless of whether or not they happen within the guidelines of a scientifically designed/valid study. Students are getting better. Was that correlation or causation? I’m not going to force students to have a mediocre experience to prove it.
      - robbiep 8y ago
        In other words, ‘just trust me’. Your business is not self evident