2 ms·

I wonder how accurate response times generated by these experiments are (and/or, how they define 'response time', which is sort of the same question). E.g. if

by stinos 7y ago

I wonder how accurate response times generated by these experiments are (and/or, how they define 'response time', which is sort of the same question). E.g. if response time is defined as the time between when the visual stimulus appears on the physical screen and the time when a click or keypress is registered, there's quite some things to take into account. I don't know browser internals, but the code 'show stimulus' doesn't lead to the stimulus being on the screen immediately. Calculation takes time. Then the rendering takes time and it's normally going to take at least a couple of frames, plus extra latency introduced by some monitors, etc, before anything effectively changes on the screen. And this might not be a fixed number (best case scenario it's just noise). And there might be an unknown amount of input latency on the mouse/keyboard/touch screen/... And that all might change if you hook up a different screen or switch browser. Now this doesn't matter a thing for toy experiments, or when doing A/B and the noise is the same for A and B, but suppose you want to do an actual experiment to measure response time to a simple and complex stimulus and for some reason the complex stimulus consequently takes a longer time to make it to the screen whereas the subject's actual response time remains the same, but you start measuring from the time where you sent the 'calculate and show stimulus' command, you might draw the wrong conclusion. From experience I know such mistakes get made, and if lucky they get discovered before publishing. But I wouldn't be certain that is always the case.

mezzopiano 7y ago
That's a fantastic question, and a worthwhile research topic in itself! You're right in your concerns, and indeed there's been an extensive discussion in the literature about how valid browser-based experiments are for basic research in psych. If you're interested, the authors of jsPsych have compared native experimental software to their JS-based one (https://dx.doi.org/10.3758/s13428-015-0567-2 https://dx.doi.org/10.3758/s13428-015-0567-2), and our lab has also done some comparisons (https://dx.doi.org/10.3758/s13428-015-0678-9 https://dx.doi.org/10.3758/s13428-015-0678-9). There's a fair amount of literature around this very topic now. In a nutshell, while there's definitely some latency and jitter involved, it's usually far smaller than human response time variability, and averaging across many trials, as well as analysis that takes into account stimulus variability, helps massively to remove noise from the analysis. Also, in many cases, we're interested in difference between times across several experimental conditions/scenarios rather than an exact absolute measurements. Thus, for many applications, browsers are usually accurate enough, and the ease of collecting more data outways the noise. In the end, the only way to deal with latencies properly is to measure and correct for them by calibrating the entire setup. That's what we've done for the new software we're building, which also takes into account render latencies and introduces some more tricks (see https://psyarxiv.com/fqr49 https://psyarxiv.com/fqr49) Hope that helps, would love to chat about this in detail if you're interested!
- stinos 7y ago
  Thanks for the reply, and good to see this actually gets studied. Results do look fairly good indeed!
  - mezzopiano 7y ago
    Sure thing, questions like this are my jam, and I'm psyched to see them on HN :-) (also, somewhat relieved that I'm not the only researcher in my field who checks HN obsessively -- glad to see everyone coming out of the woodwork)

by stinos 7y ago

mezzopiano 7y ago

That's a fantastic question, and a worthwhile research topic in itself! You're right in your concerns, and indeed there's been an extensive discussion in the literature about how valid browser-based experiments are for basic research in psych. If you're interested, the authors of jsPsych have compared native experimental software to their JS-based one (https://dx.doi.org/10.3758/s13428-015-0567-2 https://dx.doi.org/10.3758/s13428-015-0567-2), and our lab has also done some comparisons (https://dx.doi.org/10.3758/s13428-015-0678-9 https://dx.doi.org/10.3758/s13428-015-0678-9). There's a fair amount of literature around this very topic now. In a nutshell, while there's definitely some latency and jitter involved, it's usually far smaller than human response time variability, and averaging across many trials, as well as analysis that takes into account stimulus variability, helps massively to remove noise from the analysis. Also, in many cases, we're interested in difference between times across several experimental conditions/scenarios rather than an exact absolute measurements. Thus, for many applications, browsers are usually accurate enough, and the ease of collecting more data outways the noise. In the end, the only way to deal with latencies properly is to measure and correct for them by calibrating the entire setup. That's what we've done for the new software we're building, which also takes into account render latencies and introduces some more tricks (see https://psyarxiv.com/fqr49 https://psyarxiv.com/fqr49) Hope that helps, would love to chat about this in detail if you're interested!

stinos 7y ago
Thanks for the reply, and good to see this actually gets studied. Results do look fairly good indeed!
- mezzopiano 7y ago
  Sure thing, questions like this are my jam, and I'm psyched to see them on HN :-) (also, somewhat relieved that I'm not the only researcher in my field who checks HN obsessively -- glad to see everyone coming out of the woodwork)