RT caveats — why your Stroop effect might be noise

Reaction time looks like the simplest dependent variable in psychology — one number per trial, millisecond precision, nothing subjective. That's exactly why it's so easy to get wrong.

The four gremlins

1. Input lag. Most browsers don't fire keydown at the millisecond the key went down — a USB polling interval, a compositor frame and a V8 scheduler delay all sit in between. On a 60 Hz screen your true resolution is ±16 ms at best. Reimers and Stewart (2015) clocked browser-based RT experiments at ~15 ms systematic error and ~10 ms jitter.

2. Attention lapses. Participants blink, sneeze, look away. Raw mean RT is lifted by a long right tail of lapse trials; your »600 ms mean« might be a true 480 ms with five outliers dragging it up.

3. Practice and fatigue. Trial-by-trial learning inflates the first block; fatigue deflates the last. If you report collapsed means you're mixing three different states.

4. Keyboard vs mouse vs touch. These have different input pipelines. A study run on desktop keyboards in lab and on phone touch in the wild is not one study.

What to do

• Always run the timing audit. SciBLIND ships a Reimers-Stewart grade on every published RT study. If yours is below »A«, report the grade with your results.
• Report RT distributions, not just means. A violin plot or a quantile table tells a reader what the mean hides.
• Trim or model lapses. Ex-Gaussian fits (µ, σ, τ) separate the fast mode from the long tail. If you can't fit that, trim trials > 2.5 SD above the participant's own mean — document the rule.
• Counterbalance block order. SciBLIND's builder does this by default; do not turn it off unless you have a design reason.
• Pre-register the analysis. RT data has enough researcher degrees of freedom to produce any effect you want.

References

• Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47(2), 309–327.
• Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510.
• Whelan, R. (2008). Effective analysis of reaction time data. The Psychological Record, 58, 475–482.