A granny summary of an article by Dror Dotan, Pedro Pinheiro-Chagas, Fosca Al Roumi, and Stanislas Dehaene
Track it to crack it: Dissecting processing stages with finger tracking
How can cognitive scientists discover how the brain works? Well, some of them analyze your brain’s activity by connecting you to electrodes, or by putting you in an MRI machine (some even use electrodes inside the brain, if the electrodes were put there for medical reasons). These methods are good but they are not perfect. One problem is, as always, money: some of these machines are very expensive (e.g., even a simple MRI machine currently costs $150,000 or more), and higher precision usually implies a higher price. Another problem is that analyzing brain signals is complicated, takes a lot of time, and requires special expertise.
A cheaper and simpler approach is to use behavioral methods – i.e., to analyze the participants’ behavior in the experiment. In a typical experiment, the participant repeats a simple task several times (each of which is called a trial). For example, in one experiment our participants saw, in each trial, 1, 3, or 5 arrows that appeared one after another, each pointing left or right, and decided whether the majority of the trial’s arrows pointed left or right. We measured the reaction time (how quickly they responded) and the error rate. Unsurprisingly, when all arrows pointed in the same direction (e.g., ), the responses were faster and more accurate than in trials such as . You can see for yourself that the former sequence is easier than the latter, right?
Such measures – reaction times and error rates – are very useful, but they are limited. In our arrows task, each trial showed a sequence of arrows, so the cognitive processing probably keeps changing as additional arrows appear on screen. Error rates and reaction times can indicate what happens at the end of the trial, but they cannot reveal the within-trial changes in cognitive activity. To reveal such within-trial changes, we should measure the participants’ behavior continuously, throughout the trial. This is precisely what the “finger tracking” method does.
The idea is actually quite simple: in each trial, the participant responds by moving their finger from a fixed “start” position to the response location. In our experiment, they dragged their finger on an iPad to one of two response locations, one denotes “right” and the other “left” (check out this YouTube video to see how it worked). We recorded the finger movement during the whole trial – from the appearance of the first arrow until the participant touched a response button. Then, we analyzed the finger trajectories. This is where it gets interesting.
Figure 1. Trajectories of the 3-arrow trials.
In Figure 1 you can see the finger trajectories, averaged over all trials and participants, for each of the eight 3-arrow sequences. Each line in this figure (a trajectory) shows the finger’s average x coordinate in each time point. Time=0 denotes the appearance of the first arrow, and the time between consecutive arrows was 0.3 seconds. You can clearly see that each arrow was processed shortly after it appeared. For example, the trajectories of (orange, leftmost trajectory) and of (pink, rightmost trajectory), two sequences that differ in the first arrow, branch apart relatively early – 400 milliseconds after the trial started and the first arrow appeared. In contrast, the trajectories of and , which differ only on the 3rd arrow, branch apart much later – only after the third arrow appeared. This shows that the participants were processing the arrows serially: each arrow was processed as it appeared, and updated their tentative decision (and finger direction) accordingly. An alternative finding could have been that the participants would wait until the sequence of arrows finished, and only then move the finger left or right, but clearly this was not the case.
In this experiment, the arrows were presented one after another, so you may think it’s not surprising that they were processed serially. But we observed similar serial processing also in other experiments, in which all information was presented when the trial started (at time=0). For example, in this experiment the participants saw in each trial a two-digit number and pointed to the corresponding position on a number line. For adult participants, the decade and unit digits affected the finger movement simultaneously, indicating that the participants processed the two digits in parallel. However, 4th grade children processed the digits serially: their finger movement was affected first by the decade digit and only then by the unit digit. This finding suggests that it takes many years to develop the ability to process the digits of a two-digit number and in parallel: even as late as 4th grade, children are not fully automatic in this apparently-simple task.
This ability – processing the decade and unit digits simultaneously – seems to be handled by a dedicated brain mechanism, which can be impaired following a brain injury. We observed precisely such an impairment in a 70-year-old brain-injured patient. This person was an engineer and worked with numbers his entire life, but the brain injury damaged his brain mechanism responsible for parallel processing of digits, so his performance in the experiment was similar to that of the 4th grade children. This experiment highlights another important advantage of finger tracking: in cognitive psychology, many methods can obtain conclusions only for groups of people. Here we see that finger tracking is sensitive enough to reach conclusions from the data of a single person, and show how this person’s performance is different from the population’s. This is important, because it means that the method can probably be used to diagnose cognitive disorders.
The finger-tracking method has many other benefits and can reveal several other types of cognitive information. If you’re interested, you can read the full article here. It’s not too long and not very technical.