Press "Enter" to skip to content

Tutorial: Tokenized Text

Tokenized text stimuli display the tokens of a text stimulus one-at-a-time on the screen. Tokens are usually words in a sentence, but could be individual characters, phrases, or non-word strings. The color, size, and justification of tokenized text stimuli can all be customized using the methods described in our tutorial on stimulus customization. But because tokenized text stimuli are interactive and dynamic, there are a number of additional customization options.


Change the definition of “token”

By default, FindingFive uses the space character (” “) to break a stimulus into tokens. This means that if the content of a tokenized text stimulus is a sentence, the tokens will be words. To change the boundary between tokens, set the delimiter property to your desired token break:

This stimulus will have two tokens, “abc def g” and “h ijk lmn“.

Note that delimiter doesn’t have to be a character…it can also be defined as a regular expression if you set the optional property regex to true.


Change the pace of token presentation

There are two ways to change how quickly tokens are presented on the screen. First, you could have participants set the pace. In the above example, the property self-paced is set to true, meaning participants will have to hit a key in order for the second token to be presented on the screen. In other words, the screen will initially display “abc def g”, and after a participant’s keypress the screen will display “abc def g-h ijk lmn”. FindingFive will record the timing of the participant’s keypress. This comes in handy for reaction time and self-paced reading studies! (Note that you can also customize the key that the participant must press in order for token advancement. Do this by setting the key_advance property.)

If you don’t want self-paced token presentation and instead want tokens to appear on a set schedule, make sure self-paced is set to false and set the token speed property to the desired characters-per-second:

Here is a comparison of these two presentation speeds:

The first tokenized text stimulus has speed set to 8, while the second tokenized text stimulus has speed set to 32.

Add masking

FindingFive normally presents tokenized text stimuli one token at a time, keeping the previously-presented tokens up on the screen and displaying whitespace in place of upcoming tokens.

This example uses the default “plain” mode of tokenized text presentation.

This default mode is called plain mode. But FindingFive also allows you to place masking characters in place of previous and future tokens, or present all tokens as singletons in the center of the screen.

To insert masking characters on top of previous and future tokens, set your tokenized text stimuli’s mode to masked. Only one active token will be displayed, and all previous tokens will be masked using the character you use to define mask_char (default is “#”).

Masked mode using the default “#” mask.

If you want both the previous and future tokens to be masked, be sure the stimulus’s mode is set to masked and set the forward_mask property to true:

By default, there is no forward masking. But by setting the “forward_mask” property to true, you can cover previous AND future tokens with a masking character.

It’s also possible to present tokens one-at-a-time in the center of the screen. To achieve this outcome, set the stimulus’s mode to singleton:

This is the default presentation style for the singleton mode.

More details about tokenized text can be found in the Study Specification Grammar documentation.

If you have questions or suggestions, don’t hesitate to contact us at researcher.help@findingfive.com.