Let's Play Calibratio!

I spent much time with my advisor, Dave, on figuring out how to design a simulation of interaction where the task between the automation and the operator was granularizable and parallelizable. In other words, something you could do with the system that can be broken down into measurable quantities, which you both worked on at the same time.

Games are often leveraged in psychology studies, simply because they're fun! Games are perfect little simulation environments for that kind of thing. You incentivize the participants to actually participate by gamifying the task, which motivates them to work towards the desired goal state.

After much debate, we decided on designing a rhythm game kind of like Guitar Hero or Dance Dance Revolution (without all the fancy hardware). These games are great for measuring human performance because the time pressure helps mimic the type of real-world decision-making when engaging with collaborative tasks with automation.

I called the game Calibratio, a portmanteau of calibration and ratio, referring to the ratio of the metrics when calculating calibrated trust/use.

Prototyping, Piloting, and IRB Regulations

Understanding how to figure out the logic of the game algorithms and play involved building several iterations of prototypes of the game. This commenced with building a physical prototype to pilot test with the research team and pilot subjects, which were made of tape to outline the tracks, and stickies as the puzzle pieces, which I swapped out for generic-shaped acrylic resin pieces that I laser-cut.

After deducing some logic from experiencing play in real time and "real life", I then evolved the model into a more sophisticated prototype, which consisted of a wooden track and the actual acrylic resin puzzle pieces that I constructed myself in Nolop, Tufts' makerspace.

Finally, with more insights about the gameplay algorithms and design, it was time to construct the early digital prototypes on Unity, which we would also pilot test to refine into the study version.

Meet Otto, the Automation

Something I struggled with initially was designing the automation itself in the game, which I named Otto. As a thorough researcher who strives for objectivity, I didn't want to design a system riddled with bias in its name, even, much less than the effect of anthropomorphism, tone, or other latent variables which might impact perception. Essentially, I initially tried very hard to find something I realized later was non-existent: an adaptable system in a vacuum.

However, in reality, design is inherently value-laden, meaning that there is no such thing as the pure, "un-tainted" version of Otto I sought for the purposes of examining in this controlled experiment. Thus, I found that the best way to deal with the interference of system qualities was to mitigate all possible factors of Otto apparent to the participant. Otto had no on-screen anthropomorphic embodiment, and only "communicated" with users through text pop-up overlays prior to each game round.

The most salient aspect of Otto's design in my opinion was his transparency in his level of capability. The three gameplay rounds randomized the order of his capability (accuracy at 40%, 60%, or 80%), which he communicated to the player prior to each round.

Measure development

For the "who is the operator" question, I paired the behavioral simulation with a pre-task survey battery on Qualtrics measuring Hofstede's cultural dimensions at the individual level and dispositional trust traits (propensity to trust, confidence in technology). It took me a while to figure out which surveys to select because there’s just so many models of trust in automation. Most people just default to TiA (Jian-Bisantz & Drury, 2000), which is actually flawed because it isn’t necessarily the best tool for the job.

Trust is a multifactorial construct, meaning that it isn't one irreducible measure. In my study, capability-based trust was the main metric of interest because I figured that basing the outcomes on performance was most straightforward in this case. Capability-based trust was measured by querying participants whether they trusted in Otto's reliability (does he get it right?) and Otto's functionality (does he do what I need him to do?).

I chose to base most of my trust questionnaire selection on a model from a recent meta-review by Razin & Feigh (2024). What really stood out to me was how thorough and solid their analysis was. They looked at all kinds of validity and reliability measures, which I thought was a considerably trustworthy analysis method (or was it?). Plus, their meta-review covered different research areas, which I liked because it showed they didn’t limit themselves to just one narrow field. It was refreshing to see an approach that avoids getting stuck in the usual boundaries of a single research domain.

Diagram: Razin & Feigh, 2024.

To measure reliance, I wanted to see how participants actually behaved, with temporal specificity over time. So we built a continuous 0-100% workload knob directly into the game interface. At any point during a round, participants could use the arrow keys to shift how much of the sorting task they delegated to Otto versus kept for themselves. This gave me a near-real-time behavioral trace of reliance as it unfolded, not a post-hoc reconstruction of it.

For dispositional factors, participants took a series of surveys on Qualtrics before playing Calibratio. I chose to look at two categories: culture and personality. To measure culture, I chose Hofstede's five dimensions of cultural values measured at the individual level. To measure personality traits, I captured things like Propensity to Trust and analogous propensities to trust in technology.

Qualtrics Examples

Deployment and IRB Regulation

The next step was deploy the game, where the parent platform for the entire study would be Prolific, which would have the Qualtrics links to the surveys, study ids, participant compensation, and link to the game host.

Deploying the game, however, included finding a suitable platform that was consistent with the Tufts IRB which regulated human subjects studies. With help from the game developers' insights, I landed on hosting the game on a private, password-protected link on the platform itch.io, and used the non-SQL database MongoDB to retrieve data. I sought additional permission from the Tufts Technology Services in order to get this IRB approval.