A Distributed System for Studying Multimodal Interaction using IoT-based Devices

Abstract Book of the 11th International Conference on Modern Research in Social Sciences

Year: 2026

[PDF]

A Distributed System for Studying Multimodal Interaction using IoT-based Devices

Przemyslaw Mazurek, Arkadiusz Gajda; Szymon Kostrubiec; Ewelina Lazecka; Zuzanna Piwko; Adam Suchomski; Filip Wojno; Kacper Zawada; Jaroslaw Fastowicz

ABSTRACT:

Studying human interactions with the environment is an important task in behavioral research. As we grow older, numerous barriers emerge that limit interactions with the environment. This limits the acquisition of new knowledge, especially when physical interaction is required. To study these phenomena, a distributed computer system with IoT devices was developed. These devices, called widgets, do not perform specific functions nor do they have typical shapes to avoid suggesting or restricting interactions. A centralized communication solution using MQTT transmission was proposed for a diverse set of widgets (various combinations of sensors and actuators), allowing interaction sessions with interaction measurements. From the perspective of communication and application, it is an IoT system. The task of a human or group is to search for interactions, which, in a typical scheme, are based on the stimulation of unknown sensors, which is rewarded by the work of actuators (light, sound, motion). Interaction sessions can be created with random relations and implemented using Markov chains (transition matrix), allowing for the implementation of deterministic and stochastic sequences of desired interactions, which are then explored by the human.

Keywords: IoT; Distributed Systems; Embedded Devices; Human Interaction; Education