A Distributed System for Studying Multimodal Interaction Using IoT-based Devices

Abstract

The study of human interactions with the environment is an important task in behavioral research. As we grow older, numerous barriers emerge that limit interactions with the environment. This limits the acquisition of new knowledge, especially when physical interaction is required. To study these phenomena, a distributed computer system with IoT devices was developed. These devices, called widgets, do not perform specific functions, nor do they have typical shapes to avoid suggesting or restricting interactions. A centralized communication solution using MQTT transmission was proposed for a diverse set of widgets (various combinations of sensors and actuators), allowing interaction sessions with interaction measurements. From the perspective of communication and application, it is an IoT system. The task of a human or group is to search for interactions, which, in a typical scheme, are based on the stimulation of unknown sensors, which is rewarded by the work of actuators (light, sound, motion). Interaction sessions can be created with random relations and implemented using Markov chains (transition matrix), allowing for the implementation of deterministic and stochastic sequences of desired interactions, which are then explored by the human.