Creation of a Large Language Model-Based Agent for Construction Safety Dialogues in Virtual Reality

Abstract

Construction sites inherently contain many hazards that put workers at risk of injury, including potential falls, electric shocks, falling objects, and improper equipment use. Required safeguards include but are not limited to providing certified wearable protective gear or physical installations, and administering extensive training programs. Construction companies worldwide have an ethical imperative to implement and actively monitor these fundamental worker safety measures to ensure well-being and enable project success. In recent years, training applications on both PC and mobile devices have become increasingly popular due to their easy accessibility and the option for multilingual implementation. The authors present the concept of using a virtual reality (VR) environment for vocational training in construction safety training. Specifically, the authors will focus on a virtual 3D environment involving hazard recognition tasks and interaction with virtual avatars. Non-playable characters (NPC) or avatars may have pre-scripted behavior or actions while some are driven by an Artificial Intelligence (AI). Avatars are an essential part of a game and can break the immersion of the virtual world if they act in unrealistic ways. Believable Agents (BA) are virtual characters designed to exhibit lifelike behaviors and interactions. The goal is to increase realism and immersion by mimicking human behaviors and psychology. This paper aims to create believable agents (BAs) within the VR context using AI and Large Language Models (LLM). We will focus on developing these BAs in three key areas: their movement within the VR environment, their interaction with virtual construction site elements and hazards, and their interaction with the human player through dialog. Through this research, the authors explore how BAs could potentially enhance the authenticity and efficacy of VR training scenarios. A functional prototype is presented and validated using real construction safety exam questions, which the BAs answer in dialogue with the player.