3 Sim Development The simulation is developed using the Unity game engine. It provides an immersive way for students to learn about the primary pedagogical device in the course: a six-stage framework related to interpersonal influence. Students prepare a roughly ten-page case in which they learn about the challenges of a fictitious senior manager at an automotive company, who is tasked to shepherd in a new era of technology for self-driving delivery vehicles. As they engage with the sim, students individually practice tactics such as crafting a vision statement and providing feedback to employees. The six stages were played sequentially and varied with respect to difficulty. Each stage tests different concepts from the class and was developed to map as closely as possible to the course’s pedagogical framework, which is rooted in scientific evidence on social influence, motivation, leadership, work engagement, organizational culture, and change management. After completing each task, students are greeted with an employee avatar who tells them whether their communication is helpful or problematic. This feedback is tailored to each student’s response. After an extensive search we did not identify any business simulations that allow freeform text input with real-time assessment. We decided to develop this capability because students are more likely to learn skills for certain tasks if they use freeform text – that is, their own natural way of speaking and writing (Westera et al. 2020). To illustrate the value of freeform text, it is informative to consider the most important task in the simulation: crafting a vision statement for the automotive company. Many corporate vision statements, while generally positive, are usually excessively abstract. A statement such as “[w]e want to make the world a better place” might sound appealing at first, but it is hard to envision what it demands of employees. Hence, concrete vision statements, such as “[a] Starbucks on every corner”, are much better suited to influence employees. Such statements are easy to understand and visualize. Using a free response format helps students creatively think through all the possible ways they could communicate to employees, ensuring that the simulation feels natural. It is a “purer” test of their mastery of the material, because they cannot simply guess the correct option (as would be the case in a multiple- choice format). To continue the example above, students will only get a high score if they devise a highly concrete vision for the automotive company entirely on their own. A vision such as “a city full of hybrid cars” will get a higher score than “building a more a sustainable world”, because the former statement is more concrete than the latter statement. In the context of our sim, a higher score means that a student’s vision has successfully inspired a greater number of employees. Figure 1a displays the user interface that students see when they are asked to generate a vision, and Figure 1b displays an example of a response to a student-generated vision. We designed many different employee responses so students would receive feedback tailored to the strengths and weaknesses of the vision they generated, as determined by the trained LLM. Some students might receive feedback from an employee avatar about how the vision they just generated is too long, vague, or complicated. Other students might be told that their vision is inspiring
Beyond Multiple Choice: The Role of Large Language Models in Educational Simulations Page 2 Page 4