In many mission critical work domains effective scenario based training and observation are crucial to the success of complex socio-technical systems. Organizations employ many different approaches toward conducting these kinds of training sessions, but as recent high surprise disasters such as the Columbia loss and the 9.11 terrorist attacks indicate, there will always be new surprises that put that learning efficacy to the test. Anomalies will occur, new failure conditions will challenge existing systems and organizations, and these challenges of adapting to surprise and to resilience are nowhere more evident than in these mission critical domains. However, a significant amount of the training conducted in these types of organizations is not about training to be surprised, rather, its about showing individual competency and that current training is effective. These large-scale socio-technical organizations could be more resilient if they effectively exploit the opportunities from these exercises to capture learning and facilitate a deeper understanding behind the cognitive work in the domain.
This thesis proposes the {it the learning laboratory} as a support framework for such exercise design. The learning lab framework serves as a general abstraction of CSE staged world study design and envisioning techniques and extends these approaches to cope with new scalability challenges to resilience. CSE has a long history of conducting research in complex domains utilizing effective staged and scaled world design techniques to support and illustrate the critical cognitive challenges of practitioners at work. The learning lab incorporates a variety of these techniques into a common framework that can be applied to a variety of different types of exercises already being conducted in order to maximize organizational understanding and learning from scenario based observation exercises.
The initial learning laboratory framework synthesizes historical Cognitive Systems Engineering staged and scaled world design practices with findings from personal observations in three exercises: a tactical military intelligence scenario, an urban fire-fighting terrorist bombing, and a large-scale flu pandemic and terrorist attack. This synthesis exposed a number of challenges to successful learning from exercise design. These challenges were explored further in a series of critical incident interviews with a variety of researchers and practitioners in multiple domains. The insight gained from these interviews highlighted a number of decision-difficult design challenges as well as a number of support strategies for coping with specific challenges that must be planned for to conduct effective staged/scaled world studies as learning laboratories. These challenges were translated into support requirements for the design of an exercise planning tool prototype guided by the learning lab framework. Through a better understanding of the vulnerabilities that can stall learning opportunities and make conducting an exercise difficult, researchers, stakeholders, and practitioners responsible for future exercise design should consider the potential impact of taking such exercise design approaches to maximize learning yield and develop a deeper understanding of cognition at work.