CRESST Developing Tools to Improve Training and Assessment in the Military and Beyond

Supported by the Office of Naval Research, CRESST is developing assessments of complex tasks for naval officers and for K-12 classrooms.

The Center for Research on Evaluation, Standards, and Student Testing (CRESST), a major research center within the UCLA Graduate School of Education & Information Studies, has been involved for more than a decade in assisting the United States military with training tools such as training assessments, simulation-based formative and summative evaluations of recruit and officer performance, games, and intelligent tutoring systems to improve performance. More recently, CRESST researchers are working on a set of projects designed to help the Navy train its personnel and assess the quality of performance for use aboard ships supported by the Office of Naval Research (ONR), one of the premier agencies funding cognitive and learning research and development.

A central issue in current ONR research projects is the development of assessments of complex, extended tasks, as well as a goal of new assessments under development for schools by consortia directed to measure student achievement of the Common Core of State Standards in grades K-12.

CRESST researchers are focusing their attention at the Navy’s Surface Warfare Officers School (SWOS) in Newport, Rhode Island, conducting a series of studies designed to evaluate tools that aim to improve performance on tasks requiring high-level cognitive skills including situation assessment, problem solving, and decision making. CRESST Co-Director Eva Baker and Assistant Director for Technology Bill Bewley lead this research.

SWOS’s mission is to provide professional education and training in support of Surface Navy requirements to prepare officers and enlisted engineers to serve at sea. With partners from other universities  including Stanford and USC, CRESST is supporting the development and evaluation of simulations and other technology systems intended to help officers learn difficult tasks that may be ultimately applied in unforeseen situations.

“CRESST is interested in using simulations and games to assess complex skills involving problem solving and decision making, and we want to automate the assessment as much as we can, using advanced technology, “ says Bewley, who leads the project.  As the officers are trained through various types of technologies, the key is to assure that their performance meets high standards.  Data from the assessments are used for a variety of purposes.  Better than creating a single assessment, CRESST has been working on models and tools to enable the design of other assessments as well.

John Lee, CRESST Senior Researcher, has been supporting SWOS work for a number of years.

“CRESST develops and researches assessment tools that capture human performance and transforms it into feedback during and after instruction that is individualized for each user,” he says.

CRESST has helped SWOS achieve its mission by developing assessment technologies on a number of fronts, including the Tactical Action Officer (TAO) Assessment and the Conning Officers Shiphandling Assessment (COSA) Tool.

The TAO is responsible for tactical deployment and defense of the ship. He or she manages use of the ship’s weapons and sensors, directs the movements of the ship, and monitors the movements and actions of friendly and enemy ships, planes, missiles, and submarines in the region. The TAO must integrate this information to form a tactical picture of each situation, select appropriate responses, issue orders, and inform the commanding officer of actions and intentions.

The TAO Assessment is designed to capture a student’s practical application of TAO skills in a complex tactical decision-making scenario.  It provides objective measures of the cognitive aspects of TAO performance such as situation awareness, decision-making, and communication. It now makes up a large percentage of the final grades for the students and is a deciding factor in qualification for graduation.

Among other exercises in a simulator used to train for shiphandling skills, conning officers’ practice in getting a ship underway from a pier, transiting a port area, anchoring, and approaching and landing a ship at a pier.  CRESST’s Conning Officer Shiphandling Assessment (COSA) assesses four skills: situational awareness, decision-making, maneuver, and communication.  The assessment reliably measures shiphandling competence and provides evaluating officers the ability to enter comments, which helps to shed light on the main reasons for the ratings given to students.

CRESST’s work at SWOS has received the “Excellence in Practice Award” given by the American Society for Training and Development for three successive years.

CRESST research and products originally developed for Navy use have migrated to applications in K-12, higher education, and medical training.  Included in this set of innovations are ways to develop assessments of complex skills, analyze goals and standards, provide feedback, and develop databases for monitoring and adapting to the progress of students, in large-scale or game environments.  Because CRESST has projects in both sectors, it is able to leverage results from findings to benefit children and schools.