Game Development Reference
in use of the game is quite different. Here
it is the objective to evaluate if the game
has the features needed in order to be sup-
portive for training on resilient production
postgraduate business students, and industrial
participants from engineering companies.
The results of evaluation presented here are:
communication analysis, cognitive analysis, and
situational awareness. Typically a game run takes
one working day, starting with participants' brief-
ing, game practice session, gaming session, and
debriefing. For the cognitive evaluation pre- and
post-gaming questionnaires were issued. For the
situation awareness evaluation, in-game question-
naires were issued at regular intervals. The com-
munication evaluation was done by analyzing the
text chat log files produced by the internet chat
tool used during games.
Evaluation criteria : A set of evaluation
criteria needs to be determined, as it is not
possible to evaluate all available aspects
of an artifact. Such criteria can include re-
duced costs, increased speed, learning out-
Evaluation method : This determines the
process and phases of an evaluation proj-
ect. In this chapter different methods are
applied depending on the evaluation object.
of the Cosiga Game
Evaluation is always done for a specific pur-
pose. The purpose of an evaluation may differ
since evaluations focus on various aspects (see
DeGEval 2003). Therefore, in order to carry out the
evaluation process, it is necessary to evaluate and
assess the evaluation object against some criteria.
In order to ensure an objective evaluation, it is
an advantage to use quantitative measures, since
these are mostly objective. However, measuring
the change in soft skills (e.g. communication) is
difficult. Some approaches for evaluating changes
in soft-skills, used in the evaluation of two serious
games are reported below.
For the analysis of communication, experiments
were set up in a near-virtual environment - the
participants were located in the same large room
but physically distant from one another. They
were also asked to use an internet chat tool for
communication and not to talk to one another.
The analysis of the chat log from several games
showed who communicated with whom and what
was communicated. The following table shows
the distribution of the types of communication.
The results of the communication analysis dem-
onstrated that the games showed the expected
communication pattern. Information was asked
for, and given, by those expected to do so: De-
sign and Production asked; Marketing answered.
Frustration was experienced by the downstream
roles (ie. purchasing and production), but not
design and marketing. Non-task discussion was
confined to those with less to do (purchaser, proj-
ect manager and production). This demonstrates
that the Cosiga simulation's aim to simulate the
communication pattern in real-life product design
processes was achieved.
Evaluation of the Cosiga Game
Various experiments and evaluations have been
carried out since the first working version of
the Cosiga software was available in mid-1999.
These evaluations have been based on: observa-
tion, analysis of the text chat log files, analysis of
cognitive change, analysis of situation awareness,
analysis of audio and video recordings, analysis
of decision making and analysis of the knowledge
creation process. Participants have included under-
graduate and postgraduate engineering students,