Game Development Reference
In-Depth Information
There are many ways of performing an evaluation.
The main elements in the evaluation methodology
are: 1) Pre- and post testing of gamers' knowledge
in risk management and production network risks,
2) Mission, and 3) a period of playing the game
followed by a debriefing phase and finalised with
a questionnaire.
The pre-, mid- and post-test are tests to find out
if the player has gained knowledge from playing
the game. It is only on reconstructable knowl-
edge, so it does not deliver enough information
concerning if the player has improved his skills
on resilience. It corresponds to Kirkpatrick's level
The player is requested to play twice, at dif-
ferent levels and in different contexts. Analysing
the changes in how the player solves his tasks and
how he behaves gives some indication of the learn-
ing outcome regarding soft skills and increasing
awareness. It corresponds to Kirkpatrick's level
three. At the end of the training session, the player
is asked to present and explain her/ his tasks/
missions - a written task with challenges to be
solved in the game. The mission will be followed
up to get information about how the mission was
accomplished. Information of interest will be:
Did the player complete the task? What choices
did he make? This supports the reflection phase
in Kolb's learning cycle as well as Kirkpatrick's
level four. The mission element measures whether
or not the player completes the lesson, and thereby
covers what Chen and Michael (2005) refer to as
Completion Assessment. The mission will also
measure some In-Process Assessment (Chen and
Michael, 2005), however this was restricted to
what was available through the computer log and
the facilitator's assessment. The pre-, mid and post
tests, as well as, the presentation and assessment
of the report can be interpreted to be what Chen
and Michael (2005) refer to as a Teacher Evalua-
tion, as they assess the change in knowledge. The
post-questionnaire focuses on the player's view
of whether the BEWARE game is suitable as an
individual learning tool, and to track the changes
in knowledge and his/her subjective awareness
of improvement of her /his communication and
collaboration competencies.
Evaluation of the Beware Game's
Learning Outcome on Risk
Awareness and Management
One main objective of the Beware game is to
increase the skills in applying a risk manage-
ment process in enterprise networks. Applying
risk management successfully requires that the
participants know the steps of the process. The
figure below shows that around 40% do not have
any idea before they start, but after playing the
game, this percentage is very low. On the other
hand, comparing the pre- and the post tests shows
that there is a clearly increasing number of par-
ticipants knowing all, or almost all, of the steps
of the risk management process.
This could, of course, also be a result of the
introduction lesson as well as of the script they
get, thus we introduced a mid questionnaire in
the winter term 2008/2009. Comparing the results
again, shows however, that the level increases
more after the game than after the introduction.
For the winter term 2008/2009, the mid-test was
introduced for the second group only, so that the
number of evaluands is lower than in the pre and
post test.
Figure 5 is very interesting, because it shows
that in the first year of using the Beware game
for risk awareness, it was not very successful.
It can clearly be seen that in the first two terms,
the number of participants not being able to list
the internal and external risks, both within the
company (in Winter Semester 2007/2008 this was
constant) and enterprise networks was reduced by
50% between the pre and the post test, but it was
still very high, since more than 30% still answered
inadequately. The second interesting observation
was that there was not a large increase in the
number of those answering better after rather than
before the game. This goes both for risks within
Search Nedrilad ::

Custom Search