Shute - Stealth Assessment in Virtual Worlds

I will confess that I solicited this paper because it is the first time I have read anything that describes a non-intrusive, principled means of collecting learner assessment data in virtual space. Lots of the White Papers and lots of other written material insist on the inherent value of games and simulations and virtual worlds but none that I have seen, anywhere, have offered any hope of evidencing their beliefs.

Self-report data is notoriously unreliable. After Action Reports can be perilous. Formal tests do not align fully with the instructional experience.

This is really good, really complicated work that the authors are fully qualified to propose.

An interesting topic with a lot of promise. I'm not sure that the initial iterations of SCORM 2.0 can take advantage of much of this work (as it seems to be largely still developing), but I will be interested to track it's progress.

Oddly enough, I think the introduction to this paper (quoted below) has some relevent insight into the "Web 2.0" discussion going on in a few papers. To summarize, yes Web 2.0 is reshaping how people learn, but it isn't necessarily shaping how people are instructed. There will always be a need for a structured instructional program and nobody has really talked about how/if Web 2.0 technologies fit into that mold.
"ADL has been criticized for providing a means for achieving accessibility, reusability, interoperability and durability only for traditional, didactic instruction for individual learners. As early as the 2005 conference ID+SCORM at Brigham Young University, critics challenged ADL to embrace Web 2.0 attributes (e.g., services orientation versus packaged software, an architecture of participation, collective intelligence, data fusion from multiple sources). Clearly the development of self-forming, self-governing online communities of learners has seen far greater uptake than SCORM 1.0."

"A significant problem, however, is that the Defense Department - along with many other providers of instruction - simply cannot allow all of its learning to take place in the relatively open fashion common to many Web 2.0 environments. High-stakes programs of instruction leading to certification of competence require a formal process of authentication that recreational learning does not. Currently, Web 2.0 advocates have not been able to recommend a method whereby an instructional program can be authenticated in the absence of authority. Further, there is much evidence that young, inexpert learners often choose exactly that instruction that they do not need (e.g., Clark & Mayer, 2003). DoD must continue to depend on explicitly managed programs of instruction. Games and simulations can, obviously, be used to advantage when they are managed by external authority. Games, simulations, and mission-rehearsal exercises in virtual space also can be used independently by expert learners or teams of learners who have the capacity to determine when intended outcomes are, or are not, realized. There are limits. That may be about to change."

There are several excellent points made in this paper. Indeed, the DoD will continue to depend on explicitly managed programs of instruction. And yes, developers and instructional designer should try to employ new models and approaches as those suggested in the paper. The primary take away from this white paper: There is an identified gap or general lack of support for an adaptive system based comptencies in SCORM.

SCORM has been very successful at providing PORTABILITY to the DoD and other organizations. Personalized content as described in this paper will most likely require additional server-side/dynamic-delivery types of systems to accompany the content, but unfortunately these type of systems do not usually support portability and wouldn't allow the content to stand alone. In other words, you will need more than just the stand-alone SCORM package to support adaptive personalization of content. This is an important consideration and critical for us to think about when discussing the top priorities for SCORM 2.0. Is the ability for content to "stand alone" still a requirement? How can adapative delivery of assessments and other dynamic personalization of content be possible when an organization must also distribute their same content to multiple geographically independent locations, and based on additional security restraints? Is there a happy medium or a technology solution that will allow for both to be supported in SCORM 2.0? Obviously, one size will never fit all. But hopefully, we can make SCORM 2.0 flexible enough to continue to support global needs such as portability while also allowing for it to be extended to support an organization's unique business requirements for learning, education, and training.

In order for such an adaptive model system to work, many other related initiatives (models) should be in implemented as well as a method for an organization to manage their comptency data in a standardized manner. In order for stealth assessment to work, some form of standard implementation practices would need to be followed. Evidence-centered design seems like a big step in this direction. This white paper refers to the central models of evidence-centered design as being Comptency Model (CM), Evidence Model (EM), and a Task Model (TM). 

 Summary: Probably not Core SCORM. This white paper recommends improving systematic instructional design practices and conginitive aspects of the learning experience in general, it doesn't really offer or propose any new or existing technical specifications or learning technology standards as a solution set for SCORM 2.0. This is an excellent paper though, and we should consider enhancing SCORM to support adapative and immersive environments. Supporting this in SCORM 2.0 would require support for or at least consideration for new specs that might be similar to existing ones out there today such as reusable comptency definitions and shared state persistence (ssp), but will most likely require much more planning and execution than just simply incorporating existing specs.