Assessment item banks and repositories

A JISC CETIS Paper.

By Sarah Currier, Product Manager, Intrallect Ltd.

Final version (January 26th 2007).

Acknowledgements: Many thanks to the following people, who contributed greatly to the development of this paper: Phil Barker, Heriot-Watt University/JISC CETIS; Charles Duncan, Intrallect Ltd.; Steve Lay, University of Cambridge; Mhairi McAlpine, SQA; Andy Powell, Eduserv; Rowin Young, University of Strathclyde/JISC CETIS.

Please leave any comments on the talk page.

wiki version | [[Media:Assessment_Item_Banks_and_Repositories.doc|.doc version]] | [[Media:Assessment-item-banks-and-repositories.pdf|.pdf version]]

Introduction
This paper was commissioned by JISC CETIS in late 2006 to inform those with an interest in repositories in general, and those with an interest in assessment item banks, about the similarities and differences between these two technologies, in order to enhance the potential for future interoperability. Specifically, it asks the question: to what extent may an assessment item bank be considered as a kind of repository, and following from this, to what extent can interoperability and minimisation of effort and resource be achieved in a manner beneficial to the related communities of interest around these technologies?

The seed for this paper was planted at the 2005 JISC/CETIS Conference Repositories Strand meeting, which began its proceedings by trying to nominate a working definition of a repository. However, the meeting was attended by a large number of participants, representing a number of different communities of interest and approaches to repositories, so this was not a straightforward task. It was quickly deemed more useful to identify common features and areas of difference across different repository types and domains, for the sake of developing useful interfaces, services, and other standards and specifications.

As an initial exercise, participants compared functionality for three different repository types: a national learning object repository; a web-based community image-sharing service; and an assessment item bank. There was an implicit question in this exercise, namely: is an assessment item bank a type of repository? However, the assessment experts at the conference were in another room having their meeting and so an answer was not immediately forthcoming, but the identification of a large number of potentially similar requirements across item banks and the two other repository types piqued the interest of those present.

Nearly a year later the two JISC CETIS groups with an interest in this question, the Assessment SIG and the Metadata and Digital Repositories SIG, held a joint meeting, bringing together experts and interested parties from both domains to begin unpacking the question of how repositories and assessment item banks overlap and differ. This paper, alongside the JISC CETIS paper Assessment item banks: an academic perspective by Dick Bacon are key outcomes from this meeting and related discussions.

The aim of this paper is to summarise the current landscape related to repositories (particularly learning object repositories) and assessment item banks. Those who have an interest in these areas each have their own terminology for describing related concepts, so some basic definitions for the purposes of this paper are the first step.

Defining repositories: Overview
The term repository is currently used across education in a number of areas. For instance, there are repositories of research outputs such as journal and conference papers and e-theses, as well as repositories of scientific data: the raw output of experiments and other research.

Learning objects, learning object repositories and standards
Within e-learning, repositories are used to store, manage and share re-usable learning resources, sometimes known as learning objects. Assessments fit within most definitions of learning objects (Definitions of the term learning object abound: see this Wikipedia article for some examples). It is worth highlighting here that the interoperability standards most commonly used in the storage and sharing of learning objects via repositories are the IMS Content Packaging specification and the IEEE Learning Object Metadata (IEEE LOM) standard. In the assessment domain, the IMS Question and Test Interoperability specification (IMS QTI 2.1) uses IMS Content Packaging to structure, store and transport assessment items and tests, alongside a profile of the IEEE LOM for assessment metadata. In addition, both the [ADL SCORM reference model http://www.adlnet.gov/scorm/] and the emerging IMS specification [IMS Common Cartridge http://www.imsglobal.org/commoncartridge.html] include profiles of IMS Content Packaging, the LOM, IMS QTI and other interoperability standards, developed for the needs of particular communities. However, assessment item banks have not until recently been much considered by those developing learning object repositories, nor vice versa.

Defining repositories, or defining repository services?
As communities implementing different types of repository have become aware of each others' work, the question of defining what is meant by 'repository' has emerged, and proved somewhat contentious, as definition issues often are (See, for example, the archives of the JISC-Repositories email list). However, this question has been somewhat side-stepped by the current emphasis on delivering interoperability through service oriented approaches (soas) and Service Oriented Architectures (SOAs), e.g. the JISC/DEST e-Framework (see Section 2.3.1 below) and the MIT OKI OSIDs programme (see Section 2.4 below). The priority now is not on defining a repository as such, but on defining what services may be built around digital resources and how they may work together. The following sections give some key examples showing how, over time, leading e-learning initiatives have been mapping and defining the core functionalities or services involved in repositories.

Defining repositories: e-Learning and IMS Digital Repositories Interoperability
In 2003 the e-learning interoperability standards body IMS Global Learning Consortium defined repositories for the purposes of its Digital Repositories Interoperability specification, thus:

"'On the broadest level, this specification defines digital repositories as being any collection of resources that are accessible via a network without prior knowledge of the structure of the collection. Repositories may hold actual assets or the meta-data that describe assets. The assets and their meta-data do not need to be held in the same repository.' -- (IMS, 2003)"

Figure A (right) shows the IMS DRI functional architecture for such repositories.

Although the specification takes a broad initial view of repositories, it goes on to focus more deeply on guidelines for the following "core functional interactions between the Mediation and Provision layers of the DRI Functional Architecture" (ibid.), any of which may be considered as services under the service oriented approach:
 * Search/Expose
 * Gather/Expose
 * Submit/Store
 * Request/Deliver
 * Alert/Expose

Figure B (left) illustrates this focus by highlighting this core part of the functional architecture from Figure A.

The JISC Information Environment, the JISC e-Learning Framework, and the JISC/DEST e-Framework
The JISC Information Environment (JISC IE) has been in existence for some years, and represents a broad approach to information and resource provision for UK HE and FE; it has never focused solely on e-learning needs. The JISC IE’s technical architecture "specifies a set of standards and protocols that support the development and delivery of an integrated set of networked services that allow the end-user to discover, access, use and publish digital and physical resources as part of their learning and research activities." Diagram C (left) shows this architecture.

The original author of the JISC IE diagram in Figure C, Andy Powell, presented an updated version for the JISC CETIS meeting on repositories and item banks in October 2006, shown in Figure D (right).

Separate from the JISC IE, the JISC e-Learning Framework (eLF) was an early attempt, using a service oriented approach, to map and define services for which specifications and standards are required for the purposes of interoperability across the e-learning domain. These services included those needed to support repositories, e.g. searching, harvesting, and metadata management. The eLF’s 'wall of bricks' approach, whereby services and their agents were defined as Sample User Agents, Learning Domain Services, and Common Services, is still available on a legacy site, showing an early mapping of how services relevant to repositories fit in with other learning technology services.

The eLF's early efforts have more recently been folded into the JISC/DEST e-Framework, which also builds on the JISC Information Environment, and more generally on recent technological developments such as Service-Oriented Architectures (SOA), and the growing "emergence of Web Services as a means to enable Web-based applications to interact with each other using the basic communication protocols of the Web based upon open standards." The e-Framework will provide access to useful information, guidelines and implementation examples and experience related to services appropriate to repositories and other technologies used in learning and research. Various programmes of research and development around the world are feeding into the e-Framework, including the JISC Digital Repositories Programme.

The JISC Digital Repositories Programme
The JISC Digital Repositories Programme began in 2005, and has been running in parallel to, and feeding into and from, the JISC/DEST e-Framework. Two years after the IMS DRI specification was published, in preparation for the first round of funding for this programme, the Digital Repositories Review noted the importance of increasing communication across the different domains using repositories, to better promote interoperability. In order to do this, they emphasised the need to "be able to define the characteristics of repositories and seek the coherence of a common approach" (Heery and Anderson, 2005). The definition offered reflected a narrowing from IMS's perception of what repositories encompass, through distinguishing repositories of content and metadata from collections of metadata only, thus:

"We propose that a digital repository is differentiated from other digital collections by the following characteristics:
 * content is deposited in the repository, whether by the content creator, owner or third party
 * the repository architecture manages content as well as metadata
 * the repository offers a minimum set of basic services e.g. put, get, search, access control."

In addition, mirroring IMS’s early focussing on certain core functions or services, the emphasis had shifted by the following year, as repository and e-Framework activities gathered pace. In the JISC Digital Repositories Roadmap of 2006, designed to build on research already carried out and inform future developments, Heery and Powell noted:

"As more repositories are implemented there is a realisation of the potential for data to flow between repositories and other systems and for added value services to interplay with repository content.

"This perspective was put forward by Cliff Lynch in 2003:

"'[A] university-based institutional repository is a set of services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and its community members. It is most essentially an organizational commitment to the stewardship of these digital materials, including long-term preservation where appropriate, as well as organization and access or distribution... An institutional repository is not simply a fixed set of software and hardware.' Lynch (2003), cited in Heery and Powell (2006)"

"Note that the focus on the services that the repositories provide is very important, and holds true whether the governance of the repository is at a national, agency or institutional level."

Whether a repository is defined as a discrete system or as set of services around a collection of digital resources, the emphasis on services and on repositories’ ability to work with other repositories and other systems and services has become the central concern.

OKI OSIDs approach to repositories
The Open Knowledge Initiative (OKI) at MIT publishes Open Service Interface Definitions (OSIDs), which are:

"'a kind of conceptual API, which can be expressed in different programming languages. The bindings to these languages are interfaces (or the nearest thing to that concept that the language has). The interfaces require implementations by service providers, which are separate releases by vendors or other contributors.'"

There are both repositories and assessment OSIDs (OKI, 2004a and 2004b), and OKI note that these two may work together, in that the repository OSID may be used for a repository of assessment items (OKI, 2004a). The OKI Repository OSID supports discovery and retrieval of "assets", which include any kind of digital resource, and may themselves contains smaller assets (i.e. complex packages of content are supported). As with the IMS DRI specification noted above, repositories may contain assets or metadata (known as "info records"), or both (OKI, 2004b). The OKI OSIDs exist somewhat apart from the other standards noted in this paper; however, they are worth investigating, and they certainly do not contradict any of the approaches to defining repository functionality detailed here.

Defining repositories: Conclusion
Clearly, in determining what is important about repositories for the purposes of interoperability, and for the purposes of eliminating unnecessary duplication of effort across different domains (including the assessment domain), the key is looking at what elements of functionality, or services, are needed for providing access to resources. The core elements of functionality initially determined by IMS (2003) still appear to be central for repositories in the service-oriented approach world: see for instance the OKI OSID for repositories (OKI, 2004b).

However, the Repositories Strand of the 2005 JISC/CETIS Conference (noted in the Introduction above) listed 33 kinds of functionality that may be useful in repositories, and compared their use in national learning object repositories, community image sharing resources such as Flickr, and assessment item banks (Campbell, 2005: p.7). This list was put together ad hoc by repositories experts while the equivalent assessment experts were having their meeting in other room; the next step therefore is an examination of item banks from the perspective of those developing and using them.

Defining item banks: Overview
Development of interoperable systems for e-assessment has focused around the IMS Question and Test Interoperability specification (IMS QTI). As previously noted in section 2.1.1 above, IMS QTI v2.x uses IMS Content Packaging to structure, store and transport assessment items and tests, alongside a profile of the IEEE LOM for assessment metadata, making assessment systems that use QTI potentially compatible with other e-learning systems, including repositories, which also support these specifications. The first public draft of this specification was released in 1999 (IMS QTI v0.5), with subsequent work leading to the current Version 2.1 (IMS, 2006). The specification defines its scope thus:

"'The IMS Question & Test Interoperability (QTI) specification describes a data model for the representation of question (assessmentItem) and test (assessmentTest) data and their corresponding results reports. Therefore, the specification enables the exchange of this item, test, and results data between authoring tools, item banks, test constructional tools, learning systems, and assessment delivery systems.' – (Ibid.: Overview)."

Note that there are three types of data to be managed within assessment: items, tests and results reports; these are defined below (ibid.: Assessment Test, Section, and Item Information Model ).

An item (or assessmentItem) is the "smallest exchangeable assessment object", however, it is more than just an assessment question; it contains "the question and instructions to be presented, the responseProcessing to be applied to the candidates response(s) and the Feedback that may be presented (including hints and solutions)."

An assessment test is an organized collection of items, and it contains "all of the necessary instructions to enable the sequencing of the items and the calculation of the outcome values (e.g., the final test score)."

An item bank (also called an object bank in some parts of the latest version of the specification) is a "system for collecting and managing collections of assessment items".

Results reports "report the results of a candidate's interaction with a test and/or one or more items attempted." 

Defining item banks: Item banks vs. item banking systems
The IMS QTI v2.1 Information Model does not further define item banks. However, the specification does contain some further elucidation of item banks and item banking systems in its Conformance Guide, thus (ibid. ):

"'An item bank system is a tool for managing collections of items, their meta-data, and any associated usage data."

"'A conformant item bank system allows item bank managers to import and export collections of items from item packages. Item bank systems must not alter the items' assessmentItem data. Though a given tool may combine the features of an item bank system with an authoring system, to be a conformant item bank system it must still be capable of importing, managing, and exporting collections of items without modification of the associated assessmentItem data."

"'An item bank system should create a bankProfile to describe the range of features that it supports. Version 1 of this specification described an information model for objectbanks, assessments, and results which have not been updated by this version but may be updated by future versions. Therefore, the conformance of item bank systems with respect to the interoperability of item banks, assessments, and results and the associated bankProfile class is subject to change.' – (Ibid.: 4.3 Item Bank Systems )."

An item banking system is therefore seen, similarly to the current service oriented approach in the repositories domain, as a combination of the store of relevant data (the item bank) and the services supporting use of that data. The management of assessment data is clearly complex, and an item bank is only one component of such management, as seen (right) in Figure E (IMS, 2006).

However, the level of detail shown here does not draw out the complexity within an item banking system. A learning object repository could similarly sit next to content authoring tools, content packaging tools and delivery systems, while resources in a learning object repository can be similarly complex, including simple assets or more complex aggregations of resources alongside other data about their use.

In order to further unpack the specific complexities of item banking systems, Mhairi McAlpine of the Scottish Qualifications Authority presented Figure F (right) to a meeting of the CETIS Metadata & Digital Repositories SIG in March 2006.

In this model, an item bank is: "'a store which holds the item together with its associated resources and a separate database which holds the metadata and related searchable information about the item. The item banking system is considered to be the item bank plus the functionality required to enable it.' (ibid.)"

This definition is similar to previously noted definitions and descriptions of repositories in its consideration for the storage and management of resources and metadata, and for the services or functionality provided around them. It is therefore of great interest to examine the similarities and differences between these services and functionalities for an item banking system and for other repository types.

McAlpine further teases out the kinds of resources that may be held in an item bank (ibid.):
 * Base QTI file, which describes the item
 * Called resource (such as an image, a video or animation) which appears with the question
 * Stylesheet for embedded resources (such as MathML)
 * Possibly a response processing template
 * Metadata (e.g. standard IEEE LOM metadata)
 * QTI Metadata (additional metadata which describes items; this metadata is largely a profile of the IEEE LOM)
 * Usage Data (More about what usage data is and also VDEX Glossary)

Overview of differences
It was clear from the discussion at the joint meeting on item banks and repositories held in 2006 by the CETIS Metadata and Assessment SIGs that, while there are significant similarities in functionality between repositories and item banking systems, there are some significant differences in approach that are worth noting. The major differences cluster around two areas: sharing vs. security of content, and the definition and treatment of what is termed "usage data".

Sharing vs. security
Both learning object repositories and repositories of research outputs were born out of a perceived need for open sharing of resources. The interoperability standards, practices and policies around these repository types have been geared to supporting as much easy resource discovery, sharing, reuse and repurposing as possible. Some assessment items, particularly those developed for formative assessment, may well be found useful within this culture of sharing, and it is possible that these may be stored and shared alongside other learning objects. However, a key driver within the e-assessment domain is the need for security around more high stakes assessment items and assessment tests.

Usage Data
There is often confusion between what is sometimes called "usage data" in the learning object repository domain, and usage data in the assessment domain. Learning object experts occasionally refer to the kind of information that might be shared by teaching practitioners about their use of learning objects as "usage data" or secondary metadata. Indeed, it is possible that eventually those sharing assessments via item banks may also contribute this kind of information. However, the importance and complexity of assessment usage data within item banks and item banking systems is evident, and the IMS QTI specification covers this area in detail. Whether a standard repository system can be developed to encompass generation and use of this data is questionable, however, taking a service oriented approach means that tools for this area could be developed separately, and made to work with an item bank repository.

Overview of similarities
At the previously-mentioned CETIS meeting, learning object repositories expert Charles Duncan (2006) took a repository-centric look at whether a learning object repository could function as an item bank. The 2005 JISC/CETIS Conference Repositories Strand meeting also produced a comparative table (Campbell, 2005), with a wide range of possible functionality and services that users might want. Updated versions of both of these tables are available in the Appendix below. It is clear from these, and from the examination above of repository and item bank definitions and architectures, that there is a core of similar functionality or services used in managing (in its broadest sense), resources held in a repository, and items in an item bank.

Conclusion: Implementation
This paper began by asking the question: to what extent may an assessment item bank be considered as a kind of repository, and following from this, to what extent can interoperability and minimisation of effort and resource be achieved in a manner beneficial to the related communities of interest around these technologies? Clearly, in line with current service oriented approaches to implementation of e-learning systems, an item bank may be considered as a kind of repository, within an item banking system, and ultimately, within wider e-learning or e-resource provision, so long as the above-noted differences of approach are supported.

The remaining practical question is how to implement an item bank using available repositories technologies and standards to ensure interoperability, or, conversely, how to implement a repository so that it may be used as an item bank. This question needs practical investigation, which has yet to be carried out. However, a start can be made by looking at the available technologies to consider from either standpoint. Powell (2006) noted some of the possible ways forward, with the framework of the JISC Information Environment as a guide. The JISC IE encourages content providers to "offer machine interfaces (services) that:
 * expose metadata for harvesting
 * expose metadata and/or full-text for searching
 * expose resources at persistent URIs
 * expose news-feeds
 * expose context-sensitive links to resources when appropriate." – Ibid.

It also encourages everyone else to: "build user-centric and machine-centric applications on top of those machine interfaces." (Ibid.) This basic framework can therefore be used to begin thinking about how to implement interoperable item banks.

Implementing item banks as repositories
There are a number of standards that can assist with implementing the above-mentioned services. This paper has previously noted IMS Content Packaging, IMS Question and Test Interoperability, and IEEE LOM metadata as being the basic specifications to examine. Given the previously mentioned concern regarding security for item banks and item banking systems, it is worth noting that the above services do not have to be completely open to the public. Some attention should be given to access and authentication issues: Shibboleth is the emerging standard in this area.

Table A: Repository services and standards (Powell, 2006)

Implementing repositories as item banks
Table A above should be noted in developing a repository that is interoperable within the JISC Information Environment and beyond. In addition, for those developing repository systems, the following points should be taken into account when considering implementing them as item banks:

Assessment items
Under IMS QTI v2.x, an item, assessment or test, along with metadata and supporting resources, should be packaged using IMS Content Packaging. Moreover, many implementations of previous versions of QTI also use IMS CP. Therefore, for the sake of taking advantage of repository and other interoperability standards and specifications, supporting IMS Content Packaging may be assumed as a starting point. IMS QTI also defines a profile of IEEE LOM metadata for items, therefore support for this standard should also be provided. In addition, cool URIs should be assigned to every item and other resource that may need unique identification within a repository.

Item banks and harvesting
As previously noted, IEEE LOM metadata should be supported. For the sake of supporting OAI-PMH metadata harvesting, a repository purporting to provide item banking functionality should also be able to expose a simple DC metadata record for every item in the item bank; some way of mapping IEEE LOM metadata to DC should therefore be available.

Repositories, item banks and a Web 2.0 approach
While this paper has focused on the use of JISC Information Environment standards and specifications as a starting point in building item banks as interoperable repositories, there are limitations to this approach. Like Web 2.0, the JISC IE encouraged openness; however, unlike Web 2.0, it didn’t engender participation by end users. Some ideas for a more Web 2.0-based approach include consideration for:
 * generating a Google sitemap for every IMS Content Package in the item bank;
 * browser sniffing/content negotiation to serve an HTML view of every IMS Content Package to Google and other robots;
 * indexing the metadata and item content and exposing the search interface using A9 OpenSearch;
 * optionally exposing news feeds as Atom, as well as RSS.