CETIS 2006 Assessment and Personalisation

This session looked at various factors surrounding the personalisation of online assessment. The session will featured short presentations from domain experts on various topics to help spark and focus discussion, but the emphasis of the strand was on discussion and participation by all.

Participants
The interests of participants were wide-ranging:
 * informal and formative assessment;
 * eportfolios;
 * feedback for formative assessment;
 * data transfer;
 * 14-19 specialised diploma complications;
 * admissions;
 * personalisation;
 * the 'bigger picture' of assessment and eassessment;
 * services to support assessment;
 * encouraging community involvement in FREMA, the eassessment reference model project;
 * moving from traditional to untraditional assessment;
 * links to repositories;
 * new opportunities;
 * moving from CBA to online assessment.

Links to identity, portfolio and personalisation session
Clive Church chaired the session, and opened by discussing some of the issues raised in the previous day's identity, eportfolio and personalisation session:


 * Awareness of different identities in different contexts, particularly for lifelong learning. There is a need for an identity management system to support different identities for different contexts.
 * Interest in the DfES thin portfolio model for moving through educational stages and supporting the specialised diploma. In England, Becta has a deadline of 2008 for implementing the specialised diploma which will rely on portfolio-based evidnece and the ability to move information across awarding bodies and insitutions.  This raises a pressing need for interoperability.
 * Personalisation was implicit in all the discussions. It can be supported by web services based architectures which can meet a wide range of needs.
 * There are important issues around authentication, accessibility and data protection which need to be addressed.
 * Personalisation in the portfolios context means knowing about people as individuals as well as cohorts, etc; issues around the aggregation of information.
 * Learning agreements and the 14-19 diploma will affect all sectors. There are still very traditional views on personalisation that we need to be aware of when designing systems.

Workflows and services for personalised assessment
Dave Millard and Yvonne Howard from the University of Southampton presented their work on modelling the assessment domain in the FREMA project.

FREMA didn't look at portfolios or personalisation, rather concentrating on other factors and expressing services in the eframework in the way JISC required. FREMA illustrates the framework of services available for assessment. Personalisation isn't a scenario in itself, but rather fits into many other scenarios. Enabling personalisation involves rewriting workflows not services, or introducing new services to existing workflows.

The project developed a semantic wiki, available at http://denebola.ecs.soton.ac.uk/wiki, which is intended as a community website for developing and disseminating the model. A semantic wiki is a wiki in which all the pages and linkes are typed, which enables smart searching and analysis. The wiki enables representation of a gap analysis, service expressions for service usage models, service responsbility and collaborations and vocabulary for discussion. It offers opportunities for collaborative refactoring of the model, and a community-orientated way of working. There are many use cases around assessment which haven't yet been described: the wiki offers an excellent way of capturing and analysing this data.

Technical barriers and opportunities for personalisation in QTI
Steve Lay, of the University of Cambridge and co-chair of the IMS Assessment working group which develops and maintains the IMS Question and Test Interoperability specification, discussed some of the issues related to standardised assessment.

Lack of interoperabilty is a barrier to personalisation, yet in some forms of personalised assessment, particularly adaptive testing, there is no desire to interoperate because of commercial sensitivity.

However, the QTI specification does offer opportunities for personalising assessment. Steve demonstrated some of the capabilities of QTI using Graham Smith's Java implementation which can be viewed at http://qtitools.caret.cam.ac.uk/qtiv2/examples/V2examples.html. Interestingly, while Ernest Adams had been careful to warn educators about the 'smoke and mirrors' aspects of game functionality, the Monty Hall question Steve demonstrated uses similar 'cheats' to ensure that the learning outcomes of the question are met.

There was a lot of interest in the range of tools available, and in particular in reusing existing toolkits in modular systems to support personalisation.

Steve's presentation is | available from the QTITools website; you can also view his presentation to the identity, portfolio and personalisation session there.

JISC's eassessment priorities
Lou McGill, JISC programme manager with particular responsibility for assessment, outlined JISC's thinking on eassessment and discussed some of the priorities for the domain that have been identified. In a very lively session, she sought feedback from participants on the priorities and workplan that have been identified, and stressed the desire for consultation with the community to help move forwards.

Areas of interest that arose from the discussion included:
 * connecting sectors and domains;
 * making tools usable for people withough technical knowledge;
 * text assessment - a hugely complex area;
 * feedback;
 * connecting areas where there are currently missing links, such as between the JISC Plagiarism Service, augmented marking services and free text marking;
 * accessibility;
 * tagging.

The awarding body perspective
Clive Church, who works with both Edexcel and CETIS, outlined some of the issues currently important for awarding bodies.

The main priorities are currently:
 * online assessment;
 * coursework assessment;
 * onscreen marking.

CAA is seen as offering cost and time savings and easing the quality assurance process.

The 14-19 specialised diploma and 'componentised' awarding (in which various organisations are responsible for accreditation of components of a diploma which is itself awarded by a diploma awarding body) raise particular challenges. Interoperability is clearly a major priority in this area, and must be addressed urgently given the September 2008 deadline for first delivery.

The Qualifications and Credit Framework is a government-inspired log of all a learner's assessment successes throughout their lifelong learning career. These national activities will inevitably impact on the HE sector.

A realistic perspective on personalisation
Kevin Donovan, of http://www.kevindonovan.co.uk, offered an entertaining and stimulating sideways look at personalisation.

Nobody really seems to know what personalisation actually is. It's perceived as adapting content and contexts to suit individuals, fitting the needs of individuals. It is seen in the context of monolithic environments and is a recurring theme across political agendas: it is regarded as being synonymous with 'choice' in contexts such as patient choice and consumer choice.

In reality there is a huge contradiction between personalisation and existing structures, particularly in the public service context. Personalisation isn't necessarily a good thing: a common currency, for example, is an asset rather than a hinderance to the 'average' person. Academic education is seen as more flexible and prestigious than training, yet the most prestigous university courses of all are vocational: law and medicine. Personalisation is just another paradox amongst many!