How Effective are Online Assessment Systems?

This guest post is contributed by April Davis. She raises several important issues but her description of the existing scenario raises questions for me about whether we should be expecting more from 'assessment'. Should assessment be only summative, or if formative, when? - and by whom? Does assessment really help the learner? April writes:

"There are many ways for students to assess their capabilities, and online assessment systems are just one of these. They’re being increasingly used today, not just by organizations and educational institutions to gauge if students and employees will be a right fit for the job or the degree, but also by those who want to assess themselves and see how good their chances are of getting into a course or gaining employment in the organization of their choice. Most online assessment exams test attitude and personality (or psychometric capabilities), and pre-employment skills. The efficacy of these assessment systems depends on a number of factors:

• The standardization of the tests – an online assessment system comprises a question bank and a question engine. The question bank must hold standardized question definitions in organized categories and contexts and the question engine must be able to pull these definitions based on user interaction. Also, the organization using the test must be sure that the assessment and analysis returned by the system are in conjunction with their values, beliefs and needs.
  
  
• The robustness of the system – since the tests are held online, the system has to make allowances for unexpected occurrences like lost connections, users pressing back and forward buttons indiscriminately or closing browser windows by mistake, server crashes, broken connections to the database and loss of data that has already been filled, and so on. Where these tests are timed the above problems could influence the accuracy of the results.
  

"For organizations that use these assessment systems, the relative success or failure of these tests can be gauged only in hindsight – the performance and attitude of the employees that the test recommends, when assessed over a period of time, helps companies decide if online assessment systems are effective in their hiring policies.

"However, it should also be noted that these tests are not a perfect science and that every employee changes in their attitude and personality based on the nature of the job and the environment in which they work. So while the test may throw up a few suitable candidates, it’s unrealistic to hope that they’re all going to be a good fit. The key to assessing the efficacy of these tests with some degree of accuracy is to give them some time, use a large number of employees as a test pool, and allow a few variances in your interpretation of the results."

By-line:
This guest post is contributed by April Davis, she writes on the topic of Accredited Degree Online . She welcomes your questions and comments at her email id: april.davis83(@)gmail(.)com.

E-portfolios for apprentices

E-portfolios for apprentices: A guide for providers and employers. Available here.

This document, published last year by Becta is an important and well collated document. However, readers will soon be aware that it does not provide a clear sense of direction, but, as many reports do, it only describes the current mass of confusions relating to ePortfolios. The fundamental error, in my view, is that the authors still see different functions for ePortfolis relating to the varied perspectives of exam-boards, training-agencies, universities, FE colleges and employers. And thus one learner could be required to hold several different ePortfolios for different purposes - what a mess!

The opening statement in their conclusion is indicative:

"This report has reviewed the existing published information and guidance on e-portfolios as they relate to apprenticeship programmes. It is evident that there is a considerable body of information on e-portfolios but very little of it is targeted at providers and employers offering apprenticeships or at apprentices themselves."

My question therefore is 'if the education providers, the employers and the apprentices themselves do not know about ePortfolios then, who on earth does???'

Perhaps the Australian work on VET should be given better publicity here in the UK? Click here for more from the Australian Flexible Learning Network

Building Communities and Networks

I am very grateful to Jeff Utecht for releasing as a free copy, his book 'REACH - Building Communities and Networks for Professional Development'. The book is easy to read and is an excellent introduction to what the 'C' in ICT is all about. I would think that the book could easily be used as a pre-reader for a staff development day or series of workshops. My only wish was that this book could have been available some ten years ago!

Although the word 'e-Portfolio' is strangely not mentioned in the book, the whole mind-set that Utecht describes is that of the e-Portfolio user. It is not surprising, therefore, that he has written another book, 'Blogs as Web-Based Portfolios'.

The Blind Leading the Blind?

As an avid e-Portfolio watcher I spend hours every day reading students' e-Portfolios and academic articles, from the most naive to the extremely erudite. However, what was exciting some four years ago has made me become increasingly frustrated as I repeatedly read the same sort of academic papers describing small-scale 'Pilots' which never seem to get anywhere. Similarly, students from all around the world write of their requirements to explore, investigate or design e-Portfolios as an ICT competency excercise imposed upon them by their tutors.

This set me thinking: Why is it that apparently intelligent people, tutors or faculty in particular, appear to lack any reasonable knowledge of the full functionality or benefits of using an e-Portfolio system?

After some 20 years of e-Portfolio thinking it seems strange that so many school leaders have no working familiarity with the tool. If students can benefit so much from e-Portfolios, what confidence can they have in faculty who don't use them. What sort of duplicity is it that says, 'You must do this, but I don't.'

Perhaps one might ask, 'Who blindfolded you?' Is it the institution that says this would cost too much to implement? Is it faculty who say we haven't got the time? Is it educators who have not adopted Web2.0 thinking? Have staff been confused by misleading advocacy? Or is it an hierarchy that intend maintaining the status quo?

Perhaps nearer the truth is the fact that many students can see through the weaknesses of a profession that relies on orthodoxy rather than exploring a brave and exciting new world of Teaching & Learning?

As I have said in several other places, 'Without a vision the people perish!' No where is this more true than in the world of e-Portfolios. Let us therefore stop all 'Pilots' and get involved in implementing whole-school policies for e-Portfolios, NOW!

ascilite 2009

An impressive paper by Gordon Joyes and Lisa Gray(In Same places, different spaces. Proceedings ascilite Auckland2009) entitled, 'Effective practice with e-portfolios: How can the UK experience inform practice?' makes a good read, even if only 10 pages. As Lisa Gray reports, '... Gordon Joyes, myself (Lisa) and Elizabeth Hartnell-Young presented on our work at the Ascilite conference in Auckland in December, and won an 'outstanding paper award' for it .'

Under the heading 'Why is JISC exploring e-Portfolios' the following extract should be noted:

"Before learners enter higher education,within schools and 14-19 sectors, there has been an increasing focus on a national curriculum that values skills. A new qualification, the Diploma has emerged, with the development of personal, learning and thinking skills (PLTS) at its core. e-Portfolio technologies provide ways in which these skills can be evidenced. But perhaps the most important reason for considering the potential of e-portfolios to support learning and teaching is the emerging evidence from practitioners and learners of the value of developing e-portfolios, not only to support more profound forms of learning, adding value to personalised and reflective models of learning, but also facilitating the transition between institutions and stages of education, supporting application to education and employment, staff appraisal and applications for professional accreditation, and supporting learners based in the workplace. "

However, many of the points referring to mainstream education remain undeveloped. As part of my ‘evangelical’ efforts to promote e-Portfolios in mainstream education, therefore, I wonder if we should be creating a paper entitled,

‘Effective practice with e-portfolios: How can the UK HE experience inform practice in mainstream education?

To this end I would dearly like to discuss with practitioners (possibly at Cumbria?) how the lessons learnt in HE can be developed for a nationwide and all-age implementation.

The end of the BB e-Portfolio?

A recent post from Mr Woo identified one of the weaknesses of how many young learners see the e-Portfolio. But perhaps, as other Blackboard users have commented about their difficulties in using BB in a liberated teaching & learning environment, I should not be surprised.

I immediately thought of several previous posts I had written on the subject of the storage of artefacts. (see 'Ethics or common sense?' and 'Moving House?') I would not wish to say outright, "I told you so!" but I fear that in far too many cases the designers of so-called e-Portfolio systems are not thinking through the implications of their work, nor are course designers spelling out the practicalities of their strategies.

Do you own a Rolls Royce?

It's no good having a Rolls Royce, the latest high-performance Jaguar or a Lamborghini parked outside your house if there is no one to exploit its real potential or for its passengers to gain a wonderful enjoyment of the experience.


So, too, with all the recent innovations that we are presently experiencing in schools. For some schools the Interactive Whiteboard (IWB) is such a 'white elephant' where, despite the 'oos' and 'ahs' of the visitors the actual passengers (the pupils) do not rate the experience and the driver (the teacher) has hardly gained a licence to drive the IWB! In fact, some schools are disposing of their IWBs, generally due to the lack of confidence of the driver to actually get in and try the 'brute' out. Rather than being so frustrated, I would suggest that more teachers should actually get into the driving seat of their IWB, VLE or e-Portfolio and actually begin to enjoy the experience.

I'm presently writing this post at lunchtime at the ORT International VLE Seminar week - which also happens to be an opportunity for this international audience to visit BETT. I was priveledged to present a keynote presentation on 'VLEs - Evolution or Revolution?'. The reason for mentioning this is that I have been very impressed by the clarity of thought concerning VLEs in schools. If a country like Argentina can produce really creative resources by all of its teachers then perhaps teachers in the UK had better look out. Their attitude to sharing work through Creative Commons is far more productive than the protectionist IP lobby that some feel obliged to follow.

But I really must get back to the theme of e-Portfolios and ask that we start to share good practice and, like the Argentinians, begin to develop a more creative and exciting learning environment for all our students, staff and institutions included!

41 Benefits of an e-Portfolio?

The recent post by Karen Barnstable is an excellent collection of benefits that is hard to improve upon. However, all readers from around the world should read her post and preferably comment back to me or Karen or both!

She clearly identifies three separate areas of benefit: the Student, the teacher and the institution. However, whilst writing this, I realised that she missed out another area possibly the most important, particularly for younger learners, that of the parent.

Instructional Technology in Schools

"The Use of Instructional Technology in Schools examines teachers' use of the major instructional technologies over the last century — from the days of silent film, radio and slide shows through to the modern interactive whiteboard and the Web. It explores the reasons why so few teachers have used these technologies and why, even in today's digital world, the most commonly used classroom tools are the pen, paper and teaching board."

I make no apology for including this 'advert' as I helped in some of the discussions with Mal Lee about the evolving use of technologies in the UK. However, I have just been informed by Eurospan Limited, the UK branch of ACER Press, that they are offering a 15% discount on Mal Lee's books to educationists. The 'blurb' goes on to say:

"The book provides decision makers with an invaluable insight into the million dollar question: What is required to get all teachers across the nation using the appropriate instructional technology as a normal part of everyday teaching? Without question, student learning is enhanced by adopting these new technologies.

"Until now, research on why the majority of teachers use only the most basic tools in the classroom has been scarce. The Use of Instructional Technology in Schools examines this phenomenon and, most importantly, identifies what is required to achieve teachers' universal acceptance of instructional technologies."

About the authors
Mal Lee is an educational consultant specialising in the development of digital technology in schools. He is a former director of schools and secondary school principal, who has written extensively on the effective use of ICT in teaching practice. Mal Lee is co-author of Leading a Digital School: Principles and practice (ACER Press, 2008) and The Interactive Whiteboard Revolution: Teaching with IWBs (ACER Press, 2009).

Arthur Winzenried has taught in a wide variety of library and classroom settings, from primary to tertiary, and has been responsible for innovative developments in school ICT over a number of years. He is a frequent national and international speaker on knowledge management and is currently Lecturer in Information Systems at Charles Sturt University, NSW, Australia.