Development and trialling of an online diagnostic academic literacy test
Liz Craven, ELSSA Centre
At the request of the Faculties of Engineering and Information Technology, the ELSSA Centre has, for a number of years, conducted diagnostic academic literacy assessments of students enrolling in the first year of their studies at UTS. The purpose of these assessments has been to make decisions about the level and kind of academic literacy support the ELSSA Centre can offer different categories of students in these faculties. This testing has been paper-based and expensive. In Spring Semester 2007, I was awarded a Small Learning and Teaching Improvement Grant to develop a diagnostic academic literacy test that could be administered and marked using the test tool in UTSOnline. This poster presentation outlines some of the lessons learnt in the development and trialling of an online diagnostic academic literacy test for Communication for IT Professionals (CITP), a core first year subject in the Faculty of Information Technology. While a number of problems were encountered in the trialling, the time saved in the marking of the test was considerable and thus it is recommended that greater use be made of UTSOnline for diagnostic academic literacy assessment.
The diagnostic assessment consisted of a reading passage with multiple-choice reading comprehension questions, a summary writing task and a fill-in-the-blanks task testing grammar and vocabulary usage. Forty-five students took part in the trial: 22 completed the assessment online and 23, for various reasons, on paper. Students who completed the assessment online did not appear to be disadvantaged in any way.
The results were used to allocate students to the class that would best serve their academic literacy development needs. The subsequent performance of the students in class confirmed the appropriateness of the decisions made based on the assessment results. Many students gained similar marks in each of the three sections of the assessment. In cases where the mark for the reading comprehension was higher than that for the writing, the writing mark provided the better predictor of the student’s subsequent performance. This being the case, it is proposed that, in future, a diagnostic academic literacy assessment could consist of a writing task only. If the writing task is a summary of a suitable article, this task can assess academic literacy proficiency in an integrated way. It would, however, be necessary to select an article relevant to the students’ field of study to give credibility to the assessment and thus to reduce the negativity that some students display towards having their ‘English’ tested.
There are many advantages in using the essay function in the UTSOnline test tool. Students today are at ease with using computers to write and the product is easier to read. An important advantage is that a team of markers can access the same examples of writing at the same time and moderate their marks without having face-to-face contact. Each student’s summary can be cut and pasted into a Word document if required and markers can quickly compare and contrast marks they have given for each sample to check that their marking is consistent.
As a language and literacy lecturer with the ELSSA Centre, Dr. Liz Craven has worked with students from most faculties at UTS providing various types of academic literacy support. She has a particular interest in the development of tests for assessing language proficiency and in exploring the possibilities of online environments for provision of academic literacy support.