An Evaluation of Information Technology Projects for Higher Education
Back to Shirley Alexander Publications | Back to Staff Listings
Shirley Alexander & Jo McKenzie
Institute for Interactive Multimedia
University of Technology, Sydney
Australia
Executive Summary
The aims of the study
The main purpose of the study was to determine how the use of information technologies in university learning and teaching benefits student learning. The results of the study will help decision-makers to make more informed judgements about the costs and benefits of introducing greater levels of information technology to university learning and teaching. The study will also enable staff working in diverse contexts to maximise student learning outcomes, through the use of appropriate learning strategies and information technologies.
The study
The study reviewed 104 of a total of 173 projects which received funding from the Committee for the Advancement of University Teaching (CAUT) in 1994 or 1995, and which made significant use of a range of information technologies to develop student learning materials. The project leaders of 104 projects responded to a questionnaire on project design, development, evaluation and implementation. From those projects, 20 projects, representing a diversity of types of information technology, learning design and discipline areas, were selected for a more detailed study through project document analysis and interviews with staff and students. From these, a further eight were selected as major case studies. One additional case study which did not receive CAUT funding was also developed to complete the range of information technologies and discipline areas represented in the study.
The results
The project considered both tangible and intangible costs and benefits of projects within the context of a range of objects affected by the project: students, staff, departments, institutions and society.
Costs
The total of the CAUT funding received by the 104 projects reported in this study was $4,247,538. A number of costs were also identified for a range of stakeholders in the project.
Very few costs were borne by students, with the exception of one project which required students to have access to a computer and a modem. Some students reported a cost reduction, as they no longer had to pay for travel to undertake particular learning experiences. Many of the staff involved in projects incurred a high cost in terms of time, resulting in loss of research and personal time. In some cases this had a negative impact on their opportunities for promotion and tenure. For some departments and institutions, the cost of making the information technology available was high, as was the cost of the support infrastructure.
Benefits
Although the focus of the study was on the benefits for student learning, a range of other benefits, both tangible and intangible, were identified.
Benefits for students
The benefits of the projects for students were of four kinds:
- improved quality of learning;
- improved productivity of learning;
- improved access to learning; and
- improved student attitudes to learning.
Examples of positive learning outcomes which resulted from students' use of information technology (IT) projects, as illustrated by the major case studies, include:
- the opportunity for students to interact with others internationally and gain a more sophisticated and global understanding of complex international political issues, while gaining information technology literacy in the process;
- improved understanding of concepts which students are known to have difficulty with in a range of disciplines, through the use of interactive multimedia animations, simulations and microworlds;
- the development of information and technological literacy in the context of learning to solve real-world problems through the use of databases and e-mail;
- enhanced communication between part-time students and their lecturer, through the use of a computer-based conferencing tool over the Internet;
- the acquisition of information such as language learning, where a high component of factual recall is required;
- learning the skills and knowledge of a particular discipline in the culture of its use in a working organisation, through participation in a simulation over the internet;
- the facility for students to assess their own learning of concepts, through computer-based qualitative and quantitative assessment modules.
Benefits for staff
The benefits for staff of involvement in the projects included: job satisfaction flowing from the improved learning of their students; increased understanding and skills in the use of information technologies; an improved understanding of student learning, student needs and difficulties; an improved understanding of their own discipline area; enhanced enthusiasm for teaching; and, for some, an increase in personal profile.
Benefits for departments, schools or faculties
The major benefit for departments was the staff development opportunity afforded by individuals' participation in the projects. This sometimes led to significant changes in teaching approach in areas other than the designated project, as staff developed enhanced understanding of learning and teaching. For some departments, the project outcome helped staff to cope with decreased resources, without a commensurate decrease in the quality of teaching. Finally, the teaching profile of some departments was raised as a result of external recognition of the innovation.
Benefits for institutions
The major benefit of involvement in the projects for institutions, was the resulting enhancement of their reputations as innovators in teaching and in the use of new technologies. Some used the projects as examples of good teaching in their submissions to the DEETYA Quality Rounds, while others show-cased the projects at recruitment exhibitions in Australia and overseas.
Benefits for the community
The community is the ultimate beneficiary of the outcomes of these projects, particularly when they result in improved learning. Some project outcomes (CD-ROMs, videotapes, etc.) are also used by professional bodies, high schools and other community groups.
Factors affecting the outcome of information technology projects
This study highlights the range of factors that have an impact on the development of information technology projects that aim to improve learning in higher education. The use of a particular information technology did not, in itself, result in improved quality of learning or productivity of learning. Rather, a range of factors were identified which are necessary for a successful project outcome, the most critical being the design of the students' learning experiences. In the absence of evaluation data in a large number of cases, judgements were made about the success of the outcome by the team conducting this evaluation, after reviewing the project's use and interviewing staff and students where possible.
Factors contributing to a successful learning outcome
The composite profile of a project which is successful in improving learning in the originating institution, is one which has the following features:
- it aims to address a specific area of student need, such as: making it possible for students to learn particular content which is known to be difficult or not well understood; and/or providing individualised feedback to students where this is not available through other means; and/or providing tools which facilitate learning activities not previously possible;
- it uses a learning design/strategy which has been well thought through, and is often based on current and relevant literature on education in the discipline and/or of good teaching in higher education;
- the way the project is integrated into the learning experience is well thought through and implemented, and the support needs of students and staff are identified and planned for;
- the assessment of student learning is modified where necessary to reflect any changes made to the content and process of learning as a result of the project;
- the anticipated outcome is realistic, in the context of the time and budget available;
- in the case of projects which require significant software development, the project has been adequately analysed, planned, scoped and designed prior to commencing the development;
- the development team includes a skilled project manager, who carries out that role throughout the life of the project;
- the project team has adequate access to technical support and educational software development expertise;
- the project team has shared goals, and is able to resolve any conflicts which arise;
- individual members of the project team are committed to the project and have adequate time to carry out their roles and responsibilities in the project (eg. through release from teaching);
- issues of copyright and intellectual property ownership are resolved prior to commencing the development;
- evaluation of both useability and student learning is carried out at regular stages during design and development of the project, and the project is re-designed as necessary;
- evaluation of student learning is carried out in the same or similar context to the one the completed project will be used in;
- students have adequate access to the hardware and software required for implementing the project;
- staff and students have adequate access to the range of supports required for implementing the project;
- where required, sufficient funding for implementation of the project is available;
- the Head of Department/School and the Dean are supportive of the project, recognise the value of the project to the department or faculty, and are committed to its implementation;
- the institution's promotion and tenure policies recognise teaching developments as a significant contribution to the university;
- sufficient funding for the maintenance of the project is available.
In addition to the above profile, projects which have been adopted in multiple institutions are those which typically:
- include academics from more than one institution on the development team; and/or
- assist students to learn content which is common to a range of institutions;
- are relatively easy to implement in a new environment;
- have received larger amounts of funding than the original CAUT grant.
Factors contributing to an unsuccessful learning outcome
The profile below represents projects that in general were not successful in achieving the desired learning outcomes. Each of the features described below was encountered in at least one project in the study. For many projects the presence of only one of the following features was sufficient to result in its failure to achieve desired outcomes. A very small number of projects were able to achieve a positive outcome despite the presence of one or more of the following features. In these rare cases, the turn-around was a result of the project leader recognising the problem early enough to rectify it.
Projects which were not successful:
- were overly ambitious in terms of desired outcomes for the budget and time available;
- utilised particular information technologies for their own sake, without sufficient regard for appropriate learning design
- did not change the assessment of learning to reflect changed learning outcomes;
- failed to recognise the importance of the project's context of implementation and the need to think through and plan for this;
- commenced software development without adequate planning;
- did not have access to adequate technical advice, expertise and support;
- acted on technical advice provided by people lacking in the necessary knowledge and skills to provide such advice, especially in relation to the selection of hardware and software;
- did not have access to adequate relevant expertise (where projects involved significant software or multimedia development);
- had academic team members who felt they could perform all the technical functions, such as programming, graphic design, etc., but were not able to do so;
- had staff on the project team who did not value the different skills required and available for the successful project completion;
- had project teams which were unable to resolve differing opinions;
- had a project development team which did not include a member with responsibility for project management, and which did not foresee the need for project planning and/or documentation;
- had a project leader who, in view of his or her teaching release to develop the project, was allocated an extra administrative load by the Head of Department;
- did not adequately prepare students for participation in learning experiences which they had not encountered before, such as working in groups;
- over-estimated students' willingness to engage in higher level learning activities, especially when they were not related to assessment;
- used resources in the project development for which copyright clearance had not yet been obtained, and could not subsequently be obtained;
- had a project leader who was located in a faculty or school where the Head of Department was not supportive, often because he or she felt the time would be better spent on research, or did not value the project;
- developed a project which was operational on the development computer only, and could not be run on the implementation computers because of inadequate memory, disk space, etc., or because of non-existent CD-ROM drives;
- developed a project for implementation on computers which were expected to become available in the future, but which did not become available;
- conducted evaluation (if at all) only when the project was complete, and discovered that changes were required for which funds were no longer available;
- conducted limited or poor evaluation of the project because of lack of time and/or budget and/or evaluation expertise;
- did not evaluate the project in the anticipated context of use, prior to implementing it.
Although the majority of projects were not implemented beyond the institution in which they were developed, there was also evidence that some projects were not fully implemented within the originating institution. These projects typically ceased to be used when the project leader left the institution, or was allocated a different teaching load. Projects in this category were those which:
- were developed to assist students to learn content which was of interest only to the project leader, and hence was not embedded in the department's normal teaching;
- were developed within departments which did not value scholarship and innovation in teaching;
- had complex implementation requirements, resulting in significant time and risk for the academics choosing to use them.
Recommendations of the study
The major recommendations of this study concern the need for information technology projects to be developed in the same scholarly and professional manner that is currently exhibited in other areas of academic work, such as research.
While much of the early development of information technology projects has been the domain of the enthusiastic experimenter, significant educational software development has become a professional and multi-faceted activity, requiring the interplay of expertise in learning design, project management, financial management, interpersonal skills, programming, graphic design, media digitisation and evaluation.
This study makes the following recommendations for improving the outcomes of information technology projects.
Funding
The study recommends that:
- Projects which involve significant software development receive funding allocation in three separate stages, with funding of each stage contingent upon successful completion of the previous stage. The three stages are:
- project planning and evaluation;
- project development, implementation and evaluation;
- project maintenance.
- Sources of funding be identified for the purchase of IT projects developed elsewhere.
- Funding opportunities be identified to enable institutions or departments to make any necessary modifications to IT projects developed elsewhere, thereby enabling their implementation at more than one institution.
- A higher level of funding be allocated to projects which include partners from more than one institution.
- Priority in funding be given to projects which are linked to the strategic plans of the faculty or institution.
- Project developers be encouraged and supported in seeking additional funds through alliances and partnerships, to facilitate continuous improvement of their project.
- They should focus on a clearly identified need, with supporting evidence that the need exists.
- They should articulate a clear learning and teaching strategy for project development, which is based on current and relevant literature in the discipline and in higher education literature, and explain why they expect the project to be successful.
- The project plan should include a business plan which identifies the costs and benefits of the project for students, staff, the department and the institution, as well as providing evidence of market research which supports the potential use of the project outcomes, outside the originating institutions.
- The project plan should nominate the person (or the process to be followed in securing the person) who will act as project manager, and provide details where possible of that person's previous experience in project management of software development.
- The project plan should provide details of the source of technical support, expertise and advice to be used in the project.
- The application should include a comprehensive evaluation plan which articulates a well developed strategy, and identifies the resources required to implement the plan.
- The application should include a detailed implementation plan which identifies: the learning context in which the project will be used; the support which will be available to students and staff using the project; and the computers which will be used by students (where applicable).
- The application should include a detailed implementation plan which identifies: the learning context in which the project will be used; the support which will be available to students and staff using the project; and the computers which will be used by students (where applicable).
- Staff development opportunities be provided in the areas of project management, working effectively in teams, evaluation of IT projects, and legal issues related to IT development, for current and potential project leaders.
- Staff development opportunities be provided in good practice in teaching.
- Opportunities be
provided for all team members who have developed successful projects to share
their experiences and products with others.
Staff support
The study recommends that:
- Institutions which
make a significant investment in the use of information technologies in learning,
maximise their investment by:
- providing or identifying ways in which staff might receive teaching and learning, technical and evaluation support; and
- (where institutions
have policies and procedures in place) recognising and rewarding successful
IT project developers.
Project design and development
The study recommends that:
- Academics participating
in IT design and development projects are encouraged to:
- apply good teaching practice to the design, development and implementation of IT projects;
- consider revising the method of assessment of student learning to reflect changes in the content and process of learning, which are expected to occur as a result of students' use of the project;
- initiate activities which encourage students to reflect on their own learning in terms of content, process and the ways in which the IT project is encouraging an approach which they might not have encountered before;
- provide support for students who are engaging in learning activities, such as working in groups, that they have not encountered before.
Dissemination of projects
- Institutions are advised to review their policies of intellectual property and copyright to reflect the issues resulting from the development of information technology projects.
Project selection
Projects which are selected for development should exhibit the following features:
Staff development
The study recommends that:
