Doing being applied linguists: the importance of experience
SEVEN CASE STUDIES
Language-program evaluation
Accountability has traditionally been left to professionals to determine for themselves. It has been manifested through such stakeholder satisfaction criteria as client numbers, student successes on examinations and in employment, earnings and reputation. Such amorphous criteria are no longer acceptable. For the sake of the stakeholders and to make the participants better informed, as well as to improve the activity if repeated, language-program evaluation is now widely practiced. What it does is to determine to what extent the project/program is meeting the original blue-print, to examine the changes brought about by the project/program, and to question the extent to which this type of project is generalizable and should be generalizable. Was it worthwhile? Can we generalize to other situations?
In 1990 Jacob Tharu, of Hyderabad CIEFL, and I carried out an evaluation study of four projects in South India (Davies 1991b). The evaluation was concerned with externally funded English-language teaching (ELT) projects in tertiary institutions. The funding source was the British Government, through its Overseas Development Administration (ODA) and the British Council (BC) under the Key English Language Teaching (KELT) scheme. All four projects were (untypically for KELT) short term and made use of a two-way relationship between the (British) consultant’s home institution and the Indian receiving institution, following a pattern of two-way visits over three to four years. The purpose of the evaluation was to determine what success such a project using short-term consultancies had had and to consider whether or not such a model could be applied in other developmental situations.
These four projects, institutionally separate from one another, were all concerned with curriculum change. Our terms of reference were as follows:
1. the overall design of the projects and their relevance to the Indian situation;
2. the effectiveness of the UK consultancies and of local input/support;
3. the appropriateness of materials produced and their usefulness to the target audience;
4. the extendibility of the ELT materials to other situations in India;
5. the changes that were brought about as a result of the project; and
6. the extent to which local expertise could take over and sustain the work of the project.
The four projects were based in:
Anna University, Madras (English Department)
Kerala University, Trivandrum (Institute of English)
Indian Institute of Science, Bangalore (Foreign Languages Section)
Osmania University, Hyderabad (English Department and ELT Centre)
The range of undergraduate/postgraduate, compulsory/special, full-time/parttime, general/specific, large/small departments against an underlying policy of curriculum change compelled us to consider urgently the need to generalize beyond the context of any one setting.
What became clear early on was the difficulty of determining any single criterion of project success, thereby supporting views widely expressed in the literature (e.g. Brumfit 1983; Kennedy 1989; Weir and Roberts 1994; Baldauf and Kaplan 1998). Success in a project may be achieved in a variety of ways and depends on a combination of factors, such as context and personal interactions, not all of which are manipulable. For that reason we were less concerned with analysis of past achievement and more concerned with diagnosis of project experience so as to inform future policy.
We decided on four criteria for determining success of a project: product, teacher development, sustainability and extendibility. By product we meant some public expression of a project outcome. At its most informal such a public expression could be a circulated syllabus document; at its most formal a published textbook. What we looked for was some product indicative of project completion; we did not attempt to estimate the professional quality of the product.
Teacher development, the second criterion, is essential to the continuation of an institution. And while the language-teaching profession may be more concerned with research output, administrators are probably more well-disposed to the professionalism of their institution’s teaching staff. We determined on a number of indicators to show professional development, such as recognition of the necessary link between materials and methodology, appointment as consultants to other institutions, stated intention to update their materials.
Sustainability has to do with the ability and willingness to continue without the support of the consultant. We decided on indicators such as: being responsive to the need to change aims while the project was still ongoing, team cohesion shown by a strong sense of professional interaction and a sense of ownership of the project.
Extendibility concerns the relevance of a project to other contexts and therefore is determined by indicators such as an understanding at a theoretical level among the project team members of why they did what they did in the project, an awareness by professionals in other institutions of the seriousness of the project, and a capacity by the project staff to continue as a research team and mount new projects on their own, not simply continue the existing project.
We considered that in addition to these four project outcomes it was also necessary to take account of a set of pre-conditions and of inputs during the life of the project. In this way we developed a model for project evaluation which would permit both generalizability across KELT activities (and no doubt others too) and at the same time allow for some measure of prediction of likely success based on the presence of the pre-conditions and the amount of input during the project.
Evaluation of language-teaching projects is a good example of the kind of activity applied linguists are called on to perform. What makes their contribution special, that is an applied-linguistics contribution, is in my view that they bring to the evaluation a readiness to generalize through model-making, as I have tried to illustrate in this abbreviated account of the study Tharu and I carried out in South India in 1990.
