The weekly column

Article 56, April 2001

A Macro materials Evaluation: for Better or for Worse

By Ann Riddell Akin & Meral Guceri

Abstract

The aim of this article is to describe a macromaterials evaluation study which was carried out at Bilkent University School of English Language. As many previous articles (Ellis 1997) have discussed the theoretical issues of evaluation, the focus of this paper is on the actual implementation of amaterials evaluation study in a higher education institute in Turkey. The methods and procedures followed are described briefly, as are the evaluation tools which were implemented in the process. Finally, the issues, which came to light during the evaluation, are examined and recommendations for carrying out similar evaluations are given.

Background

Bilkent University is an English medium university, which has traditional faculties such as engineering, social and administrative sciences and also vocational schools such as tourism and computer programming. The School of English Language (BUSEL) provides courses for students whose English needs to be brought up to the level required by the university. It also provides freshman courses in some of the faculties and vocational schools. The English Language Support Unit (ELSU) is a specific unit within BUSEL, which offers EAP/EOP courses to tourism, computer programming and bureau management students. Each course consists of sixteen weeks of study (one semester). Students receive either four hours or two hours of tuition per week depending on their faculty and year.

ELSU carried out a needs analysis to identify both departmental and student needs and, as a result, new course outlines were designed and implemented. EAP/EOP(English for Academic Purposes and English for Occupational Purposes) task-based syllabi were considered to be the most appropriate for vocational school students since students need to be prepared for their industrial training and future careers. It is often quite difficult to find published materials which fulfill specific course objectives and are of the appropriate level of difficulty. In addition, many course books may neither be of particular interest nor be culturally suitable for a particular group of students. In our situation, it was decided that in-house materials would be developed as they would be more suitable and could target our students’ needs. Materials were designed to last for two hours covering one lesson. The types of materials which were produced included business correspondence, product demonstrations, case studies, video lectures and job interviews.

However, most of the staff were new to the unit and, although some had previously designed EAP materials, others lacked task-based material design experience. In addition, new materials had to be produced quickly to meet deadlines for photocopying and for classroom use. The new course outlines also tended to have rather general objectives such as ‘business communication’ which were often difficult to interpret into materials. Thus, owing to these issues, to ensure that the materials written were of the desired quality, it was decided that all the course materials would be evaluated during their piloting stage.

When ELSU decided to embark on such an evaluation program, as Ellis (1997) states there was indeed a lack of published material describing practical experiences of macromaterials evaluation processes. By macro we mean examining materials as a whole for a complete course (Ellis 1997). As a result, using our own experience and ideas from priormaterials evaluations, (Dougill, 1997, Rea-Dickens & Germaine 1992) an evaluation operational plan was devised, and various data collection tools were designed, piloted and implemented. In addition, at the end of the programme, to obtain feedback and reaction to the evaluation process as a whole, a metaevaluation study was conducted and recommendations were recorded.

Aims of the Evaluation

The aims of thematerials evaluation were as follows:

• To identify how successful were the materials used in the classroom.

Firstly, we examined whether the materials were interesting and motivating for students. Secondly, whether students had enough input and practice and finally, whether there were enough different types of tasks in the material.

• To examine whether the materials fulfilled the prescribed course objectives.

We examined at the end of the lesson whether students were able to perform the required tasks successfully and whether teachers’ perceptions of material and course objectives were clear.

• To examine the extent to which materials permitted students to achieve the learning objectives.

We looked at whether the difficulty of the tasks was of the required level and observed how well students performed.

• To identify whether the designs of the materials were suitable for a tasked-based syllabus.

We examined the interactional patterns that were employed in the classroom when using the materials, to determine whether the materials were student-centred and whether the tasks were meaningful and relevant to student needs. In addition, we recorded how much lesson time was spent on teacher input and compared it to how much time students actually spent performing tasks.

Methods and Procedures

As Weir & Roberts (1994) suggest, it was agreed that, owing to the aims of the evaluation, a formative style or on-going evaluation using qualitative data would be suitable for evaluating the materials. It was important that immediate qualitative feedback was obtained regarding the success of our classroom materials. The data would be used to develop materials and to decide how they should be amended. They could then be revised accordingly during the semester to be used in subsequent courses. Post-semester retrospective feedback was also collated from teachers. This could be regarded as an element of summative evaluation. It was felt, however, that this feedback would add greater depth and perspective to the study.

The macromaterials evaluation was carried out during the whole of the academic year and involved the evaluation of approximately 300 pieces of material covering 10 different courses. An evaluation operational plan was designed at the beginning of each semester and implemented. Three members of staff were involved in coordinating the process and collating and analysing data. All 14 teachers in the unit gave feedback on a weekly basis. All teachers were also interviewed at the end of the semesters for retrospective evaluation. In a semester, feedback was also obtained from a total of 500 students. Approximately 40 students evaluated each key piece of material. The feedback was collated and the results of the study were used as a basis for deciding how to amend and adapt materials for their future use.

Although data was analysed on a weekly basis by the evaluation team during the semester, reports of the results were published and distributed to relevant parties every eight weeks. The report not only highlighted the changes that needed to be made to materials, but also identified individual teachers’ strengths and weaknesses regarding material design and production. This meant that resources within the unit could be allocated more efficiently as teachers became specialists at designing particular type of materials. For example, one teacher became particularly skilled at designing video material, another reading texts.

In order to ensure that the results of the evaluation were reliable, it was necessary use a variety of different data collection tools in order to triangulate the feedback. Firstly, different stakeholders, that is both teachers and students were involved in the process; Denzin (1978) refers to this as ‘data triangulation’. In addition, data was collected using a number of different tools so that a variety of feedback was obtained from students and teachers. This is referred to as ‘methological triangulation’ (ibid.) which provides a multiperspective view of the area under investigation. The following data collection methods were used:

• Teacher Questionnaires

In planning the evaluation teachers’ workloads were taken into consideration. As a result, all 14 teachers attached to the unit were asked to give feedback on only materials for two courses per semester. This means that teachers were involved in giving feedback on approximately four pieces of material per week. Teachers were asked to use their office hours to complete this activity. Thus over a semester a teacher gave feedback on 60 different pieces of material.

Teacher questionnaires were piloted during the first week of semester and were amended where necessary. The questionnaire was designed in the form of a checklist so that it could be completed quickly. In addition, space was available for comments and teachers were encouraged to give suggestions as to how to improve materials. It was estimated that the form could be completed thoroughly in less than 15 minutes which means that teachers could complete their evaluation work in less than an hour per week (See Appendix I for a sample questionnaire).

• Student Questionnaires

The questionnaires were designed to be used after key materials had been used in lessons and they were to be administered to students at the end of the lesson. "Key materials" were identified as materials which covered areas that would be tested during the course, either during the mid-term or final exams, or in the form of compulsory assignments. Approximately 40 students were targeted to give feedback. For practical reasons all students were not involved in evaluating all materials as questionnaire completion and distribution is not only a time-consuming process but it may disrupt lessons. Furthermore, students may have become rather complacent about the questionnaire completion if it became a habitual activity.

Questionnaires were piloted and, as a result, it was decided that a quantitative-style questionnaire with closed questions would be used, as students tended not to answer open-ended questions. Moreover, it would be easier to analyse the data. (See Appendix II for a sample questionnaire).

• Teacher Interviews

All teachers involved were interviewed at the end of both semesters to give summative feedback and to follow up issues raised on feedback forms. Interviewees were given, one week prior to the interview a list of questions and issues to be discussed at the interview. Teachers were then prepared for the interview and time could be used efficiently. Furthermore, it ensured that data gathered from different teachers was consistent. Interviews lasted for approximately one hour and were held during teachers’ office hours. Teachers were encouraged to share their experiences openly and honestly. The interviews were held by one of the members of the evaluation team rather than by the curriculum department and they were private and confidential.

• Observations

Owing to time constraints, only the implementation of key materials was observed by one of the evaluation coordinators. Although it was recognised that students tend to behave differently when they are observed, the observer visited the same class a few times prior to observing the materials so that the class was accustomed to being observed. The focus was mainly on student interaction and motivation. Findings were recorded to validate and support data given by students and teachers.

Major Areas under Examination

Both teachers and students considered the following areas when evaluating the materials:

• Timing: As all the time allowances given to the materials were estimated, both teachers and students were asked to comment on their accuracy.
• Suitability of Teachers’ Notes: Teachers were asked to give feedback regarding their clarity, suitability, and whether they were easy to follow.
• The Layout of Materials: Teachers were asked to remark on the quality of the print and the layout of the materials and examine whether they were visually appealing to students.
• Motivation/Interest/Usefulness: Both teachers and students were asked to analyse how interesting, useful and motivating the materials had been.
• Variety of tasks and Activities: Both parties were asked to consider whether there was enough variety of tasks within the materials and whether they were challenging and meaningful.
• Materials in relation to Course Objectives/Learning: Teachers were asked to comment on whether they thought that the materials had attained their objectives and to give reasons for their comments.

Metaevaluation

The evaluation process itself was evaluated at the end of the second semester in order to ensure that future evaluation studies would take into consideration any difficulties anticipated or recommendations made by staff. Teachers were given a questionnaire to comment on as to whether they considered that the evaluation was a useful process and whether they had enough time to complete the weekly feedback forms in their office hours. Finally, teachers were asked if they were satisfied with the number of materials they evaluated.

Outcomes of the Evaluation

It was found that the general appearance of the materials was not satisfactory. It was realised that more care needed to be taken regarding neatness, presentation and layout which would make the materials more attractive for students. As materials were often produced using different software this tended to make the materials look visually less professional. In addition, it was decided that more visuals needed to be added to the materials in order to make them more appealing and stimulating for students.

Materials tended to be rather lecture-based and too theoretical in style and had very few tasks and activities for students to perform. Thus many materials were not conducive to task-based syllabi. It became apparent that more meaningful and challenging components would encourage students to become more involved in the materials. Thus, during the materials revision there was a shift in emphasis from lecture mode teacher driven lessons to more student-centred task-based teaching and learning.

Greater thought needed to be given to text selection as it was noted that some texts were not relevant to student needs and students lacked the schemata to tackle them effectively. More emphasis was placed on students’ interests and motivation when adaptations and modifications were made to the materials. Some materials also required editing as they took much longer than had been predicted.

Finally, it was realized that there was a lack of emphasis on language and vocabulary development in the material design. Although the syllabus is tasked-based data revealed that students were often not equipped with the language required to complete tasks.

The quality of the materials significantly improved over the year, as these considerations were included in the revision process. The points mentioned above were cascaded to teachers and they were incorporated in future materials.

Issues and Recommendations

The following issues became apparent during the evaluation process:

Feedback

Initially, teachers’ feedback was not particularly detailed and rather vague, which made it difficult to analyse the data and was time-consuming as teachers had to be approached individually to obtain clarity concerning their comments and recommendations. Awareness had to be raised regarding this issue in order to improve the quality of the feedback collated.

For future evaluation studies evaluators are highly recommended to hold a workshop including all staff involved in the evaluation so as to ensure that everyone is aware of the importance of their contribution to the process and that they provide the required level of feedback.

Time

Even though teachers were asked to give feedback regarding a limited number of pieces of materials, many teachers still commented that with their teaching load and material production commitments, they had difficulty in allocating enough time to the questionnaire completion.

Thus it is essential when planning timetables that enough time is allotted to teachers for the evaluation process. If not, this may have an impact on teacher attitudes towards the programme (Weir & Roberts 1994).

Learners

As this was the first time our students were involved in analysing materials and their learning process, it was felt that the reliability of their data needed to be questioned. If students are not accustommed to giving feedback then the data gathered may not be completely valid and reliable.

It was recognised that during an evaluation process students need to be provided with some training in order to raise their awareness of the importance of their contribution to the programme.

Questionnaire Administration

Although teachers were instructed to distribute questionnaires to students after materials had been completed, at the end of the lesson, they were sometimes administered during following lessons owing to time constraints. This in turn may affect the reliability of data obtained as students may have difficulty in remembering previous lessons or some students may be absent and not complete forms.

Thus it is important that teachers realise that immediate feedback needs to be obtained as it tends to be more accurate and because the student population may change in subsequent lessons.

Management

An enormous amount of time is required to successfully carry out a materials evaluation programme. This time needs to be planned carefully, otherwise time and resources may be wasted.

Hence it is imperative that the person in charge has enough time, the relevant skills and knowledge to be able to conduct the evaluation study. An independent party, who may have a more objective view of the feedback, could be given the responsibility. In addition, the evaluation study needs to be part of the departmental operational plan.

Co-operation and Collaboration

A large-scalematerials evaluation does require so much time, effort and commitment that it will only be successful if all parties maintain a positive attitude towards it.

As a result, it is imperative that regular feedback sessions are given to all those who have contributed to the study so that everyone can see that their efforts have been not been wasted.

Conclusion

The materials evaluation study, which took place during one academic year, proved to be extremely useful. Ongoing materials adaptations and production continued, taking the evaluation feedback into consideration. In addition, there was a knock-on effect on new materials produced, as teachers were aware of the drawbacks of the materials evaluated. Teachers’ own critical awareness regarding material analysis and production was also raised which led to teacher self-development.

Although a macro levelmaterials evaluation is difficult and time consuming, it is worthwhile as not only does it ensure the production of quality materials but it also contributes towards team building and the personal development of all parties involved.

APPENDIX I

Materials’ Evaluation Feedback Form (For Teachers)

Teacher’s Name ……………………….. Course……………………..

Week……………….. Name of the Material……………………………

Could you please answer the questions by circling the appropriate responses. Would you please kindly add comments where necessary giving further details. Thank you for your time and co-operation in completing this feedback form.

* If there is not enough room for your comments please continue over the page

 

---

 

APPENDIX II

Studentmaterials Evaluation Questionnaire

Dear Students,

The English course you are taking is quite new and so we are keen to hear what you think of the course and the materials.

This questionnaire is designed to get your opinion on the new materials to help us when we consider whether and how to develop the materials and the course further.

We thank you for the time and effort that you give us to complete this questionnaire. Your responses will help the students that follow this course in the future.

***********************************************************************************

Please circle your responses.

A) Timing

1. I had enough time to complete the work

2.  I was given too much time to complete the work.

1	2	3	4	5	6
Strongly Agree					Strongly Disagree

Comments (if any) ……………………………………………………………………………

B) Usefulness

1. The materials will be useful for my studies.

2. The material will be useful for my work life.

Comments (if any)…………………………………………………………………………

C) Interest

1. The material was interesting.

 2. The material had a variety of tasks and activities.

Comments (if any)…………………………………………………………………………….

D) Difficulty

1. I found the language of the material difficult to understand

2. I found the tasks and activities difficult to do.

3. The material made me do new things.

Comments (if any)…………………………………………………………………………….

E) Layout

1. The material was well laid out.

2. The type size of the material made it easy to read.

Comments (if any)…………………………………………………………………………….

F) Learning

1. I improved my English as a result of using these materials.

1	2	3	4	5	6
Strongly Agree					Strongly Disagree

 

2. I learned a lot from using these materials

Comments (if any)…………………………………………………………………………….

 Bibliography

Denzin N.K.(1978) The Logic of Naturalistic Enquiry, Sociological methods. A sourcebook. New York McGraw-Hill.

Dougill J. (1987) ‘ Not so Obvious" in Sheldon, L. (ed.) ELT Textbooks & Materials: Problems in Evaluation and Development, Oxford, Modern English Publications pp.29-37.

Ellis, R (1997) ‘The Empirical Evaluation of Language Teachingmaterials ELT Journal, Vol.51. No.1.pp.36-42.

Fitz-Gibbon, C.T, & Morris L.L.(1987) How to Design A Program Evaluation, Newbury Park CA Sage.

Rea-Dickens,P. & Germaine. K, (1992) Evaluation. Oxford Oxford University Press.

Weir C.J. & Roberts J. (1994) Evaluation in ELT, Oxford, Blackwell Publishers.

About the Authors

Ann Riddell Akin & Meral Guceri

Meral Guceri is at present an Assistant Director, Head of Teacher training, at Baskent University, Ankara. She was head of a teaching unit at Bilkent University, Ankara for many years. She has a Masters from Aston University. She is interested in teacher development and teacher training as well as curriculum issues.  

Ann Riddell Akin is at present an English language teacher at the UAE University in Al Ain. She worked as a teacher at Bilkent University, Ankara for five years. She is interested in materials development and curriculum issues.

Questions or comments about this week's article? Why not post them on our Discussion Forum