Our pre-seminar task was to create agreed principles in groups and apply them to evaluate the course book “Face2Face” produced by the Cambridge Press. It was a surprise for me that our group should be made up of 4 teachers working in 4 contexts, one native English speaker teacher and 3 non-native English speaker teachers from 3 different countries. These differences led to distinct perspectives on carrying out the task and intrinsic concerns for the local criteria although there was no assigned context. Nevertheless, communication and negotiation contributed to us ultimately reaching agreements on the procedures and principles. However, the practice of evaluating the course book with our principles, which I think deserves more attention and discussion, was not implemented by us together but assigned to me at our last meeting. Therefore, in this post, I will present how we created our own evaluation checklist as well as my personal understanding of the evaluation process.

Firstly, an approach is indispensable for the consistency of the whole evaluation process. McGrath (2002) presents a “cyclical” approach to materials evaluation process, in which there are three stages of materials evaluation: pre-use, in-use and post use. As none of us had the experience of using this book, we decided on pre-use evaluation, which refers to predicting the potential value of this course book based on a set of criteria. Since it is the only book we need evaluate, we should not base the evaluation on the quick flick through the whole and the first impression but develop a checklist to make the criteria “explicit” “systematic” “convenient” and “cost-effective” for the use in the following stages (McGrath 2002).

Regarding the design of the evaluation checklist, we adopted the 7 procedures from Tomlinson (2013). The first 3 steps appear to have been done in the previous seminar and therefore what we did was to compile the criteria generated by 3 groups and categorize them into specific aspects of the material use. During the process, we deleted some sub-categories and also added some to have the checklist organized in a logic and systematic way. Meanwhile, we reworded some criteria to avoid ambiguity and potentially distinct interpretations of some terminologies.  A rating scale rather than yes/no format was agreed on because value judgement is mostly made about the quality and should allow individual distinctions, according to McGrath (2013).

Littlejohn (2011) proposes materials analysis which fits a profile of the target learning context for the material suggested by Tomlinson (2013). This term suggests the written profile or the analysis should be descriptive rather than evaluative. The significance of this step lies in the understanding of the assumptions and beliefs beneath the surface and the anticipated effects of using the material (McGrath 2002). As there is no assigned context, we assumed the context by extracting some information of the target audience in the course book, such as possible age, the purpose of learning, level, etc. Based on the material analysis (see the slide below), some local criteria were designed and added to the checklist.

Most of the time was spent negotiating the procedures and the design of the criteria. Consequently, there was no time for us together to evaluate the sample unit. The 3 sections of the unit were allocated to 3 members to be separately evaluated. The slide below shows the results of the evaluation of each section.

What confused me here is that why the whole unit should be separately evaluated. The three sections are supposed to be consistently designed and each section functions differently. It could be inevitable that partial or even biased judgement is delivered unless the unit is evaluated as a whole. What’s more, conducting evaluation would not be to present the average score but to provide evidence for the score. Therefore, I decided to explore the whole unit and then give judgement based on the criteria in the checklist, which raised another problem. As it can be seen above, the first category is allocated to learning principles, which seem to be very conceptual and intangible. For example, to answer the first question and rate it, I need to ask myself another question what stimulates learning? Without a doubt, following the sequence of the checklist couldn’t be a practical and neutral approach to evaluation.

In the circumstances, I skipped the first category and started to examine others. I envisaged learning principles as being embodied in these more specific and transparent criteria involving language use, instructions, task design, visuals etc. Thus, I made a diagram (see the slide below) which shows the consistent link among these criteria and assessed the relevant elements in this sample unit.

The first round of evaluation started from 2.1 and ended with 6.2. The topic in this unit is media, which is much relevant to learner’ life. The language is natural when formal language in quality press and informal language in a tabloid, as well as daily chat, are embedded in the material. The visuals, including pictures and highlighted keys words, are appealing and echo the content. As a result, learners probably show interest in this topic and see the benefits of learning the content-specific language, which could enhance their motivation and encourage autonomous learning. Likewise, in each section, learning objectives are clearly set down and they are in line with each other. A variety of tasks, ranging from controlled practices to more productive skills practice, involve 4 skills and the signals of their sequences are clear with the page number. The task instructions are short and comprehensible. A logic sequence of various tasks and clear instructions could provide scaffolding for learners and allow for differentiation. Regarding teachers’ needs, the systematic design of the material with tasks as well as clear instructions also helps novice teachers follow and allows experienced teachers to select and achieve localization with adaptation and supplementation. All in all, I think these criteria are effectively met.

In a similar way, I rated the other criteria (see the diagram below). In this process, I have developed a deeper understanding of the outline and internal links to the evaluation criteria. What I have also learnt is that “experience is not a substitute for training in the evaluation of course books” (McGrath 2013: 124). Evaluation of materials requires certain knowledge or training and also helps develop teachers’ professional knowledge and judgement, which should be incorporated in teacher education (Sampson 2009; Law 1995 cited in McGrath 2013: 125).

References:

Tomlinson, B. (2012) “Materials Development for Language Learning and Teaching”, Language Teaching, 45 (02) pp. 143-179.