TE714 Robyn Moallemi's blog

Evaluating language teaching and learning materials is really important in successful language teaching because it encourages us to be a reflective practitioner (Schon 1983) and actively reflect on what we need/want from materials, how they are used in practice and to what effect. In essence, if we use materials in our teaching i.e. do not use the dogme approach, the evaluation of materials is vital for us to move-forward, to constantly respond to our learners needs and wants, our micro and macro contexts (McGrath 2013, p. 53) and in doing so, maintain our learners motivation and interest (Norton 2011). McGrath (2002) developed this three-phase evaluation cycle idea and referred to them as pre-use, in-use and post-use. The stress here is that the evaluation is a conscious and constant act that takes place throughout the use of material and not just in the selection stage when identifying materials for a course. In this post as in the last, the word material will refer to coursebooks.

This week’s readings (Johnson 2008; Littlejohn 2011; Tomlinson 2012; McGrath 2013) on material evaluation also referred to materials analysis and I would like to attempt to define and clarify both before I continue, as there are significant differences that will infer a rational for my evaluation framework:

Materials analysis examines exactly what is in the coursebook and its purpose is “descriptive analytical” (McGrath 2013, p. 53), investigating exactly “what they contain, what they ask learners to do and what they say they are trying to achieve” (Tomlinson 2012, p.148). Materials analysis gives the teacher, hopefully in the pre-use selection process, “a thorough means of testing the claims made by publishers and textbook writers” (McGrath 2013, p. 53) but this could equally happen in its post-use analysis.

Materials evaluation on the other hand is linked to the intended teaching and learning context of the coursebook and is “inevitably subjective and focuses on the users of materials” (Tomlinson 2012, p. 148). Materials evaluation can take place pre-use and be “predictive” where the teacher predicts the success of the coursebook in a particular context with specific learners or post-use and be “retrospective”, where the teacher attempts to assess the actual success of the coursebook, separating it from other variables, on his/her learners. The third phase of McGrath’s cycle in-use informs teaching practices and use of the coursebook as it is used, I.e. reflection-in-action (Schon 1983).

The pre-seminar task was to work in small groups to create an evaluation framework of one or two pages of chapter 4 from the ELT coursebook. Unfortunately I wasn’t able to work with anyone on the task and absence from the seminar may result in my framework and interpretation of evaluation materials being a lot more subjective than intended.

That being said, from what I have read and understood, evaluations are highly subjective anyway and “many of the lists of evaluation criteria (…) are specific to a context of learning and cannot be transferred to other contexts without considerable modification” (Tomlinson 2012, p.147). In this regard, the collaborative aspect of the pre-seminar task would have been extremely useful in encouraging us to share and reflect on how, what and why we evaluate materials but our responses would have arguably differed in light of our different contexts, especially taking into account the fact that I am the only MFL teacher out of a class of English language teachers. Likewise, our experience of teaching and of evaluating language materials would have differed significantly (Johnson et al 2008) and this too could have been exposed in our framework. According to this study by Johnson et al “the more teaching experience an evaluator has (…) the more able he or she is to view a textbook with detachment, and take account of other users’ needs as well as his or her own” (p.161). I did wonder, when set the task, if my lack of English language teaching experience would have hindered or severely contrasted with the other group members’ contributions. Are language materials in a target language removed setting such as mine, used by non-native language teachers such as me, viewed and used by native language teachers, in the same manner as materials in a target language embedded (Graves 2008) setting? From discussions in class it seems to depend on the context.

The teacher who evaluates the materials and how they evaluate them must therefore reflect the specific context, experience, wants and needs (Masuhara 2011). If the same teacher is evaluating the materials as designing the evaluation framework then these factors will ultimately affect the design and make the framework itself subjective. Having previously evaluated the suitability of a coursebook for my current language course and having only used my experience and contextual knowledge as a basis for my un-written criteria, I can fully appreciate teacher and contextual influences.

In terms of what an evaluation framework should include, its format and process, there were multiple concepts from this week’s readings, many of which resonated with me and are hopefully inherent in the framework that I created. They include:

• A 2-stage approach to materials selection (McGrath 2013, p. 54):
• 1^st stage is an “impressionistic overview” (Cunningworth 1995)
• 2^nd stage puts the coursebook under closer scrutiny with a checklist and criteria
• Questions are put in categories
• Split into categories. Categories could include: usability, generalisability, adaptability and flexibility (McDonough and Shaw 2003, p.61)
• A limited number of yes/no questions
• Preference on a rating scale for “judgements of quality” (McGrath 2013, p. 57)
• Allows space for comments and to give reasons for answers
• One subject is addressed in each question
• Questions are transparent
• Sub-totalling scores “for indicating relative strengths and weaknesses” and totalling responses for “easy comparison of different sets of materials” (McGrath 2013, p. 57)
• Evaluated by more than one: “the materials under consideration should be assessed by three experienced teachers” (Daourd and Celce Murcia 1979 in McGrath 2013, p. 58)

Tomlinson (2003b, p. 28) advised that evaluators should mind-map “a list of principled beliefs that they hold about how languages are most effectively acquired and then convert these beliefs into criteria for evaluating materials”. Reading this made me think back to last week and the set of principles for materials design that we collaboratively decided upon. The link seemed obvious; use a set of key principles that I agree with/produced, in order to evaluate how successful the material could be. I therefore applied the concepts above and arranged my principles under specific categories, each principle with a rating scale and each category with a comments box to allow for further reflection. Each category had a sub-total score box and a total score box at the end of the evaluation. This, as mentioned above, allowed for “indicating relative strengths and weaknesses” (McGrath ibid) of each category and could help to compartmentalise the evaluation and make comparisons with the other teacher evaluator. The paragraph at the end of the evaluation signals the subjectivity of the evaluation and encourage at least three teachers ((Daourd and Celce Murcia ibid) to evaluate the same material.

In attempting to create a universal evaluation framework, I have read key readings whose suggestions are cemented in research and applied some of these recommendations. However, in creating the framework and especially through the implementation of the principles of materials design as suggested by Tomlinson (2003b), I have designed a framework that can’t help but reflect my principles, beliefs, wants and needs (Masuhara 2011). One aspect of the evaluation that does this is the lack of evaluation on the course book’s instructional language. Although I do share the same belief as Skutnabb-Kangas that “when the child’s mother tongue is valued in the educational setting, it leads to low anxiety, high motivation, and high self-confidence, three factors which are closely related to successful programs” (in Siegel 2003, p.197) and would further apply this to adults in certain contexts, the language of instruction is a contentious issue and one that I did overlook.

Irrespective of this subjectivity, my intention was to create an objective framework that had universal potential. My success ultimately resides in others’ subjective interpretations, situated in their experience and context, which brings us back to one of the first points made about materials evaluations being “inevitably subjective” (Tomlinson 2012, p. 148).

This is my evaluation framework: Evaluation Framework-1jd1pn0

References:

Graves, K (2008) The language curriculum: A social contextual perspective in Language Teaching 41 (2): pp.147 -181

Johnson, K., et al. (2008) A step forward: investigating expertise in materials evaluation. ELT Journal 62 (2): pp.157-163.

Littlejohn, A. (2011) The analysis of language teaching materials: inside the Trojan Horse. In: Tomlinson, B. (ed). Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press. pp.179-211.

McGrath, I. (2013) Teaching Materials and the Roles of EFL/ESL Teachers: Practice and Theory. London: Bloomsbury.

Schön, D. (1983). The Reflective Practitioner: How professionals think in action. London: Temple Smith

Siegel, J. (2003) Social Context in Doughty, C, J., and Long, M. (2003) The Handbook of Second Language Acquisition, Blackwell Publishing: Oxford (pp 178 – 197)

Tomlinson, B. (2012) Materials development for language learning and teaching. Language Teaching 45 (02): pp. 143-179. doi: doi:10.1017/S0261444811000528.