Category Archives: Uncategorized

Materials Evaluation

On the surface, materials evaluation is a straightforward exercise in determining the efficacy of materials to be used in the teaching context but scratch a little deeper and it is a quagmire of subjectivity and beliefs that appear to be impossible to get away from. We also come back to the issue of relationships and power between stakeholders. Tomlinson & Masuhara (2018, p. 52) noted that “the prime users of commercially produced materials are learners their prime buyers are administrators”. Administrators in this sense could be teachers but also director of studies, senior teachers, or ministerial appointees, among others. We can also add teachers as users of materials, especially in the classroom context. Tomlinson (ibid) found that “in a few institutions the classroom teachers selected the coursebooks and that in no institutions were the textbooks selected by learners”.

This layer between materials and learners/teachers places responsibility on those choosing materials to make the optimal choice for the intended users. It also brings how we evaluate materials into closer scrutiny and how our beliefs and understanding of language learning affect the tools we use for evaluation. The very act of evaluating materials can bring new insights and understanding about what teachers want from materials (Mishan & Timmis, 2015, p. 72) and highlight our beliefs regarding what aspects of materials of materials are important.

During the task this week evaluating materials and Tomlinson and Masuhara’s framework (Tomlinson & Masuhara, 2018, pp. 69-71) was our preferred catalyst. There were strong requests for criteria focusing on authenticity of listening materials, which brought up differing opinions on the nativeness principle versus intelligibility principle (Levis, 2005, p. 370), and English as a lingua franca. One side argued that authenticity is important, but this led to attempts to define authenticity and “it is impossible to engage in a meaningful debate over the pros and cons of authenticity until we agree on what” authenticity is (Gilmore, 2007, p. 98). Non-native speakers with accented English are equally as authentic as native speakers with various regional accents and evaluating materials purely on accent content without insight into the breadth of accents represented would leave two different types of audio equally assessed if we were not to look into more detail regarding the origin of these accents. We then dig further into issues such as the authenticity of studio-recorded audio (or any material used in the classroom for that matter) versus comprehensibility, which raises semantic problems defining comprehensibility and whether graded language is needed: Loschky’s findings suggest modified input does not “facilitate comprehension relative to non-modification of input” (1994, p. 315). These issues were almost impossible to navigate in our short two-hour workshop and with the need to press on, we found that separating materials analysis and materials evaluation was equally challenging.

The literature distinguishes between evaluation and analysis (Tomlinson & Masuhara, 2018, p. 56), and highlights some pitfalls of mixing them together as well as its effect on weighting, which was another headache for our team. Analysis of materials are typified by factual questions (Mishan & Timmis, 2015, p. 88) and generate definite answers from questions such as “how many chapters does it contain?”, or “does it provide speaking opportunities?”. This type of question was posited as effective by a member of our team because, from experience, it was simple to use but, as noted by Tomlinson and Musahara (2018, p. 55), these types of questions are open to bias from the author or can be interpreted from different perspectives which will lead evaluators to different scores and therefore results. In our example, it is not clear what can be deemed a good number of chapters because it is not possible to evaluate materials based on number of chapters. From our evaluation tool, you can see in the content-specific criteria that we have put a question regarding task-based learning into our evaluation. It suggests we are looking for task-based learning materials, and that task-based learning is preferable to other approaches, although that was not one of the underlying beliefs of our group. Despite this being presented to us in our reading, we added these questions into our evaluation tool and this promptly led to disagreement about how binary options can be represented on our cline of 1 to 5. This naturally led to how to weight different criteria, because guidance notes for teachers is not equal to cognitive challenge of materials.

Evaluation Tool

The evaluation tool we used to evaluate English Unlimited B1+

Ultimately, our evaluation tool did not weight the criteria and the Teacher’s book is weighted less because we could not think of more criteria. Universal criteria is not the most important category but is weighted that way as a result of not giving time to think about and discuss weighting. When we used the criteria for evaluating, we came across problems that were impossible to evaluate but time pressure, and our own experiences of teaching left us at an impasse, so we did not rewrite or delete the question.

Time and complexity is an area that has been highlighted (Tomlinson & Masuhara, 2018, p. 61; Mishan & Timmis, 2015, p. 97) that affects how teachers use evaluation tools. Teachers are busy teaching and would not be able to go through a protracted pre-evaluation process in addition to evaluating the materials. Materials selected for an institution will be used by teachers of all experiences and novice teachers will be disadvantaged with evaluation tools that can only be understood by experienced teachers, and thus return unreliable data. This is without considering evaluation from learners (ibid, p. 98), an area that we had not considered. The reality of the situations in which evaluation tools will be used compared to the research are far apart, and this is most notable in how the different responsibilities overlap.

Theresa Clementson, one of the authors and editor of the course book we evaluated (Rea, et al., 2013), advised us to be careful about overlap between what teachers should do in the classroom and what materials are designed to do and this issue came up with evaluating materials based on long-term learning goals, and with engagement and motivation (ibid, p. 91). I would argue that long-term learning goals are not within the remit of materials creators and lies firmly in the learners’ own hands along with their teachers. There are clear limits to what materials can realistically deliver, and this is further muddled by the decision-making processes behind how the course book was put together. It was surprising to discover that the editor does not have authority on all aspects of the book and the publisher, and its marketing team, wield considerable influence on visual media and layout. This information shed light on a question that we had grappled with throughout the evaluation task regarding the discontinuity of the visuals used within one unit, with a mix of cartoons and photographs that were integral in some activities but were totally irrelevant in others.

We noted that these issues were probably borne from spending too little time preparing our framework and exploring our beliefs before creating criteria for evaluation, and that with more time we could have produced a more objective evaluation tool that had itself been evaluated. However, our situation mirrored those in many teaching centres and schools around the world where materials need to be evaluated and ready for new classes.


Gilmore, A., 2007. Authentic materials and authenticity in foreign language learning. Language Teaching, 40(2), pp. 97-118.

Levis, J. M., 2005. Changing Contexts and Shifting Paradigms in Pronunciation Teaching. TESOL Quarterly, 39(3), pp. 369-377.

Loshchky, L., 1994. Comprehensible Input and Second Language Acquisition: What is teh Relationship?. Studies in Second Language Acquisition, 16(3), pp. 303-323.

Mishan, F. & Timmis, I., 2015. Mateials Development for TESOL. Epub ed. Edinburgh: Edinburgh University Press.

Rea, D. et al., 2013. English Unlimited B1+. s.l.:Cambridge University Press.

Tomlinson, B. & Masuhara, H., 2018. The Complete Guide to the Theory and Practice of Materials Development for Language Learning. New Jersey: Hoboken.



Examining our beliefs in the function and purpose of materials

This week’s seminar involved a ranking exercise, a higher order cognitive task in Bloom’s taxonomy. We were required to complete some sentence stems about materials and rank them. It proved to be far more challenging than I, and my group imagined and we found it impossible to rank the sentences about materials.

Prioritising attributes of materials

A flat hierarchy of importance

The sentences exposed our underlying beliefs about the function and purpose of materials. It also further exposed how unprincipled our beliefs were and, similar to beliefs in teaching, our approach to creating materials did not necessarily match our stated beliefs.

I think this loose understanding of our own beliefs in materials may lead to problems later on when evaluating materials. We have baggage regarding the the effectiveness of published materials, but cannot clearly articulate which of our beliefs that the evaluated materials impinge upon.


Course books, materials, and our relationships with them

During our seminar, 11th February 2021, introducing the module, we discussed a few questions regarding materials and our relationship with them. Some varied and interesting opinions were shared in the seminar and on our discussion board. I would like to talk about defining materials, the paradox of course books, the economics of creating materials, and our relationship with materials as learners and teachers.

One definition of materials that stood out was that they could be anything “designed to assist learners achieving desired learning outcomes” (Colburn, 2021) and this aligned with Tomlinson and Masuhara’s definition (2018, p. 2) that “anything that can be used by learners to facilitate their learning of the language”. They go on to list some examples both tangible and intangible. From a teacher’s perspective, this definition could present challenges and opportunities in the classroom.

Lesson preparation necessarily requires forethought about what to use and possible outcomes of activities and language production. To share this lesson and describe the materials for the lesson, encompassing all aspects of materials would be difficult if not impossible to communicate to another teacher. However, the flexibility and options available to teachers in the classroom are endless when anything can be used as teaching materials. Evaluating its effectiveness is another challenge, though. Giving materials some semblance of order would be helpful for teachers to choose materials for their lessons and describe them to others.

Gray (2016, p. 95) offers a more limited but helpful categorisation of materials: published materials, including print and digital, and audio-visual; authentic materials, those which were not originally designed for language learning; and teacher-made materials, which could be anything the teacher creates to supplement or replace other materials. I do not see how we can easily fit tasks and activities into these categories. Tomlinson and Masuhara (2018, p. 2) would classify a discussion as material but it does not satisfy the criteria for any of Gray’s definitions. In practical terms, I think that having materials that fall outside of these categories is not an issue that negates the benefits of making the discussion of materials more manageable for teachers.

Despite the plethora of materials and the wide choice, the course book remains a staple in the majority of classrooms (Tomlinson and Masuhara (2018, p. 25) referenced a British Council survey in 2008 that I cannot find). Even though many teachers “found them uninteresting and not relevant” (ibid, p26), we persist and supplement and adapt them rather than replace them completely. In a school in Norway, the course book has been removed and this loss was lamented (Standal, 2021) citing the extra work required to create lesson materials that course books, traditionally, provided. This was echoed by another respondent citing the economics and role of teachers, “[c]reating materials is so time consuming and we don’t get paid to do it”. There is a paradox that published materials are considered to not be very good, but we want to use them, and the thought of no course book is worse than having one. This is possibly the reality of teaching butting against an ideal teaching environment: with all the time in the world, we would create bespoke materials for every learner and lesson.

That leads us to an issue at the heart of materials development and second language teaching/learning: the economics of creating materials. My experience of creating materials is an expectation that all teachers can create quality materials as part of non-teaching-day work. It either speaks of the high esteem in which teachers are held or the underappreciation of the work involved in creating materials. That is purely in the realm of teacher-made materials, but this attitude appears to permeate published materials, too. The industry has switched from paying a fee and royalty payments to a fee with ever reducing royalty payments, and to offering a fee only (Zemach, 2018). The knock-on effect of this is the reduction of materials creation to a side job and control of content moving from the materials creator(s) to marketing managers focusing on global/international editions that do not differentiate for regions (ibid). Zemach alluded to piracy and the increasing number of free extras offered on top of the course books sold being a driving force in pushing publishers to offer less and less money to professionals who would write and create materials. That Zemach can receive more royalties from selling self-published materials online at less than $1 than from course books sold at retail for more than $30 highlights the absurdity of the situation. These high prices increase the barrier to language learning (at institutions) driving learners to seek cheaper and/or free materials.

The learner has been very much left out of this discussion thus far, but their language learning is the focus of teachers and materials producers. They form an important part of the raison d’etre of teachers and materials. I have spent most of my teaching career in the private sector where learners have a closer relationship with the teacher than materials and are afforded more control over materials. This can take the shape of how much a course book is used to the type of tasks and activities used. I can imagine on a distance learning course that the relationship might be different with materials being closer to learners. A reduction of the power and influence of learners would be the state-education classroom and I have seen in language learning classrooms in Japan how materials dominate. Teachers work through materials given to them and learners have no input. However, the recent onset of online learning has upended some of these relationships. My nephew has been castigated for missing live online lessons at school and is content to work through the tasks set outside of that. Learners can interact directly with materials at home whether they be published or authentic and can directly influence those through social media, gaming, and other forms of communication.

The move of online education to the front and centre of our lives offers exciting opportunities and promises paradigm shifts. I am very excited to see how the democratisation of materials creation and the prominence of the learner’s agency will change what we consider materials and how we create it.


Colburn, L., 2021. TE714 – Week 1 – Discussion Board Post, Brighton: University of Brighton.

Gray, J., 2016. ELT Materials: Claims, critiques and controversies. In: G. Hall, ed. The Routledge Handbook of English Language Teaching. s.l.:Routledge, pp. 95-108.

Standal, I. U., 2021. What type of materials do I use?, Brighton: University of Brighton.

Tomlinson, B. & Masuhara, H., 2018. The Complete Guide to the Theory and Practice of Materials Development for Language Learning. New Jersey: Hoboken.

Zemach, D., 2018. Sausage and the law: how textbooks are made. [Online]
Available at:
[Accessed 10 February 2021].