1

Materials Evaluation | The Framework of Evaluation

What is being evaluated? When and how the evaluation is carried out? Who is going to evaluate? What kind of evaluation criteria will be used?

The materials evaluation seminar (week3) was a good opportunity to ‘act as if a material evaluator’. Like almost all Japanese teachers do, I used a coursebook that is approved by the Japanese Ministry of Education and published by a Japanese textbook publisher when I taught English in Japan. Generally speaking, the existence of coursebook is ‘common-sense understanding’ in Japanese educational values.

I depened on the coursebook, followed a syllabus on it, and conducted the exams based on the content of the coursebook. It was crucial to teach 40 students in a class at the same time equally. As a result, it seemed that the aim in teaching English was to complete teaching the coursebook, which sounded unproductive cycle. However, by reflecting the materials and its use in teaching, it can be productive. Therefore, considering my own framework to evaluate the materials including the coursebook and teacher-made handouts is beneficial for teacher development.

Before I talk about my own framework, I would like to look at some questions:

Materials: what is being evaluated?

 

In terms of materials, they can be anything that is used in classroom, and evaluated. As the contents of materials, illustrations, gender stereotypes and cultural aspects can be subjects for evaluation (Mishan and Timmis, 2015). In the beginning of the school year, the first things that caught my attention were visibility of a layout and attractiveness of illustrations. The coursebook I used in a Japanese secondary school had Anime illustrations, which I assumed that the publisher wanted to visually attract students to their coursebook.

When and how is the evaluation carried out?

Materials evaluations can be carried out at:

• Pre-use
• Whilst-use
• Post-use stages

During the each stage, several evaluation instruments and observation and trialing methods can be used. To evaluate materials, systematic approach is proposed by some researchers (Mishan and Timmis, 2015; Tomlinsom, 2012), such as checklist or making criteria. Particularly, Tomlinsom and Masuhara (2004, cited in Tomlinsom, 2012: 147) proposed questions for making evaluating criteria.

a) Is each question an evaluation question?

b) Does each question only as one question?

c) Is each question answerable?

d) Is each question free of dogma?

e) Is each question reliable in the sense that other evaluators would interpret it in the same way?

In the seminar, one of groups talked about how they came up with their criteria for materials evaluation in their presentation. They exploited Tomlinsom’s questions and combined them with their briefs to make their own evaluation criteria which interested me as I considered my own framework.

Who is going to evaluate?

Evaluators can be publishers, teachers or learners. In terms of teachers as evaluators, recording the materials use is helpful for its evaluation. Teachers can keep records how to used the materials and how they worked, which I personally believe that recording is also vital for teacher development. Even though all the recorded materials use cannot go well with all students in the same way, we can apply them in a different way to a different situation.

Evaluation criteria by me

My evaluation framework stems from 5 principles for the materials design I mentioned last post.

• Contextualised: Are the materials culturally relevance?

I personally believe that Japanese students need to be able to express themselves in English with Japanese mind. Thus, contextualised materials would be beneficial.

• Teachability: Are the materials teachable for each purpose: to pass university entrance exams: foster communicative competence: and expose the learners to language in authentic use?

The criteria could be difficult to meet but implementing the three aspects to materials are necessary in Japanese context. This is because students needs for a short-term are to pass the exams and for a long-term is to foster communicative competence. There are also a few opportunities to experience authentic English use out-of-class in Japan. Therefore, other workbooks or supplemented materials should be accompanied and provided if it is necessary.

• Adaptability: Are the materials adaptable to different class sizes and levels?

• Flexibility: Do the materials stretch/challenge the learners?

In Japan, secondary school classroom sizes vary depending on the area. The number of student might be a very small number or a large number, up to 40. Within the large class, students’ English proficiency also varies from low to high. Therefore, adaptability and flexibility are vital in the materials evaluation.

• Student interest: Are the materials visually attractive (illustrations/layouts) and interculturally effective?

Whether the materials are visually and interculturally interesting can play an important role in keeping their attention to English language.

Here, I have shared my evaluation framework. The framework is shaped based on my belief, experience and Japanese educational values. However, it could be changed or improved as I go back to teaching again and through analysing the outcomes of my teacher-made materials use.

Manami

 

References

Jolly, D., & Bolitho, R. (2011). A framework for materials writing. In B. Tomlinson (Ed.), Materials development in language teaching (2nd ed., pp. 107– 134). Cambridge: Cambridge University Press.

Mishan, F., & Timmis, I. (2015). Materials development for TESOL. Edinburgh: Edinburgh University Press.

Tomlinson, B. (2011). Materials development in language teaching (2nd ed.). Cambridge: Cambridge University Press.

Tomlinson, B. (2012). Materials development for language learning and teaching. Language Teaching, 45(2), 143–179.