Developing a Pre-Evaluation Framework and Applying it to English Unlimited

This experience opened my eyes to the world of materials evaluation and as I started to read about the topic I realised how vast and interesting this topic is. I felt as I was trying to understand this subject that it came down to one very important question, who is the evaluator? This question became more and more prolific as we attempted to develop a framework as it is through the particular perception of the evaluator the materials will be graded and conclusions of its sufficiency will be deduced. This brought me back to philosophy and particularly ontology. Social constructionism seemed most appropriate during this project as it assumes that we create a model of the social world and how it works and language is an essential system in which we construct our reality.

Weighting system results

Using this philosophical concept I began to realise during our conversations as a group that developing a hypothetical context in which to develop a framework and then apply that framework to a unit of a particular book would be impossible as we would still all have different results. Therefore, we devised a weighting system in which the evaluator first has to identify themselves and their preferences. This is because so many factors play a part in a teacher’s or evaluator’s life that contribute to the way they interpret the materials. Also, I feel that an evaluation is essentially a snapshot in time and once the evaluation has finished the evaluator will change and may evaluate the same materials again in a different way if they were asked to repeat the exercise later on (click on this link to see the presentation slide: evaluation presentation group b. The research was based on literature which was cited during the presentation and can be found below or at the bottom of this post).

One of the major limitations of the project was lack of time. This resulted in only a first draft of the evaluation questions and when we applied the questions to the textbook we realised that they were seriously insufficient. However, we did manage to check the questions with one of Tomlinson’s systems for devising evaluation questions so we felt that to some degree our questions were roughly framed within an evaluation style.

The collection of data became another issue fairly early on. We adopted a mixed methods approach to data collection as we agreed quantitative data had the ability to generalise enough and qualitative data gave an extra aspect for interpretation after the process. We imagined many scenarios when thinking about the analysis of the data, for example, a teacher being asked by their DOS or a DOS looking for themselves (just to give a couple of examples).

Also, due to time constraints we didn’t embed our framework into the bigger process an academy or school might go through. Firstly, we wanted to add that there would be an initial evaluation which would include materials analysis. This armchair evaluation 1 would probably be were the evaluator selects possible textbooks from a wider selection. Also, we never had time to develop the system for the evaluator to learn or follow when weighting the categories. Initial thoughts were to give each category a descriptor to reduce ambiguity and also offer them prompting questions to help guide them to know their personal weighting preferences.

Finally, we didn’t develop all the possible methods of analysing the quantitative data and analysing it with the qualitative data. This would have been a very interesting process as it may have changed our questions and even our weightings as evaluators.

Since presenting this to the class I have had many ideas in a way to improve this framework (some of which I have explained above) and as a result I am happy to attempt to offer this (with the revisions) to the school I work at presently. The DOS has agreed to hold a teachers focus group in March where the teachers will be asked to distribute weightings to the categories. Once the data is collected I will attempt to make an average weighting for each category that will hopefully better represent the teaching staff as a whole. Then I will conduct an evaluation of 4 potential books and compare this data with the weightings to hopefully offer the teachers a book in the autumn term which best fits everyone. However, in use evaluation and retrospective evaluation cannot be forgotten. I will require this extra data to firstly, see if the pre-evaluation of the book is sufficient and secondly to decided if this framework is useful or not.

In essence, this project isn’t over, as it has barely begun.

References

Chambers, F. (1997) Seeking consensus in coursebook evaluation ELT Journal 51 (1): pp.29-35.

Ellis, R. (1997) The empirical evaluation of language teaching materials. ELT Journal 51 (1): pp.36-42.

Johnson, K., et al. (2008) A step forward: investigating expertise in materials evaluation. ELT Journal 62 (2): pp.157-163.

Littlejohn, A. (2011) The analysis of language teaching materials: inside the Trojan Horse. In: Tomlinson, B. (ed). Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press. pp.179-211.

Masuhara, H. (2011) What do teachers really want from coursebooks? In: Tomlinson, B. (ed). Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press. pp.236-266.

Masuhara, H., et al. (2008 ) Adult EFL courses. ELT Journal 62 (3): pp.294-312.

McDonough, J., Shaw, C. & Masuhara, H. (2013) Materials and Methods in ELT: A Teacher’s Guide. 3rd ed. Oxford: Wiley-Blackwell.

McGrath, I. (2002) Materials Evaluation and Design for Language Teaching. Edinburgh: Edinburgh University Press.

McGrath, I. (2013) Teaching Materials and the Roles of EFL/ESL Teachers: Practice and Theory. London: Bloomsbury.

Mukundan, J. & Touran, A. (2013) A Review of Textbook Evaluation Checklists across Four Decades (1970-2008)In: Tomlinson, B. & Masuhara, H. (ed). Research for Materials Development in Language Learning. London: Bloomsbury.

Pryor, S. (2013) The Developments and Trialling of Materials for Second Language Instruction: A Case Study. In: Tomlinson, B. & Masuhara, H. (ed). Research for Materials Development in Language Learning. London: Bloomsbury.

Roberts, J. T. (1996) Demystifying materials evaluation. System 24 (3): pp.375-389. Tomlinson, B. (2011) Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press. Tomlinson, B. (2012) State-of-the-Art Article: Materials development for language learning and teaching. Language Teaching 45 (2).

Tomlinson, B. (2013) Developing principled frameworks for materials development. In: Tomlinson, B. (ed). Developing Materials for Language Teaching. London: Bloomsbury.