Framework for Evaluating ELT Materials

Posted on February 29, 2016 by Alexandra Collins

For last week’s seminar we all had to read up on the development of frameworks for materials evaluation. In groups we then had to create our own framework that we could see ourselves using when faced with the task of evaluating a textbook (or other ELT materials). Some groups furthermore had to apply their framework to the evaluation of one particular textbook. These groups then presented their frameworks and the findings of their evaluation in class.

The group I worked in did not have to apply their framework to the textbook. What follows therefore is our idea of a general framework which we arrived at after reading a number of articles on the subject and which I adapted after seeing the other groups’ presentations (Framework for Evaluation of ELT Materials):
Whilst I agree with authors like Ellis (in Tomlinson, 2011: 214) who argues that materials are most effectively evaluated ‘in action’, the reality probably is that most evaluations happen pre-publication initiated by publishers who are interested in the opinions of teaching professionals regarding a new product and maybe some limited ‘test-driving’ of that product. The framework we devised is therefore to be used in a situation where one evaluator or a small group of evaluators test the value of a new coursebook in a teaching context he/she/they are familiar with. If evaluators do not work alone but collaborate on an evaluation, I think it is essential that they work within the same context in order for the evaluation to be most effective.

Evaluator’s Principles
The first step in the evaluation process, in my opinion, should be to establish the evaluator’s/evaluators’ principles. When we talked about and established our principles during our first seminar we realised that there were many that we shared, some that were particularly valuable to us personally, some derived from current trends in SLA pedagogy and some based on our beliefs about learning in general. In any case, our principles have to be known to us and those who choose us to do an evaluation, as they will colour our opinions and analysis of the material. Once principles are established they can be compiled in a list of criteria against which the materials are checked to find out to what degree the materials meet these criteria (e.g. in a questionnaire using the Likert Scale).

Profiling
The second step involves the profiling of the group of learners the materials are aimed at, of the teachers who will be teaching with the materials and of the setting in which the teaching is to take place. In combination the different profiles describe the proposed context of use. I agree with Masuhara who suggests that often the analysis of teacher’s wants and needs gets ignored beside the analysis of learner needs (in Tomlinson, 2011: 236-266). Masuhara holds that both are equally important, if one is to provide teachers with materials that are truly useful to them and consequently effective for teaching. As in the first step profiling should result in criteria that can then be tested against the materials.

Materials Analysis
The third step should look at the material itself. I like Littlejohn’s idea of analysis involving various stages going from ‘objective description’ through to ‘subjective analysis’ and finally to ‘subjective inference’ (in Tomlinson 2011: 185). It is true that at the first level it is possible to be quite objective as you describe the overall appearance of the material, the layout and staging of content, publishing details etc. As it is impossible to analyse a whole book in minute detail the selection of parts of the textbook that are worthy of closer inspection becomes more subjective. However, what follows, i.e. a step-by-step description of exactly what happens in a task is then objective again, i.e. still an analysis. I think it is useful, as Littlejohn suggests, to see ‘task’ at this point to mean simply ‘what we give students to do in the classroom’ (Littlejohn quoting Johnson 2003: 5 in Tomlinson, 2011: 188). A more TBLT (Task Based Language Teaching) definition already presumes a specific interpretation of ‘task’ which is not applicable to all tasks and therefore cannot be applied in the evaluation of textbooks that are not informed by TBLT. The last level of analysis is more subjective than the second as it moves away from simply describing what’s there to the drawing of ‘some general conclusions about the apparent underlying principles of the materials’ (Littlejohn in Tomlinson, 2011: 197), e.g. by looking at who does what with whom the evaluator can guess at the roles that teachers and learners are expected to take on during an activity.

Figure 8.7 (Littlejohn et al, 2008 in Tomlinson, 2011: 196)

Evaluation
Once steps one to three have been completed the evaluator can look at all of the criteria together combined with his/her findings from the analysis of the material itself. This would be the actual evaluation stage. For example, the evaluator might have decided that learner-centredness and collaboration are amongst his/her valued criteria. The profiling shows that the teacher is happy in principle to hand over control to his/her students from time to time and confident about classroom management in such situations. The profiling also shows that students have experience of and are comfortable with working in a team. The analysis of the material finds that several tasks include group work activities. Overall it would then seem that the material is suited to the context as well as the pedagogical aims. As suggested earlier it would be most useful if these claims could then be tested by using the material with real students (preferably different classes in the same institution taught by different teachers) over a period of time.

References

Tomlinson, B (2011) ‘Introduction: principles and procedures in materials development’ in Tomlinson, B (ed) Materials Development in Language Teaching (2nd edition). Cambridge: Cambridge University Press

Littlejohn, A (2011) ‘The analysis of language teaching materials: inside the Trojan Horse’ in Tomlinson, B (ed) Materials Development in Language Teaching (2nd edition). Cambridge: Cambridge University Press

Masuhara, H (2011) ‘What do teachers really want from coursebooks?’ in Tomlinson, B (ed) Materials Development in Language Teaching (2nd edition) Cambridge: Cambridge University Press

Materials Design – A Writer’s Perspective

Posted on February 26, 2016 by Alexandra Collins

In this week’s seminar an ELT textbook writer came to talk to us about her work and we were able to ask a few questions and hear her insightful answers that reflected her many years of experience.
When we had been thinking about our principles and how we could see them informing any of our own potential ELT material design, we had mainly been concentrating on our own teaching experience and knowledge of learning theories and SLA as sources of information. It was enlightening to learn or be reminded that in the real world of materials publishing the first set of criteria to be met is determined by the publisher. These criteria are essentially of a practical nature – the publisher’s main concern is that the new product is popular and sells well. So the initial brief might be that the materials author produce something ‘conventional, generic’, something that appeals to as wide an audience as possible and nothing too ‘exciting or new’. In comparison to our rather idealistic principles informed by the latest pedagogical trends this does not sound very exciting but rather restrictive. However, being confronted with this reality drove the point home that materials design is not only or even not predominantly a creative process but instead about ‘business’.
Nevertheless the textbook writer confessed that overall she really enjoyed her work and that creativity did come into it. In fact, the challenge she seemed to relish most was to get from the point of creation, which in her words is ‘just like writing a lesson’, through the many often ‘tedious’ stages of editing, checks against other writers’ input and negotiations with the publisher to arrive at a product that you are proud of that, however, also fits the more restrictive criteria. Although it might feel frustrating to be unable to follow your ideals within the world of publishing, I can see that meeting the challenge of marrying business with education is a distinctive and valuable skill and rewarding in its own right. It seems that the main traits you need are openness (to listen to other peoples’ ideas), flexibility (to meet the diverse needs of publishers, teachers and learners), conviction and belief in yourself and your ideas, determination to fight the battles worth fighting for (against publishers?) and a thick skin. Interestingly, our writer also said that a ‘big ego’ does not help in textbook design: you do not get credited for all your hard work other than receiving a paycheck at the end.
In response to our questions we found out a lot about the process of and current trends in materials design. For example, a lot of the language used in a textbook now gets drawn from or at least checked against corpora of written and spoken English. This seems to indicate that there is now an interest in teaching language as it is used in the real world (granting that technological advances in recent years were a prerequisite which made the compiling of real language samples and access to the resulting corpora possible). In the case of English most of the corpora used in this way are compilations of native-speaker language samples. However, ELF (English as a Lingua Franca) corpora are beginning to exist (e.g. VOICE) and might inform (parts of) future textbooks of English.
Another trend is that of ‘UK-centricity dying out’. According to our textbook author, publishers do not want lots of information and references to Britain and its culture anymore but instead aim for products that have an international appeal (e.g. no UK celebrities). She admitted that she had found it difficult at first to ‘get rid of her cultural baggage’ but that over time she got good at spotting ‘cultural bias’. I am sure that this is the case, but I also wonder how achievable this really is when all coursebook designers working on a package together are in fact British and working in Britain? This is probably why publishers sometimes give the option of editing a textbook package to suit the country they are selling to…
To end with our textbook writer pointed to the fact that throughout her time as a materials designer she also worked as a teacher and always welcomed the opportunity to reconnect with the profession after times of immersion in materials design. In her opinion materials writers should be (recent) teachers and never lose touch with the reality of the classroom. I couldn’t agree more! It is easy to forget very quickly what it actually feels like to teach and I cannot imagine that you can possibly come up with effective lessons from an outsider’s perspective.

Ludus

Posted on February 25, 2016 by Alexandra Collins

I just came across something rather exciting that I must have known once: the Latin word ‘ludus’ means ‘school’ and ‘game’! I was reading ‘Ready Player One’, a novel by Ernest Cline about a Massive Multiplayer Online Game (MMUG), and there it was (p 69). The main character in the story, Wade, suddenly has the same revelation and this marks a turning point in his virtual game quest.

This made me think about why these two meanings, which seem very different from each other today, were once seen as closely connected. An internet search unearthed this book: ‘Games: Purpose and Potential in Education’ edited by Christopher Thomas Miller. So far I have only had the chance to read sections but the following quote is particularly interesting in this context:

“[W]e mentioned the Latin word ludus, which was used for institutional or performance games. A most interesting finding, perhaps even shocking to some readers, is that the same word belonged to the world of education as well. Teachers were in fact called magister ludi (literally, Game Master) and what we could today call schooling was called ludus (literally, Game). […] education was the means to groom children into real men and citizens of the nation, through the tutelage of a wise teacher, as in the case of Alexander the Great, who was tutored by Aristotle. In this sense education or schooling (ludus) was an institutionalized means to engage youngsters in autotelic activities meant for the development of a free person. While historical differences are great and would require hundreds of pages to be accounted for, the ideal aim assigned to the education system is strikingly close to our perception of what schools should be made for.” (p 17)

The expectation that some readers might find it “shocking” that the concepts of ‘game’ and ‘school’ used to be covered by one word in the civilisation we often see as the origin of our own, shows that we are now conceptually quite removed from such an understanding. Maybe this historical connection accounts to some extent for our continued interest in using games for educational purposes?

By the way, if you like me struggle to get your head round the concept of MMUGs (and do not want to become a player to get a clearer idea), give ‘Ready Player One’ a go…

References:

Cline, E. (2011) Ready player one. London: Random House Publishing

Miller, C.T. (ed.) (2009), Games: purpose and potential in education. New York: Springer

Textbook Projects

Posted on February 24, 2016 by Alexandra Collins

A tweet by Alec Couros about the BcCampus Open Textbook Project caught my eye the other day as it reminded me of something I had read about in Brian Tomlinson’s introduction to Materials Development in Language Teaching. In the article I had particularly liked the parts about textbook projects (Tomlinson in Tomlinson ed., 2011: 10 & 24/25), i.e. material design collaborations between teachers, curriculum designers, publishers, researchers and other experts. I am not particularly fond of centrally produced, prescriptive textbooks so the idea of a collaborative enterprise of this kind is very appealing and I can certainly see why Tomlinson would call it “productive”. Unfortunately I was unable to find anything more about the projects Tomlinson was involved in. However, I believe the contributions below also have relevance to ELT.

The purpose of the textbook project in British Columbia, Canada, is to make educational resources affordable and accessible to a higher number and a wider range of students. These open educational resources (OERs) are based on the following principles of open education:

Retain – i.e. no digital rights management restrictions (DRM), the content is yours to keep, whether you’re the author, instructor or student.
Reuse – you are free to use materials in a wide variety of ways without expressly asking permission of the copyright holder.
Revise – as an educator, you can adapt, adjust, or modify the content to suit specific purposes and make the materials more relevant to your students. This means making it available in a number of different formats and including source files, where possible.
Remix – you or your students can pull together a number of different resources to create something new.
Redistribute – you are free to share with others, so they can reuse, remix, improve upon, correct, review or otherwise enjoy your work.

I also came across the ‘Global textbook project: new horizons in textbook marketing’ (Pitt et al, 2009). Here the context of teaching is not ELT but Business Studies. This project was the brainchild of Professor Rick Watson of the University of Gerogia’s Terry College of Business. After failing to source a textbook he deemed appropriate for his graduate class he decided “that the textbook for the course would be written, reviewed, and edited by the course members and ultimately used by the course members” (299/300) using Wiki software to create a Wikibook. The main goal was to create teaching materials that were relevant and free to the student-authors as well as students on subsequent courses. Although the students had to work extremely hard to produce a textbook of a high enough quality to pass on to future classes as well as management of the project being considerable, the end product (maybe not so much ‘end’ as future users can make contributions of their own) and the concept were a success overall. The project continued to grow into the ‘global text project’ which aims to open up more textbooks that are freely available to the user.

I don’t see why the above principles could not be transferred to ELT. While the workload involved in the projects seems considerable for both students and teachers, the obvious attraction of the products they generate lies in the fact that these are affordable, likely to be more relevant to the unique teaching context, flexible (as they allow changes over time) and encourage a high level of interaction between all interested parties.

References:

Pitt, L., Nel, D., Van Heerden, G. & Chan, A. (2009), Global textbook project: new horizons in textbook marketing. Marketing Intelligence & Planning. 27 (3): 297-307

Tomlinson, B. (2011) Introduction: principles and procedures of materials development. In Tomlinson, B. (ed). Materials Development in Language Teaching. (2nded) Cambridge: Cambridge University Press. pp.1-31.

‘Demystifying Materials Evaluation’

Posted on February 23, 2016 by Alexandra Collins

Ivory Tower Syndrome

Let me start by saying that this article left me completely ‘mystified’! I just could not read past the language. I like interesting and unusual words but just in the first five pages I read there were several that I have never come across (and that is without counting the Latin ‘inter alia’ – p. 375 – or French ‘reculer pour mieux sauter’ – p. 377 !). What on earth are ‘terminological obfuscations’ (p377) or an ‘ineluctable problem’ (p. 380)? Of course, I am a non-native speaker of English but I have lived in this country for over 20 years, managed to complete an undergraduate course and am currently studying at Masters level. So I don’t think education is the problem. Who is Roberts writing for? As he is talking about educators and teachers being involved in the evaluative process (377/378) it seems to me that he must consider them to form at least part of his ‘audience’. His writing style, however, is highly academic (or is it just the language of someone who has spent too much time in education for the privileged?) and surely I am not the only one to find him difficult to understand?! Even more frustrating than Roberts’ choice of words are some of the sentences he comes out with. Here’s an example:

The model assumes (and since we are talking here about an ideal, we will for the moment take for granted the complete scrupulousness and dedication of all involved at all stages between conception and adoption) that the evaluation process begins not even as late as the moment at which the materials designer types the first plan, but within a short time of the conception of the first germ of an idea for a set of materials for a certain target population of learners, in just the same way as the first flush of elation following an idea for an article, for instance, is soon tempered by doubt and self-questioning. (Roberts, 1996: 377)

I am convinced that I would face criticism, if I came to my tutors with an essay draft that included anything like this. Is Roberts because he is a published author of authority in his field beyond that kind of criticism?

Maybe I am being a bit unfair to question the value of Roberts’ article purely because of his writing style or maybe things have changed since its publication anyway. However, the reason I have chosen to write a post about this article on my blog is that it exemplifies an issue I have with academia in general. In our seminars we are talking about how not only experts’ but teachers’ voices need to be/are starting to be heard more and how teachers should have/are starting to have an impact on educational policy making. If articles like the one by Roberts, however, are still considered to be quintessential reading, then I fear the wisdom of experts is likely to stay in expert circles as his writing style is, in my opinion, pretty incomprehensible to people of average higher education (if such a thing exists?). If experts and practitioners want to share their knowledge and work together in order to improve education, then we need to communicate in a language that we all understand so that communication is not hindered by time-consuming decoding, interpreting by third parties and possibly even second-guessing and so that the meaning of what we say is accessible to as large a number of participants as possible.

Roberts, J T (1996), ‘Demystifying Materials Evaluation’, System 24. No. 3. pp 375-389

Materials Design: Principles and Frameworks

Posted on February 22, 2016 by Alexandra Collins

Principles/Frameworks vs Spontaneity/Inspiration:

Words to describe the process of materials desing

Materials Design Process

The topic of our last seminar was ‘Principles and Frameworks for ELT Materials Design’. To start with our discussion centred around the question whether ELT material designers do and/or should base their work on a set of underlying principles and follow a framework during the design process. Apparently a lot of experienced designers deny or are not aware of being guided in that way but instead believe that they use their inspiration and intuition (see Tomlinson, 212: 152/153). In our class we agreed with some findings by Johnson (2003 in Tomlinson, 2012: 153) which indicate that as a novice materials designer you might want to have some guidelines to follow and that it is therefore useful to study how designers move along the process of creating new materials. As we are all potential future designers of ELT materials, Paul thought it would be a good idea for us to reflect on our own personal beliefs about what we think makes for good teaching materials.

What are the principles that guide materials design?

To help us with the reflective process we were given an exercise which made us first list our own personal principles, then compare these with those of other members in our group as well as between groups. Finally we were given some statements by experts in the field of materials development to compare with the ones we had come up with as a class.

Commonalities and Differences

The exercise was interesting and thought-provoking. At first I thought that I had worked out my principles over the course of my studies on the MA so far and during the recent process of essay-writing. I suddenly found that my beliefs about effective teaching/materials had very much become coloured by my studies of SLA last year and the learning theories I had immersed myself in over the last few months. What struck me most was how much closer to the practice of teaching some of my colleagues’ principles were (especially those of the diploma students)! I particularly loved the straightforwardness of one of my colleague’s statements: ‘Materials should be teachable’. How simple and to the point! Obviously what is and isn’t ‘teachable’ is very subjective. But however much I might be in love with a particular theory or convinced by someone’s research findings, in the end effective materials are about whether I can teach with them in my particular class tomorrow.

My Group’s Principles

When I first started the MA I was exasperated by how theoretical and removed from the classroom a lot of the research seemed and I spent a lot of time and effort ‘fighting’ theory. However, over the last few months I have completely fallen in love with theory and could endlessly keep reading, philosophising and thinking. It’s about time that te714 brought me back down to earth!

All this is not to say that there wasn’t a lot of agreement amongst all of our statements. Most of us seemed to think that materials should be engaging, interesting, learner-centred, flexible, varied, context-sensitive to name but a few.

I suppose the exercise showed us that even ‘principles’ are not as static as you might think. They are, of course, subjective but they can and need to be continuously reviewed first by ‘being brought to the surface’ of your own thinking and also by being discussed with other professionals.

‘Silent Period’

To finish I would like to reflect on one more thing. One of my principles did not make it into my group’s final selection: ‘Materials should encourage/leave space for a silent period’. Until the discussion last Thursday I hadn’t realised how important this is to me at the moment. When I first came across the notion of a ‘silent period’ in SLA last year I thought about it quite literally as in some learners needing a period of listening to a new language before they can start talking (this has been observed in particular with children and has been seen in connection with first language acquisition, which of course has an extensive ‘silent’ period before ‘proper’ language is produced starting from around age 1).

Reflecting on my own process of learning since the beginning of my course at uni I have started to think of it more as times when you do not directly engage with new learning material or even times when you are engaged in an activity that is completely unrelated to the subject you are learning about. I have found, for example, that over the course of last summer when I completely disconnected from anything to do with ELT, somehow everything had become much clearer and at a much deeper level at the start of my second year. It’s as if my brain by being ‘left to it’ made connections ‘independently’ of conscious thought. It seems I generally need a silent period at the start of learning something new/being exposed to something unfamiliar, some time to just listen to my tutor, my peers’ ideas, to ‘read, read, read’ or to walk away from it completely. I don’t like the pressure of having to create something while I haven’t got a grip on it/how it fits in yet. It’s as if my brain needs to put on a brake so that it can organise my thoughts, internalise new knowledge and make connections to what I already know. This has made me wonder whether I give my students time for that to happen…

References:

Johnson, K. (2003) Designing Language Teaching Tasks, Basingstoke: Palgrave Macmillan

Tomlinson, B. (2011) Materials Development in Language Teaching, 2nd Ed, Cambridge Language Teaching Library, Cambridge: Cambridge University Press.

Tomlinson, B. (2012) “Materials Development for Language Learning and Teaching”, Language Teaching, 45 (02) pp. 143-179.

Working, Playing, Learning

Posted on February 16, 2016 by Alexandra Collins

I have decided to start this blog with a quote that a Facebook friend recently shared with me: firstly because I wholeheartedly agree with it and therefore it can help the reader ‘read’ me and decide whether I might be the kind of person they want to engage with. Secondly, I think it fits nicely within the context of ‘learning’ and therefore, by extension, ‘learning materials’, i.e. TE714, the module I am writing this blog for.

“This is the real secret of life – to be completely engaged with what you are doing in the here and now. And instead of calling it work, realize it is play”

Alan W. Watts

Like Watts I am fascinated by Eastern philosophy and believe that being ‘engaged’ means to be fully present in the moment. This is how I try to go through life and this is how I have attempted to approach my MA course at Brighton uni (I am a part-time student of MA TESOL with ICT in my second and hopefully final year).

As a teacher I would be thrilled to see this level of engagement in my students! Sadly that is, of course, very rarely the case. However, I believe that by leading by example and by creating interesting activities and using engaging materials I can maybe, hopefully, bring about conditions in the classroom that make my students temporarily forget they are there to ‘work’ (that seems to be most students’ interpretation of ‘learning’). And that they, at least occasionally, find themselves having fun in a lesson, with learning just happening almost accidentally.

Hello world!

Posted on February 16, 2016 by Alexandra Collins

Welcome to your brand new blog at University of Brighton Blog Network.

To get started, simply log in, edit or delete this post and check out all the other options available to you.

For assistance, visit our comprehensive support site and check out our Edublogs User Guide guide.

You can also subscribe to our brilliant free publication, The Edublogger, which is jammed with helpful tips, ideas and more.

Alex Collins on ELT Materials

"Asking questions is what brains were born to do." Alison Gopnik

Monthly Archives: February 2016