Throughout this blog the idea of evaluating tasks has been central to the on-going evolution of material design. Ellis (1998) states that evaluation can be used to determine the pedagogical merits of one activity or it can be used to facilitate the process of curriculum development. This is what Ellis calls micro-, and macro-evaluations.

Macro-evaluation can be defined as evaluation that seeks to answer one or both of the following questions:

1. To what extent was the programme/project effective and efficient in meeting its goals?
(Accountability evaluation)

2. In what ways can the programme/project be improved?
(Development evaluation)

Macro-evaluations are less likely to have much concern for the teachers in the classroom, whose focus is on the learning and development of the students. They are less likely to look at a programme as a whole, and will focus on whether specific activities and techniques appear to ‘work’ in the context of a particular lesson. A teacher’s macro-evaluation is probably going to manifest from a whole host of micro-evaluations, taking place throughout the set period of time. Micro-evaluations are characterised by a narrow-focus on specific aspects. In terms of materials, we might ask whether a particular activity is effective or efficient in achieving learning goals. Although, to determine these two states as satisfactory is difficult to do without a systemic and principled approach to evaluation.

As mentioned in my previous post, re: evaluations of course books and materials. The evaluation can be conducted at one or at all three stages: Pre-use, during use, and post-use. Similarly, tasks can be evaluated at different stages of planning, use and reflection. Ellis (1989) suggests that you can start with an external evaluation (pre-use) examining the tasks claims about: intended audience, proficiency level, the context in which writers of materials intend them to be use, the way the language has been organised into teachable units, and the writer’s’ views on language and methodology. This is can be followed by an internal evaluation (in-depth) investigation of the aspects of presentation of the materials, the grading and sequences, the kind of texts used, and the relationship between exercises and tests.

With some many considerations and the many different reasons for evaluating a task, a balance needs to be found based on the purpose and context. When looking at using authentic texts for example, there is a positive aspect of naturally occurring language input, but this may have an equal weighting of the negative considerations for vocabulary overload. The overarching theme (throughout the blog) being that it must be the needs of the learners that are at the core of these considerations.

There are seven dimensions of evaluation put forward by Ellis (1989) that are applicable to both micro- and macro-approaches:

I. Approach
II. Purpose
III. Focus
IV. Scope
V. The evaluators
VI. Timing
VII. Types of information

Taking these seven dimensions into account, I will look at how they would apply to a micro-evaluation of a task.

The approach is about looking at the method or beliefs that the task is created for. Approaches are not polarised but are part of a large spectrum. In an attempt to simplify this Ellis (1998) bookends the spectrum with two generalised models: focused and unfocused tasks i.e. more behaviourist and objective moving toward communicative and sociocultural perspectives. The approach needs to classify:

• The type of input (e.g. verbal or nonverbal) and
• The procedure i.e. activities the learners are to perform (experiential or elicitative language use).
• The language skills the students are going to be using (receptive or productive).
• The intended outcome of the task, that is, what the learners will have done on completion of the task.

The approach informs the second dimension, which is to reflect on the purpose of the task. This is also connected to the timing dimension. The purpose is trying to measure and give feedback on learning, or to develop and reflect on the task (accountability and development). A full description of the purpose requires objectives of the task, ‘real’ world or pedagogic to be clear. Real world is asking learners to approximate in class what is required of them outside, whereas pedagogic requires some language activity which is not found in the real world but does facilitate language acquisition. In the context of micro-evaluation it is difficult to establish whether the learners acquired new linguistic knowledge or improved fluency.  Therefore it may be necessary to evaluate whether the hypothesised methods effects on language acquisition have been achieved instead of defined learning objectives. Commonly, timing looks at whether the task is going to be formative or summative in its feedback. Micro-evaluations can do both depending on when the task is completed within the curriculum. The task’s timing will go some way to  aiding the data that determines its focus.

Focus is looking at the effectiveness and efficiency of the task. This appears to be a difficult dimension to evaluate. Firstly, it may be necessary to judge what and how well something has been learned upon the task’s completion. If the learner shows gains in those aspects of language proficiency, which were the starting objective, then the task can be considered ‘effective’. If something is learnt is hard to clarify and requires scope. To glean if something is efficient it would need to be compared to other tasks that have similar structure and objectives, which is very difficult to do. It must also use the same types of information. This data can be drawn from traditional objective models such as test scores. While, a more holistic approach could be considered, where the evaluation is directed a greater variety of information such as: documentary information of student work (coursework), self-reflection and discussion of progress and development, and observational feedback from the classroom.

To recognise which of these types of information is needed, one must address the scope. This tries to value if the task meets its ‘stated’ goals, or whether to examine the goals themselves (are the goals appropriate for the learning question). The internal scope uncritically accepts the goals of the programme, while an external scope submits the goals for critical scrutiny. External evaluation of scope will inquire if unpredicted learning has taken place. Two types of information collection should be considered: information relating to the outcome of the task that is the focus of the evaluation. Secondly, the data relating to whether, the learners can perform the task (or similar ones) without pedagogic support?

Finally, the evaluators themselves need to be considered. Who is conducting the evaluation? Are they within the institution or and external candidate? Internal evaluations may be considered more effective as it needs involvement from ‘all stakeholders’ who have investment. However, outside ones may offer a more objective, ‘fresh’ perspective and may give more credibility. Another option is for an advocacy evaluation, this asks various parties to argue their case for their own bias and stances. It can be argued that this is more likely to foster development, but only if that is the goal of the evaluation (Ellis, 1998). An evaluation needs clear distinctions between its conclusion and recommendation. In the cyclical nature of materials design, recommendations should be given based on the conclusions drawn from the seven dimensions all being considered.

Evaluations are very difficult for me to comprehend out of context. It is important to remember that it is the clarity of what is being evaluated that must come first. The systemic and principle approach can only work if it is guided by its purpose. I feel that the more I engage with task creation and evaluate the approach and scope of my tasks. There are going to be suitable for my students, and the curriculum aims and are more likely to satisfy the focus of macro-evaluations both internally and externally. Evaluation will also allow for more reflection and development of my teaching beliefs, style and effectiveness.

References

Ellis, R. (1998) The evaluation of communicative tasks. In: Tomlinson, B. (ed). Materials development in language Teaching. (2nd ed) Cambrdige: Cambridge University Press. Pp212-235.

Ellis, R. (2011) The evaluation of communicative tasks. In: Tomlinson, B. (ed). Materials development in language Teaching. (2nd ed) Cambrdige: Cambridge University Press. Pp217-238.