A Task-Based Analysis of Undergraduate Assessment: A Tool for the EAP Practitioner

This brief report summarises an empirical study that identified the range of tasks under-taken by neophyte undergraduate students and thereby provides EAP practitioners with information

BRIEF REPORTS AND SUMMARIES

TESOL Quarterly invites readers to submit short reports and updates on their work These summaries may address any areas of interest to Quarterly readers.

Edited by JOHN FLOWERDEW

University of Leeds

ALI SHEHADEH

United Arab Emirates University

A Task-Based Analysis of Undergraduate

Assessment: A Tool for the EAP Practitioner

KATIE DUNWORTH

Curtin University of Technology

Perth, Western Australia

䡲 Within the field of English for academic purposes (EAP), one issue that attracts consensus in the literature is the importance of needs analy-sis As Benesch (2001) observes, “needs analysis offers detailed informa-tion about the linguistic and cognitive challenges students face in aca-demic settings” (p 61) Although there is rather less unanimity on what those needs might be—see, for example, Pennycook’s (1997) critique of vulgar pragmatism—all approaches nevertheless accept certain funda-mentals One fundamental is the need for EAP teachers to be aware of the discourse types that occur in the domains to which EAP students will progress However, it is not always a straightforward matter to determine what these discourse types might be Thus it is necessary for EAP tutors

or curriculum designers to be familiar with the work that their students, who may be from a range of discipline areas, will be required to do Certainly numerous studies have been conducted into specific, and usu-ally written, academic genres, particularly at the graduate level, but there

is a paucity of widely available information on the complete range of tasks that undergraduates are required to undertake This brief report summarises an empirical study that identified the range of tasks under-taken by neophyte undergraduate students and thereby provides EAP practitioners with information for their ongoing efforts to design pro-grams

THE STUDY

The research took place on the main campus of an Australian univer-sity with approximately 35,000 students, more than 25% of whom are international The single site was selected because it was anticipated to generate richer data than a less focused approach and because it per-mitted local inspection of primary sources In other words, the study would use the material that students themselves encounter rather than filtered descriptions from respondents from a variety of institutions Within the study university, the tasks identified for analysis were com-pulsory, summatively assessed (i.e., allocated a mark that contributed to the final grade), and set in a student’s first semester of undergraduate study It is now widely acknowledged that summative assessment strongly influences and may even drive student learning (Birenbaum, 1997; Brown & Knight, 1994; Dochy & McDowell, 1997; Hanrahan & Isaacs, 2001; Maclellan, 2004; O’Donovan, Price, & Rust, 2004; Shepard, 2000; Struyven, Dochy, & Janssens, 2005; Swanson, Norman, & Linn, 1995; Wil-liams, 2005) The choice of first-year tasks was predicated on the belief that the influence of a pretertiary EAP course is most likely to be felt at an early stage of a student’s academic career, prior to experiential familiarisa-tion with the linguistic, academic, and social norms of the university

In Australia, academic degrees consist of specified groups of study units within a certain discipline Students are provided with unit out-lines, substantial documents that summarise the content of the unit of study and, among other things, describe the assessed tasks and assess-ment procedures These docuassess-ments provided the major source of data for the study When interpretation of tasks was problematic, clarification

or expansion was sought through semistructured interviews with unit coordinators, a process that involved 45 academic staff

From a campus-wide total of 139 units identified, 112 unit outlines, or 80%, from 32 discipline areas were obtained This percentage was con-sidered sufficiently high to yield reliable results, particularly because no disciplinary area or academic department was unrepresented in the ob-tained sample So that it would be possible to identify any interdiscipli-nary differences, the results were stratified according to the four aca-demic divisions of the university at that time: business, engineering and science, health sciences, and humanities The number of compulsory units for each area varied considerably In business, all majors share a common first year, and so the division produced only five compulsory units, while engineering and science produced a total of 59 Neverthe-less, the diversity of the assessed tasks the outlines generated was as great for the former as for the latter, providing further support to the argu-ment that a sample higher than 80% of the total would not have gener-ated markedly different results

More than 360 separately assessed tasks were identified, each of which was then classified into a task type according to six criteria that had previously been determined Lack of space precludes a detailed descrip-tion of how the criteria were selected, but they were devised following a process of integration of criteria used in a number of previous studies into university-level tasks, such as those of Bridgeman and Carlson (1983), Hale et al (1996), Horowitz (1986), and Moore and Morton (1999) Reference was also made to taxonomies of language proficiency, such as the Common European Framework (Council for Cultural Co-operation Education Committee, Modern Languages Division, Strasburg, 2001) and the model devised by Bachman (1990), so that the final classification system would be of particular use in an EAP context The six criteria were

Nomenclature of the task as named in the unit outline, for the insight it

provided into the academic’s intentions

Quantity of output, as measured by the number of words

Time allocated to task completion

Predominant macro-skill required for completion, selected because many EAP

courses are skills based

Source of information for task fulfilment, selected in order to recognise the

contribution of all macro-skills to the task

Cognitive demands, evaluated through the use of Bloom’s (1956) still

fre-quently cited Taxonomy and the work of Hale et al (1996) For example,

tasks could be categorized as belonging to a domain of “knowledge of specifics,” which required the linguistic skills of describing, explaining, listing, or summarizing; or to a domain of “knowledge of universals and abstractions in a field” (Bloom, 1956, p 202), which required the func-tions of evaluating, extrapolating, and analysing abstracfunc-tions

From the process of applying these criteria to each task, 19 different categories of task emerged, as described in Table 1 All assessed tasks, including those set in examinations or class tests were placed within the

19 categories In passing, it should be noted that tests and examinations featured prominently in three of the four divisions, in spite of the ex-panding body of literature that details and recommends alternative forms of assessment in higher education Tests and exams absorbed 50%

of all marks in business, 51% in engineering and science, 31% in health sciences, and 12% in humanities The impact of their content in refer-ence to the overall frequency and value of certain tasks is therefore extremely important

The tasks were then compared according to their relative frequency of occurrence in each division, and for their relative status in each division

in terms of marks awarded Both measurements were necessary to gain

an impression of the overall relevance of the task from the perspective of

a broadly based EAP program For example, an individual task might be valued at, say, only 10% of the total marks for a unit but might occur in

a large number of units Conversely, a task might attract a high percent-age of marks in one unit but not appear in any other units within the division and might therefore be seen as an outlier in the overall context

RESULTS

Table 2 shows how frequently, in percentage terms, a given task

oc-curred within each division For example, the first listed task,

participa-tion, occurred in 100% of the compulsory first-year units offered within

the division of business, 18% of those in the division of engineering and science, and so on The table shows that the most frequently occurring tasks, that is, those tasks which occurred in more than half the total units

for each division, are participation, multiple-choice questions, short answers, and timed essays in business; short answers, multiple-choice questions, and

TABLE 1 Categories of Task

Task type Task category

Predominantly speaking

and listening based

Formal presentations Participation Predominantly writing

and reading based

Annotated bibliographies Article/book reviews or critiques, literature reviews Case study reports

Extended essays, more than 1,200 words Journals, diaries, and learning logs Multiple-choice and true–false questions Reports on experiments, research, or field experience Short answers, including formulae, algorithms, definitions, and labelling or describing diagrams

Short essays up to 1,200 words Summaries of texts, information synthesis Timed essays, usually part of a test Practical Computer tasks (emphasis on the technology)

Designs, drawings, folios Laboratory work and workbook reports Library task (becoming familiar with the library and its resources) Nonlaboratory-based practicals

Other (a few tasks which did not merit their own category but could not be incorporated into any other group)

laboratory/workbook reports in engineering and science; short answers and multiple-choice questions in health sciences; and participation in humanities.

Table 3 illustrates the mean percentage score (rounded to the nearest whole number) allocated to a particular task when averaged across all

units obtained for each division For example, the first listed task,

par-ticipation, was allocated an average of 7% of the total marks in a unit

within the division of business, 3% of the total marks in engineering and

science, and so on Thus, whereas participation occurred as a task in all

units within the division of business, it was not accorded many marks and was therefore not highly valued as an assessed activity Those activities

which did attract the most marks in three of the four divisions were short

answers and multiple-choice questions, with 47% or more of the total marks

being allocated to these activities In humanities, which includes the departments of art, design, and architecture as well as social sciences, the

highest marks were allocated to designs, drawings, and folios and reports on

experiments/field trips.

Thus short answers and multiple-choice questions featured strongly in both frequency of occurrence and value in terms of marks awarded in three of the four divisions

In terms of the results by skill, tasks which involved some kind of oral output appeared in all divisions, although they were not allocated a high status with regard to marks, ranging from 4% of the total in engineering

and science to 15% of the total in humanities In all cases, participation

TABLE 2 Frequency of Occurrence of Tasks Within Academic Divisions (Percentages)

Task Category Business

Engineering and science

Health sciences Humanities Annotated bibliographies 20 0 4 6

Designs, drawings, folios 0 9 0 32 Experiments/field trip reports 0 27 15 48 Essays, extended (1,200+ words) 0 0 15 26 Essays, short (up to 1,200 words) 20 9 23 32

Formal presentations 40 5 19 19 Journals, diaries, learning logs 20 0 4 32 Laboratory/workbook reports 0 57 38 0

Nonlaboratory-based practicals 0 0 8 3 Multiple-choice questions 100 50 65 16

was considerably more important with regard to both frequency and

value than formal presentations, although the nature of participation and

the criteria by which it might be assessed remained somewhat opaque, even following interviews with unit coordinators

Perhaps unsurprisingly, assessment of written work varied among

dis-cipline areas Timed essays in examinations or tests were accorded the

highest status in business, constituting 17% of the total marks awarded,

but nontimed essays (short essays and extended essays) only attracted marks

in double figures within humanities Written tasks which were experien-tially focused in some way, through conducting an experiment or un-dertaking a field trip, appeared in all divisions except business, while

problem-solving tasks in the form of case studies appeared in all divisions except engineering and science Reflective writing, in the form of

jour-nals, diaries, or learning logs, appeared in almost one third of units in

humanities, but featured less strongly elsewhere In summary, extended writing of some kind represented nearly half of all marks awarded in humanities, around 40% of marks in business and health sciences, and under a quarter of total marks in engineering and science

Tasks overtly based on reading, such as critiques, reviews; annotated

bibliographies; and summaries, synthesis, were comparatively rare across the

board Indeed, reading of any kind did not appear to play the major role that might have been expected Subsequent analysis of all unit outlines revealed that required reading matter often consisted of a set of

lecturer-TABLE 3 Status of Tasks Within Academic Divisions (Percentages)

Task Category Business

Engineering and science

Health sciences Humanities Annotated bibliographies 1 0 2 1

Designs, drawings, folios 0 5 0 21 Essays, extended (1,200+ words) 0 0 4 9 Essays, short (up to 1,200 words) 2 1 3 6

Experiments/field trip reports 0 6 5 13

Journals, diaries, learning logs 4 0 1 8 Laboratory/workbook reports 0 10 10 0

Nonlaboratory-based practicals 0 0 2 1 Multiple-choice questions 23 11 17 9

compiled readings, lecture notes, or single chapters from text books Few units listed more than two or three set reading texts

DISCUSSION

What have these findings to do with EAP courses? Whatever the para-digm in which an EAP course is framed, it is important for teachers to be aware of the practical demands that will be made of students after gradu-ation Given the large body of research that demonstrates the primacy to undergraduate students of assessed tasks, it can be argued that the EAP syllabus should prioritise those tasks which course participants are most likely to encounter, whether it be from a pragmatic perspective, to emu-late them, or from a critical perspective, to learn how to contest them EAP professionals, who are themselves frequently graduates from hu-manities, education, or social science disciplines, are not automatically aware of the range of disciplinary distinctions that exist, so it is important that professionals have access to such data With this knowledge it is possible not only to incorporate the most featured and relevant tasks into the syllabus, but also to explore with students the many issues that might arise with regard to assessment, such as misunderstanding the task itself, differing interpretations of assessment rubrics, lack of familiarity with assessment criteria, potential conflicts about the nature and function of the task, and uncertainty over the rights and responsibilities of students For example, research indicates how open to interpretation are the ab-stract terms used in the rubric of assessed tasks (Chanock, 2000) and in assessment criteria (O’Donovan, Price, & Rust, 2004)

Furthermore, familiarity with the assessed activities undertaken at an institution not only indicates to university students “the kind of intellec-tual work which is valued” (Maclellan, 2004, p 20), but also provides EAP teachers, who are not always used within a tertiary environment them-selves, with a window into the organisation’s underlying value system and epistemological position

This does not mean, of course, that those forms of assessment which students are most likely to encounter and which will most impact on their grades should be uncritically foregrounded in EAP programs In-stead, assessment must take into account both the activities which pre-cede the actual task and the way in which students are taught at the tertiary level For example, a narrow focus solely on future assessed out-put would, as the results of this study indicate, heavily weight content toward the development of certain forms of writing at the expense of other skills Yet as we know, the construction of the knowledge used in producing the content for the various writing genres is facilitated

through a wide range of activities, only some of which involve writing For example, the literature has long made reference to the difficulties experienced by international students participating in tertiary-level dis-cussion groups and seminars or in integrating with the local student population (e.g., Chalmers & Volet, 1997; Mullins et al., 1995) Space does not permit a discussion of the reasons for this, which are numerous and complex However, it could certainly be argued that unless students are also encouraged to develop the aural/oral skills that will facilitate their acquisition and construction of disciplinary content knowledge, the advantages conferred by the development of any assessment- related writing skills are greatly diminished

Nevertheless, if EAP courses are to dedicate at least part of their syllabi

to the modelling and emulation or critique of those tasks that students are later to encounter (and it is certainly the case in Australia that this is what many programs purport to do), then it is essential that EAP prac-titioners have an active and ongoing awareness of the assessment envi-ronment for which they are preparing their students, whatever their discipline

Although this research is a single-site study carried out in Australia, it contributes to the body of knowledge and provides a comparison for other broad-based investigations EAP is a wide field that incorporates a range of learning contexts, as Casanave (2004) points out Within that overarching paradigm, decisions on local course content need to be made, and there is considerable value in having available to EAP prac-titioners the kind of data that will assist with making sensitive, informed, and reflective choices

THE AUTHOR

Katie Dunworth is an associate professor in the Department of Languages and In-tercultural Education at Curtin University of Technology, Perth, Western Australia She has more than 25 years of experience in TESOL as a teacher, administrator, and researcher Her particular research interests are educational assessment and the internationalization of higher education.

REFERENCES

Bachman, L F (1990) Fundamental considerations in language testing Oxford: Oxford

University Press.

Benesch, S (2001) Critical English for academic purposes: Theory, politics, and practice.

Mahwah, NJ: Lawrence Erlbaum.

Birenbaum, M (1997) Assessment preferences and their relationship to learning

strategies and orientations Higher Education, 33, 71–84.

Bloom, B S (Ed.) (1956) Taxonomy of educational objectives London: Longman.

Bridgeman, B., & Carlson, S (1983) Survey of academic writing tasks required of graduate and undergraduate foreign students Princeton, NJ: Educational Testing Service Brown, S., & Knight, P (1994) Assessing learners in higher education London: Kogan

Page.

Casanave, C P (2004) Controversies in second language writing: Dilemmas and decisions

in research and instruction Ann Arbor, MI: University of Michigan Press.

Chalmers, D., & Volet, S (1997) Common misconceptions about students from

South-East Asia studying in Australia Higher Education Research & Development, 16,

87–98.

Chanock, K (2000) Comments on essays: Do students understand what tutors write?

Teaching in Higher Education, 5(1), 95–105.

Council for Cultural Co-operation Education Committee, Modern Languages

Divi-sion, Strasburg (2001) Common European framework of reference for language: Learn-ing, teachLearn-ing, assessment Cambridge: Cambridge University Press.

Dochy, F., & McDowell, L (1997) Assessment as a tool for learning Studies in Educational Evaluation, 23, 279–298.

Hale, G., Taylor, C., Bridgeman, B., Carson, J., Kroll, B., & Kantor, R (1996) A study

of writing tasks assigned in academic degree programs Princeton, NJ: Educational

Testing Service.

Hanrahan, S., & Isaacs, G (2001) Assessing self- and peer-assessment: The students’

views Higher Education Research & Development, 20, 53–70.

Horowitz, D (1986) What professors actually require: Academic tasks for the ESL

classroom TESOL Quarterly, 20, 445–482.

Maclellan, E (2004) Authenticity in assessment tasks: A heuristic exploration of

academics’ perceptions Higher Education Research & Development, 23, 19–33.

Moore, T., & Morton, J (1999) Authenticity in the IELTS academic module writing test: A comparative study of Task 2 items and university assignments In R Tulloh

(Ed.), IELTS research reports 1999 (Vol 2; pp 64–106) Canberra: IELTS Australia.

Mullins, G., Quintrell, N., & Hancock, L (1995) The experiences of international

and local students at three Australian universities Higher Education Research & Development, 14, 201–223.

O’Donovan, B., Price, M., & Rust, C (2004) Know what I mean? Enhancing student

understanding of assessment standards and criteria Teaching in Higher Education,

9, 325–335.

Pennycook, A (1997) Vulgar pragmatism, critical pragmatism, and EAP English for Specific Purposes, 16, 253–269.

Shepard, L (2000) The role of assessment in a learning culture Educational Re-searcher, 29(7), 4–14.

Struyven, K., Dochy, F., & Janssens, S (2005) Students’ perceptions about evaluation

and assessment in higher education: A review Assessment & Evaluation in Higher Education, 30, 325–341.

Swanson, D B., Norman, G R., & Linn, R L (1995) Performance-based assessment:

Lessons from the health professions Educational Researcher, 24(5), 5–11, 35.

Williams, K (2005) Lecturer and first year student (mis)understandings of

assess-ment task verbs: “Mind the gap.” Teaching in Higher Education, 10, 157–173.