Literature in Secondary School to Its Learning Assessment in the 12 TH Year National Exam

Literature plays a core role in the class of Portuguese Language, a specialised context for teaching literacy competencies. This text reports a study about knowledge and skills privileged in literary texts’ reading in national exams. A document analysis of seventeen exams of Portuguese, used by the Ministry of Education between 1996 and 2012, was conducted to identify the “reading objects” chosen for the exams, the text structures that were the focus of the questions and the reading operations requested. The results show that besides the central role of the national literary canon in the assessment, contemporary poetry texts are the preferred “reading objects”. This allows the conclusion that questions about literary contents aim to evaluate student’s critical thinking regarding works by canonical authors. The study also showed that reading literature is perceived as the act of understanding and (re)constructing the meanings of a text by an active and competent reader.


Introduction
In recent years, the teaching of Portuguese has been at the centre of a searing debate that has provoked a dynamics of rupture with syllabi, grammatical terminologies and the textbooks.
On the other hand, the teaching of literature in language classes is usually viewed as constituting a rather pacific matter.Nevertheless, this argument cannot presently be sustained in relation to its present status within the subject of Portuguese Language.Under this context, the overhauling that this subject has undergone as a consequence of the Curricular Revision of 2001 allows us to acknowledge that the new syllabus (Coelho, 2002) reconsiders the position of literature in the instruction of secondary school pupils and encourages the aperture of Portuguese language classes to a range of plural discourses.
The new syllabus also recommends that the evaluation of learning be considered in relation to the different stages of teaching, recurring to (in)formal procedures adequate to the object under assessment.Additionally, in the last year of the 3 years that make up Secondary Education, the Ministry of Education and Science will carry out an External Summative Assessment on all the learning of students.
The multiplicity of goals and contents of the subject of Portuguese Language and the social construct of the subject as a site for the acquisition of literacy contribute, on the whole, to the peculiarity of its process of evaluation.As a conclusion, and as Castro and Sousa (1998, p. 47) sustain, reading in school contexts is almost invariably regulated by the principles of evaluation.Thus, taking into account both the novelty and the relevance that evaluation assumes in the school system and the insufficiency of empirical studies in this field (Fernandes, 2007, p. 126), we consider it to be of the utmost importance to study the way in which the designers of the Secondary Education Curricula have materialized External Summative Assessment (ESA) of learning for 12 th year students regarding the subject of literature.
We therefore selected to study ESA in relation to the learning made by students with the aim of trying to identify what type of knowledge and skills were essential for the National Examinations in Portuguese language.To accomplish this, we carried out an analysis of the question sheets of the 12 th year (Portuguese B and Portuguese) with the aim of detecting the Pereira da Costa & Carvalho da Silva Bellaterra Journal of Teaching & Learning Language & Literature. 8.3 (Aug-Sept 2015) ISSN  'official perspective' on the evaluation, on the reader, on literary reading and on the teaching underlying those examinations from 1996 to 2012.
To that effect, we have structured this article into four essential points.Following the introduction, we synthesize the views of several specialists on the problematics associated with the evaluation of learning and we discuss the matter of literary reading.Next, we define the criteria for the creation of a corpus and specify different options regarding methodology.
In the central section of this text, we introduce the results and in the last section, we present a discussion of the results followed by conclusions that can be drawn.
The main objectives of the study described in this article are the following: • To characterize the knowledge and the skills required in the domain of literary reading for 12 th year examinations.
• To verify to what extent the question sheets for Portuguese B / Portuguese evaluate objective contents that structure the syllabus for this subject.
• To make explicit the different conceptions on reading and reader which underlie the examinations.
Taking into account these objectives, we adopted six analytical dimensions in the question sheets as the guiding principles.We will describe these in more detail in the section on the methodological framework for this study.

Sociopolitical Background: The 'Evaluation Era'
In this section, we will discuss different issues related to the conceptualization of learning skills evaluation; we will then proceed to tackle the problematics of literary reading, in the subject of native language.A quite recent exponential growth of evaluation activities, in almost all aspects of human interaction, for the sake of transparency, accountability, planning, and for the sake of everything else at stake, allows us to affirm that we have entered the 'evaluation era' (Broadfoot & Black, 2004, p. 19).
As far as research on assessment of students is concerned, evaluation has come to be seen as a necessity in order to claim that learning has been successfully attained.This 'evaluation revolution' has been translated into the quantitative gathering of data that legitimizes comparisons and verdicts.Inevitably, in order to accomplish this, external evaluation necessarily implies a uniformity of contents and procedures.In Portugal, the National Examination complies with those guiding principles and takes up the task of certifying the learning of students and of selecting those aiming at a Higher Education; at the same time it monitorizes the performance of both schools and teachers.
Alien to the process of teaching, ESA is felt as an intimidation by many teachers.A defensive reaction on the part of teachers is translated into the organization and subordination of activities to the attainment of the goals required by the examinations designed by outsiders.
The tradition set by past examinations, ends up outlining a de facto syllabus.In this way, it is submitted to control what Cardoso (1999, p. 82), identified as "what is taught, how it is taught, what is learned and how it is learned".As a conclusion we may say that the National Exams puts pressure on teaching, learning and, also, internal evaluation (Natriello, 2009).

Modalities of evaluation
The initial conception of Formative Assessment applied to the learning of students is already quite distant from more recent conceptions (broader and of cognitive and constructivist roots) that have been propounded, such as Formative Assessment (Nunziati, 1990), Authentic Assessment (Wiggins, 1998), Alternative Assessment (Gardner, 2006) and Alternative Formative Assessment (Harlen, 2006;Fernandes, 2005).
Of an acknowledged difficult application, Formative Assessment, however, constitutes one type of assessment that stresses procedures without disregarding results, it is both continuous and interactive, and implies the participation of students.The means used for collecting evidence for learning have as their main objective that of 'steering' the vast majority of skills.Consequently, feedback is essential to this process of assessment.This conception is, ultimately, in conformity with the key elements of formative evaluation identified by the OECD (2005, p. 44).In contrast, ESA carries out a controlling mission, it classifies, excludes and selects.
Presently the cohabitation of Formative and Summative Assessment is, nevertheless, advocated.This actually raises the question of the unexplored relationship between both and points at the need for clarification of the borders that separate them (Harlen, 2006) features until it reaches the other extreme where the evaluation of learning is located, Formal Summative, criterial, which constitutes the sole responsibility of the teacher or of an external evaluator.Formal Formative and Informal Summative assessments are located in between.
Despite the features that differentiate them, both make use of feedback in order to serve the syllabus and are evidenced in a built-in manner in any conventional classroom work.Ideally, the teachers take both into account.
We feel that the study that follows supports the argument that Alternative Formative Assessment should become prevalent in classroom practices.All the data collected through this process can and must be used by teachers in summative evaluations as a part of their teaching responsibilities.Summative assessment can, in this way, become a richer, brighter and more useful synopsis of the entire learning process (Fernandes, 2007).

The Question of Literary Reading: Some Theoretical Perspectives
In the context of Portugal, and as pointed out in our introduction, the debates over the teaching of native language tend to point to a central topic: that of a crisis in the teaching of literature at schools.This takes place due to the fact that the social practice we denominate as 'Teaching Portuguese' is submitted to cross currents motivated by the diverse perspectives under which it is addressed.These perspectives have been described according to four essential paradigms: (i) of transmission of cultural inheritance; (ii) of 'personal growth'; (iii) of the necessities of adult life; (iv) and last, of 'cultural analysis' (Poulson, 2005).
Under these circumstances the differences regarding the contents and the organizing principles of the Portuguese Language Syllabus for Secondary Education (Coelho, 2002) have unleashed a movement that questions some options, namely those that are related to the role and functions of canonical literature.In such a mediatised debate, some see in the reconfiguration of the syllabus for Portuguese Language an inadmissible reduction of pupil contact with literature.It has been perceived as reduction that seemingly overlooks the concern with literary reading, with the development of the pleasure for reading and with the formation of good readers.This is a concern that is, nevertheless, addressed in the Syllabus under the headings 'Ends' and 'Objectives' which have served as guidelines for course curricula (Coelho, 2002).In order to reach that end, the Syllabus recommends a diversified set of texts (e.g.informative, autobiographical, argumentative and, also, texts selected from the mass media) to be read by a reader who is "active, able to select information and to formulate, confirm or correct hypotheses and capable of building sense" (Idem: p. 20).
According to this new understanding, the reading of literary texts should aim at the development of skills such as those of "comprehension and interpretation of texts […] where aesthetic and rhetorical effects prevail" and at the discovery by a student / reader "of the relevance of literary language in the exploration of the potentialities of language" so as to attain an "enlargement of a perceptive understanding on the world" (Coelho, 2002, p. 7).
Aguiar e Silva (1982), in turn, has always sustained that the teaching of literature cannot be fully conceived out of the framework of language and that language cannot be studied disregarding the study of literary texts since a literary text is an "elaborated structure, a sophisticated vocabulary, and a set of subtle authorial intentions" (Bauerlein, 2011, cited by Buescu, 2014, p. 7).
Literature, moreover, constitutes a medium or a type of knowledge that carries out a distinct role in the process of educational and cultural formation of the young (Aguiar e Silva, 1982) and, consequently, a school for the masses will have to provide access to that symbolic capital for all students (Aguiar e Silva, 1998Silva, -1999, pp. 24-25), pp. 24-25).The interpretation of literary texts plays a relevant role on a personal and civic level, making it possible to live with others and to understand their values and backgrounds and the experiences that have moulded human communities over time (Bernardes, 2005, Colomer, 2009, Bernardes & Mateus, 2013).
Colomer ascribes to literature an educational power attributing to it the possibility of a differentiated look over the world and, consequently, values the selection of works that may awaken a desire for the sharing of interpretations.In order to do that, the school has to dedicate sufficient time to its practice (Colomer, 2007, p. 125).
Furthermore, reading and the interaction with any literary introductory texts of renowned literary quality for children and youths, in which the semiotic qualities of literature are already present, consolidate literary competence and its main components (Azevedo, 2005), amongst which we may cite not just plain reading skills but also intertextual reading skills (Fillola, 2010).
Focusing on the readings that children and youth, as 'literature consumers' make use of outside of the school circuit, Lluch (2010, pp. 105-106) points out that "reading has changed places", and that the books being read were pointedly devised in order to "please a young reader who has grown up with computer game audio-visual stories or with stories originally created for television series".The nature of those books may be considered closer to what may be considered as para-literary and can compromise the consolidation of literary skills in young adults (Fillola, 2004).
Taking the above perspectives into account and reflecting an ample consensus on the benefits of reading for any individual, the Syllabus for Secondary Education recommends a canon of writers / works of what can be termed as 'classics' as well as a varied corpus for individual recreational reading.

Objectives
This study aims to identify the knowledge, skills and competencies that the 12 th year Portuguese Language National Examinations address and intend to evaluate with regards to literary reading.The data was collected by Costa (2012)

Sample
The object of analysis selected for this study was the 12 th year written Portuguese National Examination set by the Ministry of Education and Science as external assessment.
Accordingly, we sought to compile a body that was relevant to and representative for both our research object and objectives.The study therefore focused on the question sheets of seventeen exam papers.The first ten of these fell under the Portuguese B Syllabus and the rest under the Portuguese Syllabus (Coelho, 2002).

Instruments
In order to uphold the aforementioned research object and objectives, we undertook a documentary analysis of the question sheets, with the aim of discerning the significance and consequences of their content using a qualitative approach.Exam papers for Portuguese A were out of the scope of our research because it is a literary subject with a different syllabus,

Procedure
With the exam analysis, we aimed at identifying elements that reveal specific ways of reading a literary text.To achieve this end, and due to the fact that our research is limited to the assessment of the learning of reading skills -understanding/interpreting literary texts/literature -, we conducted a content analysis, in line with the Bardin's approach (2004).
We read the exam samples and, in order to carry out a more precise and structured investigation, we defined a set of dimensions and categories, which became apparent a posteriori.
We began by categorising the exam papers, assigning each of them two letters, NE (National Exam), followed by the year in which it was administered.Accordingly, the exams are, henceforth, referred to as NE 1996, NE 1997 and so on, up to the 2012 exam.We then established seven analysis criteria, namely: (i) structure of the question sheets; (ii) role played by reading in the exam and in the assessment framework; (iii) reading objects selected by the evaluators; (iv) textual structures focused on by questions; (v) reading operations required by the items of the questionnaire; and finally, (vi) requests that may elicit a demonstration of declarative knowledge on literary contents.
The categorisation of each of these aspects is shown in the results section.In order to better explain our findings the results are presented in tables and graphs with a brief explanation of each.

Structure of the question sheets and fields of assessment
The first results on this item of analysis, and classified at the onset as 'two-part structure' and "three-part" structure, indicated that the preferred structure was always a three part one.In the case of Portuguese B, the three-part structure reveals two core areas of the subject.Group I and Group II assess the competence of literary reading and Group III addresses the assessment of writing skills.
In Portuguese, always of a three part structure, the assessment of literary reading proficiency is invariably present, in the first section of the exam.The middle section assesses non-literary reading skills and also explicit language knowledge.The last section focuses on the assessment of autonomous writing skills.

Framework and importance of reading in the exam and in the assessment agenda
The number of items included in the Portuguese B exams and their respective scores underline the value that the evaluator places on the assessment of literary reading.In fact, from NE 1998 to NE 2005, six questions (Groups I and II) out of seven questions corresponded 70%, or 140 points of the 200 total score assigned to the written exam.Only 30% were reserved for writing.
For the Portuguese exam, 70 points were attributed (35% of the total) to the four questions that dealt with interpretation of a literary text, and 30 points (15%) to the question centred upon demonstration of declarative knowledge of literary texts / literary works.
The high concentration of questions in the first and second sections of the exams for Portuguese B attests to the privileged position of literary reading as compared to writing, which gets only one question.scholarly opinion which claims that the knowledge on the structure of the mother tongue is transversal to reading and writing skills and to oral comprehension and expression to which it effectively aids and develops (Silva, 2008).

Characterisation of reading objects -texts
With the aim of analysing the texts selected by the evaluators for the literary reading section, we began by defining the categories of text fragment and full text.
We observed that the text fragment (20%), much less frequent than the full text (80%) in Portuguese B exams, was always taken from the canonical texts included in the listing of recommended works of the Syllabus.We also noted that 'textual clippings' always safeguarded their unity, coherence and cohesion.We can also conclude that pupils could always deduce the answers in the text fragment without having to consider any literary work as a whole.
The Portuguese exams show different data.In these, text fragments were the most selected item, 71% of all cases.Texts were taken from the epic poem Os Lusíadas (Luís de Camões), from the dramatical narrative Felizmente Há Luar! (Luís de Sttau Monteiro) and from the novel Memorial do Convento (José Saramago).The full text (29%) was selected to evaluate the reading of poetry by Fernando Pessoa and his heteronyms.

Types of text selected
Regarding the types of text chosen to evaluate the reading competence of pupils, and which were considered as appropriate to achieve the goal training proficient readers and of evaluating capabilities, skills and attitudes, as outlined in the Syllabus for this subject, it was found that poetry was chosen as reading object in 70% of the exams for Portuguese B and that for Portuguese there was a predominance of epic texts and poetical texts in 43% of the exams.
Finally, and ex-aequo, we have narrative and descriptive texts and drama.

Requested reading operations
Aiming to examine the cognitive operations that the student/reader is expected to perform in order to understand a literary text, we adopted the taxonomy determined by Dionísio (2000).
Following Eco (1983) and Schank & Lebowitz (1980), who sustain that Identification and Inference are to be considered as underlying actions to the multiple reading operations, Dionísio adopts these categories as fundamental to the reading of any literary text.Following even Tollefson (1989), Dionísio defined seven categories that guided this section of the research.
Considering the different interpretative movements that may be requested from pupils, Dionísio (2000) identifies other categories beyond the categories of Identification -present in Analysis of the questions shows that Inference and Identification are more present than other categories and, consequently, are attributed higher scores in both the older exams (in Portuguese B, 48 % and 26%) -and the more recent ones (Portuguese, with 44% and 17%).
We may, therefore, deduce that for the evaluators a pupil will only show reading competence in relation to a literary text whenever s/he is able to (re)construct meanings upon extracting the information explicitly present in it and in the meanings to be inferred between the lines.
The extra demand for the cognitive operation of Justification, in Portuguese (14%), certainly means that the evaluator requires from the pupil an in-situ focus, requesting that the text be read 'from the inside out' and that the paths selected for comprehension be explained.
Mobilization of knowledge plays a relevant role evidencing that the authors of the Syllabus consider that reading comprehension evaluation must demand student input in the construction of the meanings of any text.

Description of writing "requests" to demonstrate knowledge
The analysis of this item began with the establishment of five categories, namely: (i) type of text, (ii) author targeted, (iii) theme, (iv) word limit and, lastly, (v) assigned score.
Portuguese B exams required the elaboration of a discussion essay on a theme developed in the works of the authors of the syllabus -Garrett, Antero de Quental, Cesário Mensagem' were used as prompts for the pupils to express their reading opinion in an essay with word extension set between 100 and 200 words.
For Portuguese, the tendency was to write a descriptive 80 to 130 word essay, in which pupils, based on their reading experience, expressed their interpretations and critical judgments of the key themes in the canonical works of Camões, Pessoa, Sttau Monteiro or Saramago.

Discussion and Conclusions
The main objective of this study was that of characterizing the capacities, skills and knowledge in the domain of literary reading as evaluated in the National Examinations for Portuguese Language in Secondary Education.Our results corroborate those found by Sousa and Castro (1989) in that evaluation in areas of reading and writing is a constant paradigm for both the exams of Portuguese B and Portuguese.Furthermore, we observed in relation to the structure of the exams that even when the evaluator may show a clear preference for a threepart structure, only in Portuguese this three-part structure coincides with the areas that can be evaluated in an exam: reading, writing and grammar.
An analysis centred upon the status of the literary text in the exams, on the number of questions about interpretation and on the scores attributed allows us to affirm that the reading of literature is a domain in the teaching of language that occupies a privileged position in the examination.We may confirm, in this way, that the teaching of the native language is inseparable from the teaching and learning of literature and from the corresponding knowledge associated to it (Portuguese B).In Portuguese, the evaluation and the presence of different 'types' of texts and the reflection over language denote a repositioning of language and of literature.
Poetry texts were the type of text most often chosen by evaluators for Portuguese B exams.In addition to confirming the hegemony of literary texts, these results allow us to conclude that, between 1996 and 2005, the type of reading stressed is (exclusively) that of literature.Portuguese exam papers, regardless of their stress on the literary (of epic, poetic, narrative and dramatic nature) reserve a meaningful place for non-literary texts and, in this manner, stress the presence of plural discourses in the exam.Those options certainly mean that the proficient reader is also an individual able to understand several types of texts.The privileged position granted to poetry texts denounces the relevance provided by the designers of the curricula to authors that the syllabus ends up canonizing for the instruction of fluent readers.It is important to stress the cases of Camões and Pessoa, poets that may be considered as having been 'schooled' once they became part of the currricula and were made as 'literary examples'.
From the analysis of the cognitive operations, elicited in the exam questions, the conclusions that we draw are of didactic-pedagogical relevance.At the same time that we acknowledge the role played by Inference, Identification, Mobilization and Justification in Portuguese B and Portuguese, it is appropriate to conclude that, for the evaluator, the reading of a literary text is based upon mental operations of a high degree of complexity carried out by a dynamic pupil.
From the analysis of categories centred upon writing for the expression of declarative knowledge on the literary works, it is possible to draw conclusions over the conceptions about the teaching of literature and on the evaluation of its learning.In fact, when requesting the opinion of a pupil on an aspect of a work, this question, oftentimes, creates an opportunity for the expression of a personal, appreciative reading of the literary texts.This question shows, moreover, a cultural and humanistic calling for the subject of Portuguese B, which originates in the relative importance attributed to literary metalanguage and to the centrality conferred to a literary text as an object of study and as representative text for a specific aesthetic movement.The Portuguese examination, in turn, when demanding a critical commentary, requires complex literary operations since the pupil has to elaborate a more personal reading and criticism by explaining a position on the texts.
Based on the collected data and attempting to provide a response, albeit brief, to the key initial question and the objectives, our research concluded that in the Secondary Education National Exams: (i) those responsible for the curriculum value the knowledge, expertise and skills acquired by students in the subject of Portuguese B/Portuguese, which were developed through reading texts and, in particular, reading literary texts/literature -a clearly preferred field of teaching-learning the native language and assessment; (ii) for the most part, the preferred reading objects are full texts that represent different types of text, yet the poetic text is prevalent; Pereira da Costa & Carvalho da Silva Bellaterra Journal of Teaching & Learning Language & Literature. 8.3 (Aug-Sept 2015) ISSN 2013-6196 (iii) students/readers are required to perform reading operations with varying degrees of complexity, such as inference, mobilization, justification, synthesis and to make value judgments, which comprise a significant number of «rich» interpretation operations; (iv) the critical and reasoned opinion of students with regard to literary texts and their reaction to reading, expressed in relevant judgments and commentaries, have been consistently valued; (v) understanding the explicit meanings of the texts is also an assessment object, evidenced by the relatively high occurrence of the identification reading operation; (vi) the value attributed to the «voice», to the active role of the students/readers, the development of their ability to discover the hidden meanings of the text and the expression of critical thinking, as demonstrated by questions that elicit inference and value judgments, allow us to conclude that literary reading competence is not seen as the ability to assign unique meanings to literary texts and that these are not perceived as «transparent texts»; (vii) literary texts and the activities fostered by their presence as well as reading have a noteworthy centrality in the exam, which leads us to speculate that they have an equivalent status in the classroom; (viii) demonstration of knowledge about the literary texts selected for exams (categories, diegetic facts, formal components, excessive literary metalanguage) and a formalstructural type of analysis are not favoured in the assessment papers; (ix) replication of content conveyed in the classroom, of «ready-made» readings, is not assessed within the national exam; (x) neither a diachronic perspective of literature, nor the demonstration of knowledge about the text are particularly valued, as confirmed by the small number of questions that elicit classification; (xi) this notion of the reader as a builder of the meanings of a (literary) text also brings about the idea that the work's contextual data, the historical and cultural information and facts related to the author's life are not an end in themselves, but rather a foundation that enables and facilitates the understanding of each text; (xii) the value placed on literary reading as assigning meaning to the text is consistent with the high incidence of questions about semantic-pragmatic structures; from two language subjects: Portuguese (Portuguese Language for Arts, Sciences and Humanities Courses) and Portuguese B (Portuguese Language for Sciences Courses).The exam samples researched refer to a time span between the academic years of 1995/1996 and 2011/2012.
comprising a greater number weekly hours and a reduced number of students (Humanities) between 1991 and 2002.Pereira daCosta & Carvalho da Silva   Bellaterra Journal of Teaching & Learning Language & Literature.8.3 (Aug-Sept 2015)   ISSN 2013-6196After characterising individual written tests, we analysed those specific sections related to the assessment of literary reading.Our expectations were set on the detection of any manifestations of the 'official discourse' on the domain of reading.This study comprises two phases.The first focuses on the exams given out between 1996 and 2005; the second targets those used between 2006 and 2012.This distinction enables the content of the exams to be collated and to show in a clearer way the continuities and discontinuities evidenced by the collected data.
questions of literal comprehension where answers are present in the text -and Inferencewhich allows the reader, based on his prior knowledge, to fill in blank spaces.Those additional categories are Classification, Justification, Mobilization, Synthesis and Value Judgement.Classification is the inclusion of a textual element in a category previously learned; Justification is understood as a meta-process demand focused on explaining the reasons for a particular answer.Mobilization corresponds to the consideration of information already existing prior to any reading activity.Synthesis consist of the interpretation actions which involve «analysis, selection, generalization, and information organization» (Idem, p. 187) and, finally, Value Judgement which allows the reader to submit to evaluation whatever has been read.Pereira daCosta & Carvalho da Silva   Bellaterra Journal of Teaching & Learning Language & Literature.8.3 (Aug-Sept 2015)   ISSN 2013-6196 Verde, Vergílio Ferreira, José Saramago, Sttau Monteiro and Fernando Pessoa.Themes such as 'The symbolic dimension of Aparição and Memorial do Convento' or 'Myths in Pereira da Costa & Carvalho da Silva Pereira daCosta & Carvalho da Silva