Learning Opportunities Through a Student Generated Assessment Tool

The development of learning autonomy and the acquisition of professional discourse in an additional language are current priorities of higher education content and language integrated learning (ICL-HE). Research regarding assessment of these objectives is lacking. This pilot study explores the use of formative and self and peer assessment in the teaching of professional discourse in an ICL-HE context. It examines how student generation and application of an instrument to assess written professional discourse creates a knowledge building environment (KBE) and opportunities for learning and tracks the learning behaviour trajectory of one student. Interaction analysis is applied to audio and video data collected during a professional discourse writing activity. Classroom data provides evidence for the creation of KBE and opportunities for learning. Longitudinal data collected from one student reveals increasing orientation toward the learning object. The student generation of assessment instruments is a viable formative assessment technique in ICL-HE contexts.


Introduction
The acquisition of disciplinary knowledge and the professional discourse through which it is represented is a fundamental part of the process of professionalization which occurs during higher education (Airey & Linder, 2009).Current emphasis on international collaboration and communication has created the additional need for professionals to develop second language competencies and acquire professional discourse not only in L1, but also in an additional language.Multilingualism is seen as a necessary prerequisite for access to the information flux in multilingual international professional communities and an enabling factor for international collaboration ( Fortanet-Gómez, 2013;Moore, 2014).In the institution studied, the focus of integrated content and language learning (ICL-HE) is on the acquisition of professional and academic discourse in an additional language, with corresponding content and linguistic objectives.

Theoretical framework
This pilot study explores assessment in the teaching of professional discourse in an ICL-HE context.The approach taken involves the purposeful identification and planning of learning objectives related to the acquisition of professional discourse in L2 (Llinares, Whittaker & Morton, 2012).Assessment criteria for professional discourse text genres are established through the application of a framework which explores how meaning is created on an ideational, interpersonal and textual level.
Developing learning autonomy is a fundamental goal of higher education.This study thus presents the application of assessment for learning, or formative assessment (Popham, 2008) for the development of autonomy in the learning of professional discourse.As explained by Ribas (2010), in assessment for learning text assessment criteria should reflect clearly defined descriptors along with specific values as needed.In the case under study, establishment of assessment criteria, an integral part of assessment planning, is placed in the hands of students.
Self and peer assessment are also used in this study to increase the learning potential of assessment activities and encourage the development of autonomy.Previous studies (Martínez-Ciprés, 2011) of peer and self-evaluation of written texts in CLIL contexts have involved student familiarization with pre-established content and language-specific criteria previous to their application to the written text in hetero-or auto-evaluation.Marzano (2010) suggested the use of 'translation' of pre-defined criteria to 'student-friendly' language previous to their application in hetero-or auto-evaluation.In this study, assessment criteria are not predetermined.Instead, public student generation of assessment criteria is performed to create an assessment instrument.This instrument is then applied in the peer assessment of written texts.
This study draws upon Second Language Acquisition research which applies the concepts of dynamic assessment (Lantolf & Poehner, 2008;Leung, 2007) and 'scaffolding'.
Instances of corrective feedback for the scaffolding of language learning (Lyster & Ranta, 1997) are identified.Readers are referred to previously published articles by Sauro (2009) for a more complete description of Lyster and Ranta's categories of corrective feedback.
Scaffolding in this study is also defined as the use of material and conceptual artifacts to support learning.The presence of the class-generated evaluation tool during peer assessment encourages student orientation toward criteria that might have otherwise been forgotten.
The concepts of knowledge building environment (KBE) and opportunities for learning are also applied in this study.The concept of knowledge building environment is adapted from Brouwer and Wagner's (2007) application of community of practice and the process-based concept of knowledge building community (Hoadley, 2012).For the purposes of this study a KBE is defined as an intentionally created environment in which the goal of creation of knowledge is explicit.Knowledge is not only shared among members of the classroom community, but extended.In addition, agency is a key driver in the knowledge building process.That is to say, the members of the KBE must have control over the process of knowledge construction.
In line with other authors (Brouwer &Wagner, 2007;Doehler, 2010;Moore, 2014) learning in this study is conceptualized as a dynamic social process, situated in communities of practice, and which requires a longitudinal perspective.Opportunities for learning in this study are identified as interactional phenomena which focus on content and linguistic aspects of a specific learning object during the social process of the class generation and application of an assessment instrument.Situated analysis of interactional phenomena looks at instances of teacher-initiated repair (mediation) focusing on linguistic structures (instructional feedback) during the class generation of the assessment instrument as potential acquisition sequences (De Pietro, Matthey & Py, 1989).These moments of repair or corrective feedback (Lyster & Ranta, 1997) allow for clarification and extension of student generated knowledge regarding linguistic aspects of the text under study.
Addressing the need to reflect the temporal aspect of learning, changes in learning behavior over time (Learning Behavior Tracking) are explored with respect to one individual and one specific learning object.This approach is based upon the concept of learning behavior tracking (LBT) (Markee, 2007), comprised of learning object tracking (LOT) and learning process tracking (LPT).In the present study LOT involved identifying interactional fragments where a specific learning object was deployed by a participant.Additional analysis was performed to identify discrete moments when the learning object is not deployed to strengthen the evidence toward the longitudinal aspect of learning.LPT, as applied by Markee (2007, p. 409), involves: "[...] developing conversation analyses of how and when participants orient to, and potentially incorporate, particular learning objects that occur in different speech events in their interactional repertoires."In this study LPT involved interactional analysis of video and audio recordings to identify how (in what way) and when (in what interactional contexts) one student oriented toward the learning object at discrete points of time throughout the process and incorporated it into her interactional repertoires.
Because the current study tracks learning in an integrated content and language learning (ICL) setting, the learning object chosen, the 'objective' section of a scientific abstract, includes both content and linguistic aspects.The 'objective' section of a scientific abstract was chosen as a learning object because it is a concept to which the student herself orients during the class elaboration of the assessment instrument.
Previous research in higher education CLIL settings has focused on academic discourse in language learning and assessment in multilingual CLIL classrooms (Moore, Nussbaum & Ploettner Bellaterra Journal of Teaching & Learning Language & Literature. 8.4 (Nov-Dec 2015) ISSN 2013-6196 Borràs, 2013;Moore, 2014).A lack of research exploring assessment in ICL-HE, particularly the learning potential of student-controlled assessment in the teaching and learning of professional discourse, exists.This study attempts to fill this gap, exploring how class generation and application of a content and language specific assessment instrument creates a KBE and learning opportunities while tracking longitudinally the learning behaviors of one student.

The study context
This exploratory study takes place at a small private university in Catalonia, Spain.Courses At the beginning of the project, the basic concept of survey studies in general was introduced through class discussion and writing in pairs.Data collected included audio recordings (smartphone recording performed by students) and written texts taken from the initial session.This was followed by a series of sessions dedicated to student familiarization with the genre through the analysis of authentic health science abstracts for structure, content, and linguistic aspects.The students also designed and carried out in small group (4 students) a survey study, which they presented in both written and oral format.
After writing their abstracts, students were informed that they would be evaluating their own written abstracts, an activity which required the formulation of appropriate assessment criteria.During this session, a blank word document was projected on a classroom screen for editing during the class discussion and consensus of assessment criteria.After eliciting suggestions for general criteria from the entire class-group, the teacher modeled the entry of class-determined criteria on the screen.Small groups were then given the task of developing criteria for each abstract section.The small groups subsequently presented their criteria to the open class for discussion and consensus; with each group typing in the class criteria for their section on the screen.The final document was then saved and photocopies made by the teacher to be used in the assessment activity in the following session.Data was collected in this session through video recording of the classroom interaction.
In the third session student pairs applied this class-generated assessment instrument to assess their own and their classmates' abstracts.Data collection included video and audio recordings by smartphone of the student pairs during this process.

Data collection, treatment, analysis and research ethics
Students were informed that they would be participating in a research project in education, of the students during the small group discussion.The mobile phone recordings were attached to emails and sent directly to the teacher's email address while the students were still in the classroom.The audio recordings were then converted to a standardized format using Format Factory software and saved in an external hard drive for future analysis and selection.Video recordings were similarly downloaded and transformed to a standardized format and stored in an external hard drive for viewing purposes.All audio and video recordings were viewed initially by the researcher and initial content analysis was recorded in field notes and table format to aid in data selection.Selected audio and video recordings were transcribed by the researcher following the system presented in annex 1.
Interaction analysis explored instances of teacher scaffolding (clarification, reformulation of concepts) during class generation of the self-assessment instrument.In addition, data collected during the co-construction of the assessment instrument was examined for evidence of criteria for knowledge-building environment.
Longitudinal tracking of one student's learning behavior was performed using interaction analysis to identify the two components of learning behavior tracking (LBT): learning object tracking (LOT) and learning process tracking (LPT), discussed above.
Video and audio data, transcriptions, and analysis were revised by the researcher in collaboration with informed colleagues (language teaching experts who also teach content and language integrated subjects within the same institution) in monthly informal data sessions,

Opportunities for learning/ knowledge building environments
Identification of opportunities for learning was performed through analysis of classroom discourse which focused on instances of instructional feedback during the generation of the assessment instrument.Teacher reformulation and clarification can be seen in the following fragment, in which student A speaks to the open class on behalf of her small group.In the fragment she presents the assessment criteria for the objective section of a survey abstract while one of her colleagues types the agreed-upon criteria on the screen visible to all the class: Fragment 1 1 Student A: they have to be structured\ (.) Here, teacher mediation (other-initiated repair) in the form of clarification (line 3) and reformulation (line 7) is observed.The pause (line 1), repetition of the utterance (line 2), and the rising intonation (line 2), all seem to indicate that student A doubts the veracity of her contribution and could be interpreted as a request for assistance.The teacher repetition of the statement (line 3) initiates a repair sequence in which the word "objectives" is stressed.The student responds with a repetition of the utterance which is finally terminated by the teacher's contribution of the term "presented" (line 5), which the student ratifies in line 6.The sequence ends with a final reformulation by the teacher of the original contribution to "clearly stated".
The incorporation of this instructional feedback during the creation of the instrument allows for further extension and development of initial student affirmation regarding the content of the objective section.
Another instance occurs in Fragments 2 and 3 when the teacher detects that the group assigned to establish criteria for the objective section has failed to mention previously studied linguistic criteria for their assigned section.(line 6) An example of collaborative extension of knowledge occurs when other class members contribute to the elaboration of the missing linguistic criteria.(line 10) .Immediately before this fragment student A had presented her group's assessment criteria for the objective section.The teacher had repeated these verbally and asked for student confirmation of these criteria.we propose to Teacher: right\ (.) we (.) we proposed to (.) this study aimed to (.) do you remember we talked about these things/ (.) ok \ (.) so\ (.) this could be ((reads from the screen)) aim of the objective (.) to give the way of the work (.) don't have many objectives (.) as far as language goes (.) you could have (.) ((as if she was reading a text)) the purpose of this study was to (.) like student B said\ (.) do you remember we talked about the infinitive of purpose/ (.) yeah\ (.) you could write that there too as well\ (.) the infinitive of purpose or the purpose of this study was to The data presented here again indicate teacher scaffolding in the form of clarification (lines 6 -9) and remodeling and extension of student ideas (line 10) during the public generation of the assessment instrument has provided opportunities for expansion of students' current orientation toward the learning object during the classroom interaction.
The fragments presented here provide an image of a process in which extensive teacher scaffolding occurs.They also provide evidence that the process in this case complies with some characteristics of a knowledge building environment, as defined previously.They represent an intentionally created environment in which the goal of creation of knowledge is explicit.Knowledge is not only shared among members of the class, but also extended by them.In addition, agency is a key driver in this knowledge building process, which delegates responsibility for the construction of knowledge to the students

Tracking learning longitudinally
Learning behavior of one student (Student A) was tracked longitudinally following the deployment of the learning object of objective section content and language.The decision to follow Student A's learning behavior was made on the basis of completeness of data.The learning object chosen to follow throughout the process included both content and linguistic 71 Ploettner Bellaterra Journal of Teaching & Learning Language & Literature. 8.4 (Nov-Dec 2015) ISSN 2013-6196 aspects of the objective section.This learning object was chosen because the student herself oriented to it in the generation of the assessment instrument.

Data collection point 1: Introduction of the topic of survey studies
In The teacher then projected on the screen the titles of several recent surveys taken from mass media, and class discussion took place regarding the topic of the survey, the population surveyed, how the data was gathered and the results of the survey.
After class discussion on the content of these studies, students were instructed to write in pairs a brief text summarizing one of the survey studies discussed previously.Students were instructed to use their mobile phones to audio record their conversation while writing the text and to send the audio file as an e mail attachment to the teacher's e mail address.
Student A and her partner wrote a description of a survey study previously discussed by the class entitled: "Germany is the Most Popular Country." The transcription of the interaction, not included here due to lack of space, reveals no orientation toward the objective of the study, the 'objective' section content or language.
In the conversation while performing the task, Student A and her partner orient toward the following items: • The topic of the conversation (who they are and which study they are going to write about) • What were the results of the survey?
• Who was surveyed?
• How were the data collected?• Conclusions Neither the conversation nor the written document reflects an orientation toward the concept of the objective of the study or why the study was carried out.There is no statement of the objective of the study or use of lexical items such as purpose, objective, aim or the infinitive of purpose.Language & Literature. 8.4 (Nov-Dec 2015) ISSN 2013-6196

Bellaterra Journal of Teaching & Learning
The written description of the survey study which was elaborated by student A and her partner is presented in Figure 1.The keyboard is passed on to them and the teacher asks them to present their criteria.
A pause is followed by the teacher repeating her request.Student A looks at Student S several times and then he begins their presentation.In this fragment Student S starts out with several false starts, and his utterance finishes on a rising tone of voice (lines 1-2).The combination of false starts and upturned intonation could indicate either linguistic difficulties or doubts as to the content or as to the language he has employed to express what is presumably the function of the objective section: "The aim of the objective".
After the teacher acknowledges his statement with a continuer ("uh huh" in line 3) a pause occurs.Student A's intervention at this point (line 4) orients toward the form or structure of the section: "the objectives have to be structured".As stated previously, a repair sequence follows in which the utterance "the objectives have to be structured" is reformulated by the teacher as "objectives clearly presented" (line 10).
In the following interventions Student S again focuses on the content of the section, which is "to give the way of the work" (line 11), whereas Student A again focuses on the text structure: "then you don't have a lot of objectives because xxxx it's more probably that the work will be…" (lines 13-15).Again, her utterance is completed by the teacher with the word "better" (line 16).Her repetition of the term "better" (line 17) would seem to indicate once again that she accepts the candidate term to complete the idea she is expressing.The teacher then marks the end of the sequence with "ok" and her reformulation of the idea: "so... you want objectives clearly stated and not too many objectives...great." The data indicate that at this point Student A and her group orient toward the content and the form of the 'objective' section, but do not contribute the specific linguistic criteria studied in previous class sessions.
Of particular interest is when the teacher asks the students for language that might be found in the section, it is other students who respond with the appropriate linguistic structures.
Student A's group does not orient to this aspect at this point.
The assessment instrument generated by the class as an outcome of this discussion is presented here:  Students A and G first evaluate an abstract on life satisfaction and behaviour.In the first minute they begin to look at the abstract and observe that basically it does not meet any of the criteria.They then start to assess the objective section.Focus on specific linguistic structures is revealed when she and her partner discuss the meaning of the word "purpose" and in her deployment of the metalinguistic term "infinitive of purpose" immediately after she says: "objectives...the purpose of this study was…to relate…ok… infinitive of purpose...yes" (lines 8-10).Again, the sequencing of her interventions indicates an understanding of the meaning of the term "infinitive of purpose".
The orientation toward linguistic criteria is again revealed when later they evaluate the objective section of a second abstract: Fragment 6 1 Student A: aim\ (.) the purpose of the study was to (.) 2 infinitive of purpose (.) to assess (.) Once again, student A's interventions manifest an orientation toward linguistic aspects of the learning object when she employs the metalinguistic term "infinitive of purpose".Her understanding of the term is reflected by the sequencing of her intervention, in which the use of the term is immediately preceded by her statement of the actual infinitive of purpose: "The purpose of the study was to…" Her repetition of "to assess" is followed by the statement "vale" (meaning "ok") in Spanish, reaffirming the strength of her decision.

Conclusions
The results of this study indicate that the process of student generation and application of an community to create new public knowledge.In the class generation of the document projected on the screen, ideas are made available to the classroom community in a way which allows them to be discussed, interconnected, revised and adapted.This potentially involves "deep" constructivism and encourages intentional learning by placing the responsibility on students for performing the planning and execution of an assessment process.Ideally, the role of the teacher remains that of modeling, scaffolding and facilitating the planning, checking, and revision of texts, with major responsibility for these tasks placed in the hands of the students.
In the public establishment of the criteria, allocation to students of the responsibility for the creation of the assessment instrument, first in small groups and later in open-class discussion, allows for active student participation.Submission of the criteria to class discussion and public writing of the criteria (the process of typing them up on the screen) creates both a temporal and physical space for content and linguistic reflection.The screen provides a public physical space (artifact) which allows for reflection on both content and language.The time spent typing creates a temporal space for reflection.The 'malleable' nature of this public writing, which the students themselves produce, forsees active participation on the part of the students.Public student-produced written production in the presence of the teacher provides a physical and temporal space to allow for teacher remodeling and scaffolding of the process.
The longitudinal data reflect increasing student orientation to a specific learning object.Changes in student A's orientation to genre-specific language and content of the learning object may be due to multiple factors, including scaffolding in the form of instructive feedback during the second session as well as the presence and application of the class generated material artifact when assessing the writings.
The current study has several limitations.Firstly, regarding data collection, video and audio recording of students potentially influenced the interactions, if only by encouraging them to speak in English as opposed to their L1 (Spanish, Catalan or French in this case).The presence of recording devices in the classroom and the fact that students were told that they were being recorded for research purposes may have influenced their willingness to speak spontaneously or present their ideas.Secondly, the fact that the researcher who analyzed the data participated in the process could have influenced the interpretation of the data.As stated, data analysis was discussed in depth in data sessions with peer teaching experts, but this factor cannot be ignored and future analysis should include data collected in similar didactic sequences led by other professionals.Finally, the increasing orientation toward the learning object presented here reflects data drawn from only one student's participation in the process.
Complete data sets taken from fellow classmates need to be examined to explore orientation toward the learning object the process.
The student generation of assessment instruments has disadvantages.Firstly, the process is much more time consuming than providing students with predetermined assessment criteria.Temporization of the didactic sequence thus requires careful planning to allow sufficient time for students to come up with their own criteria and for the classroom discussion of the appropriateness of the suggested criteria.Inappropriate planning can easily lead to teacher domination of the process.Secondly, it challenges traditional student-teacher roles.The teacher must relinquish control over the process and be prepared to discuss and accept criteria which he/she had not contemplated.Lack of active participation on part of the students is also very possible.Aside from allowing students the necessary time for the activities, active student participation may also be encouraged by including it in the final course mark.Finally, in this study the process was applied in a class with relatively low number of students (25), Larger groups may require innovative adaptations such as subdivision of the class into multiple work groups or innovative digital solutions including the use of forums or platforms for collaborative work.
This pilot experience indicates that the student generation and application of assessment instruments is feasible and can potentially produce positive results regarding the creation of learning opportunities for professional discourse.Future application of this research should be aimed at maximizing student control of the process, while maintaining the role of teacher scaffolding as necessary.
This study contributes to the growing body of "grassroots" ICL-HE classroom research on assessment.It also contributes to the growing body of research evidence which supports the use of conversation-analysis-inspired interaction analysis to longitudinally track learning behavior.Future studies along the same line should be oriented to tracking learning behavior over longer periods of time or exploring data from the same data set to observe if similar longitudinal changes in learning behavior occurred for other students as well.The class generation of content and language-specific assessment instruments has also been piloted by the author in other classes and for other project-related activities (peer evaluation of student generated power point slides for presentations).Future studies could continue along this line.
are designed and taught by language experts with in-depth knowledge of professional discourse in health sciences and, specifically, the teaching of professionally related content and discourse in L2.The subject, entitled 'Basic Concepts in Non-Experimental Research', is taught in the second year of a Health Science program by the researcher, an ICL-HE specialist.The learning objectives of the subject include the ability to read and produce professionally-related discourse in the form of a scientific abstract of a non-experimental study in the health sciences.The scientific abstract is a concise text which summarizes the contents of a research article.Its purpose is to briefly present to the reader (usually a member of the professional community) a summary of a research study.Scientific abstracts are published on professionally related databases which are consulted by professionals when planning research or updating clinical practice.Health professionals must, therefore, be familiar with the genre in order to carry out their professional obligations.The scientific abstract classically includes five sections, each of which has a specific function carried out by characteristic lexical and grammatical structures.This study focuses on the orientation toward the functional, lexical and grammatical aspects of one of the sections -the 'objective' section -which has the function of communicating the purpose(s) of the research study presented.This function is performed through the use of typical linguistic structures such as: "In this study we sought to...", "the aim of this study was to..." Typical structures in the objective section of the abstract include the infinitive of purpose.Teaching/learning process and data collection Data presented were collected at 3 different points in a project-based learning experience about survey-based research in the health sciences.The class included 25 students and 2-hour sessions were held twice weekly during one semester (60 contact hours).Ploettner Bellaterra Journal ofTeaching & Learning Language & Literature.8.4 (Nov-Dec 2015)   ISSN 2013-6196 and for research purposes they would be filmed and audio-recorded at different points during the didactic unit.Voluntary written informed consent for video and audio recording of classroom interactions and the use of written work for analysis was obtained from the students at the beginning of the semester.A total of approximately 4 hours of audio and video recordings of classroom interactions were collected throughout the semester.Video recordings were made with one or two cameras positioned at the front and the back of the classroom during classroom interactions or focused on individual student pairs during pair work activities.Audio recordings of pair work activities were performed through student selfrecording by mobile phone or by personal digital voice recorders placed on the table in front Ploettner Bellaterra Journal ofTeaching & Learning Language & Literature.8.4 (Nov-Dec 2015)   ISSN 2013-6196

Fragment 2 1
Teacher: So (.) you want objectives clearly stated 2 and not too many objectives\ (.)Great\(.) 3 Teacher: ((directs comment to the class)) Do you 4 agree/ What do you think/ 5 (no verbal response) After the lack of student response the teacher refers to linguistic criteria.Teacher mediation again becomes important for the identification of criteria in the following fragment.(what kind of language would you expect to 70 Ploettner Bellaterra Journal of Teaching & Learning Language & Literature.8.4 (Nov-Dec 2015) ISSN 2013-6196 see in the objective/ (.) what's a typical (.) what are typical languages you are going to see in the objective/ (.) Student B: an initial class session the general topic of survey studies was introduced in open teacherfronted class discussion.The teacher began the class discussion by asking the students what a survey study was and by projecting on the screen a document which said: "What is a survey?Give some examples."

Figure 1 :
Figure 1: Student A's written description of survey studied

Figure 2 :
Figure 2: Class generated assessment instrument assessment instrument can potentially create a knowledge building environment and provide opportunities for learning in the ICL-HE context.Learning opportunities or potential acquisition sequences during the public generation of the assessment instrument are exemplified in instances of teacher scaffolding in the form of clarification, remodeling and extension of student ideas.Dynamic assessment and teacher mediation allows for extension of community knowledge reflected in the physical artifact of the class generated assessment instrument.Data collected during the student application of the assessment tool also reflect opportunities for learning.Students are made responsible for the generation and application of evaluation criteria for a specific text genre, with the objective of increasing awareness of the text register and genre requirements, as well as increasing self-regulatory skills.The process goes beyond the mere sharing of knowledge and requires a collaborative effort on part of the classroom Ploettner Bellaterra Journal ofTeaching & Learning Language & Literature.8.4 (Nov-Dec 2015)   ISSN 2013-6196

PloettnerBellaterra
Journal of Teaching & Learning Language & Literature.8.4 (Nov-Dec 2015)   ISSN She and her partner orient toward content (aim of the study) when she says as she reads: "was to relate...ok...satisfaction with several different health habits and behaviors of physical therapy...aim of study...yes... no" Her decision that the aim (the content) is stated in the text is immediately preceded by her statement of the content of the section.