Utvikling av tekstvurderingskompetanse. Lennart Jølle Send innspill til:

Transkript

1 Utvikling av tekstvurderingskompetanse Lennart Jølle Send innspill til:

2 Kontekst Utvalgsprøver Vurdererpanel

3 Skrivevurderingsforskinga Produkt (evalueringsforsking) Nøyaktighet Konsistens Prosess (skriveforsking) Konstrukt modell, skala, oppgavetype Atferd Kognitive prosesser, bakgrunn, verdier, trening

4 Vurdereratferd Avhandlingsarbeidet er knytta til det prosessuelle, til vurdereratferd. Ingen ambisjon om å se «behind the curtain» Vurderingsdialogen som studieobjekt

5 Målsetting Kunne si noe om hvordan lærere som skoleres i å vurdere elevtekster, sammen tar avgjørelser om tekstkvalitet. Kunne si noe om hvordan denne vurderingspraksisen utvikler seg over tid (innenfor et praksisfellesskap). Kunne si noe om a) ulike (vurderings)praksistyper og b) hvordan disse er i spill under vurderingsarbeidet.

6 Materiale Transkripsjoner av lydopptak av vurderingsdialoger Samlet inn ila ett år, 3 vurderingspanelsamlinger 5 hovedvurderere (men bidrag fra 28 vurderere) 26 vurderingsdialoger (med 1-3 tekstvurderinger pr dialog)

7 Undersøkelse 1 Hvordan vurderer uerfarne vurderere elevers skrivearbeid innenfor et vurderingspanel, og hva skjer med denne praksisen over 8d når vurdererne blir mer erfarne og kompetente? Hvilke potensielle effekter har endring i vurderingspraksis på kvaliteten av vurderingene?

8 Table 1. Referents used by the raters when assessing writing within the National Assessment Panel: Familiar assessment practice prior to NAP membership (Known referents prior to joining NAP) Categories Definitions Examples from data A. Reference to text Referring to text in order to make a judgment She doesn t mention herself, she just talks, she talks on behalf B. Citation of text Citing text in order to make a judgment I don t think it s appropriate to Dear Tor, right. C. First scoring Referring to a score made after a first reading before the pair/group assessment D. Comparing Comparing texts within the same corpus E. Text know-ledge and classroom practice Referring to text knowledge and own classroom practice Yes, I have marked it as expected [M3], but that s fine really [to give M4]. You re not supposed to compare, but if you look at the other one here But a typical introduction when writing an argumentative text is to make a claim. New NAP-related assessment practice (NAPrelated referents) F. Meta-discussion Discussing different aspects of the scoring task, e.g. the quality of the assessment tools G. Plenary discussion Referring to statements from plenary discussions H. Guidelines Referring to the normative document Guidelines I. Expert Referring to one of the designers of the writing test or guest lecturer Isn t it the support question [in the scoring rubric] that misleads us? But we talked about that a little bit yesterday, and that, eh, if you don t have any mistakes in punctuation Look, to get an M5 the writer has to get close to writing all the words correctly, but some random errors may occur. And I think they mentioned that as well; when it comes to spelling it s not that easy to use.

9 April 2012, gruppevurdering. Vurderer R14, R32 og R33

10 Det jeg har fremhevet: Endring av vurderingskultur går sakte Delvis må det forklares gjennom de felles vurderingsressursenes uklare status. Delvis må det forklares gjennom arbeidets kompleksitet: De felles vurderingsressursene må «fylles med mening», og det gjøres gjennom gjentatte vurderinger over tid. X-faktor? (vurderernes utgangspunkt (tekstkompetanse/ vurderingskompetanse/profesjonskompetanse (taus kunnskap/ verdier))) Rikt (heterogent) responsmønster +/ Henvisning til vurderingsressurser = Kvalitetsvurderinger Kvalitativ skrivevurderingsforskning balanserer retorikken ift ekspertkompetanse innenfor skrivevurderingsarbeid (jf interrater reliabilitet)

11 Utvalgt litteratur Cooksey, R. W., Freebody, P, & Wyatt-Smith, C (2007). Assessment as Judgment-in-Context: Analysing how teachers evaluate students writing. Educational Research & Evaluation, 13, Evensen, L. S. (2009). Vurdering av skrivekompetanse, en kompleks utfordring. In O. K. Haugaløkken, L. S. Evensen, F. Hertzberg, & H. Otnes (eds.), Tekstvurdering som didaktisk utfordring (15-23). Oslo: Universitetsforlaget. Evensen, L. S. (2012). Underveis mot et tolkningsfellesskap: lærerstemmer om elevtekster. In S. Matre, D.K. Sjøhelle, & R. Solheim (eds.), Teorier om tekst i møte med skolens lese- og skrivepraksiser ( ). Oslo: Universitetsforlaget. Fasting, R., Thygesen, R., Berge, K. L., Evensen, L. S., & Vagle, W. (2009). National Assessment of Writing Proficiency among Norwegian Pupils in Compulsory Schools. Scandinavian Journal of Educational Research, 53, Huot, B. (2002). (Re)articulating Writing Assessment for Teaching and Learning. All USU Press Publications. Book 137. Retrieved September 20, 2013, from Lim, G. S. (2011). The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters. Language Testing, 28, Linell, P. (2001). Approaching Dialogue. Talk, interaction and contexts in dialogical perspectives. Amsterdam: John Benjamins. Linell, P. (2009). Rethinking Language, Mind, and World Dialogically. Charlotte, NC: IAP, INC. Lumley, T. (2002). Assessment criteria in a large-scale writing test: what do they really mean to the raters? Language Testing, 19, Maxwell, G. (2001). Moderation of assessments in vocational education and training. Brisbane, Queensland: Department of Employment and Training. Purves, A. C. (1992). A Comparative Perspective on the Performance of Students in Written Composition. In A. C. Purves (red.). The IEA study of written composition (Vol ). Oxford: Pergamon. Rommetveit, R. (1992). Outlines of a dialogically based social-cognitive approach to human cognition and communication. In Wold, A. H. (Ed.), The Dialogical Alternative: Towards a Theroy of Language and Mind (pp ). Oslo: Scandinavian University Press. Sadler, R. (1987). Specifying and Promulgating Achievement Standards. Oxford Review of Education, 13, Sadler, R. (2011). Academic freedom, achievement standards and professional identity. Quality in Higher Education, 17, Suto, I. & Greatorex, J. (2008). What goes through an examiner s mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal, 34, Thygesen, R., Berge, K. L., Evensen, L. S., Fasting, R. B. (2007). Sluttrapport: nasjonale prøver i skriving som grunnleggende ferdighet [Final report: National tests in writing as a key competency]. Stavanger: Nasjonalt senter for leseopplæring og leseforsking, Universitetet i Stavanger. Weigle, S. C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11, Wyatt-Smith, C., Klenowski, V., & Gunn, S. (2010). The centrality of teachers judgement practice in assessment: a study of standards in moderation. Assessment in Education: Principles, Policy & Practice, 17,