AHELO feasibility study. Informasjonsmøte i KD 1. desember 2009 Spesialrådgiver Jan S. Levy

Transkript

1 AHELO feasibility study Informasjonsmøte i KD 1. desember 2009 Spesialrådgiver Jan S. Levy

2 Project background Why undertake the study? After decades of quantitative growth in HE and its growing internationalisation, consensus on the need to ensure quality provision for all but information gap on learning outcomes. Existing rankings have not focus on LO s Carry out a feasibility study to provide a proof of concept (Tokyo, 2008) Feasibility study goals? To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible. A research approach to provide a proof of concept and proof of practicality

3 Project summary: strands of work Discipline strand in Economics Basis for discussion of expected learning outcomes delivered through Tuning approach. + contextual data Discipline strand in Engineering Basis for discussion of expected learning outcomes delivered through Tuning approach. + contextual data Generic skills strand International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problemsolving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts. Research-based Valueadded or Learning gain measurement strand Several perspectives to explore the issue of valueadded (conceptually, psychometrics), building on recent OECD work at school level. + contextual data

4 Participation in the three strands of surveying Discipline strand in Economics Belgium (Fl.), Italy, Mexico, Netherlands Discipline strand in Engineering Australia, Japan, Sweden Generic skills strand Finland, Korea, Mexico, Norway

5 Project summary (cont.d) Who? Data will be collected from a targeted population of students who are near but before the end of their first 3-4 year degree. When? The testing window is from August 2010 to April How? Why? Joint steering by governments, agencies and HEIs through IMHE OECD s role is to establish broad frameworks that guide international expert committees in charge of instrument development. Several options: Use existing instruments, combine existing item pools to cover frameworks, or develop new test material? The outcomes will be used to assist countries to decide on the next steps.

6 Feasibility study outputs Two reports on the feasibility of reaching international consensus on defining the intended learning outcomes in two contrasted disciplines and in a cross-national fashion Three reports on the scientific and practical feasibility of implementing an international assessment focusing on disciplinespecific skills in economics and engineering and on generic skills One report on the measurement of value-added in higher education, net of students incoming abilities and differences in institutional selectivity A final conference and its proceedings to discuss the outcomes of the feasibility study as well as the potential impact of AHELO Less tangible: Enhanced understanding of the project and buy-in from stakeholders

7 Country participation update Critical mass attained (10 participants) Scope for 4-5 more countries Formal requests from 4 US states, Russian Federation Discussions with Turkey Volunteer requests from HEIs (France, Germany, Spain) Interest from NME non-eea countries (Kuweit, Colombia)

8 Internasjonal organisering Ledes av OECDs utdanningspolitiske komite og av IMHE GB Oppnevnt særskilt gruppe fra deltagende land (GNE) ansvarlig for gjennomføring Ekspertgrupper på de enkelte temaer OECD stiller strenge kvalitetskrav til utvikling, oversettelse og tilpasning av undersøkelsesinstrumenter Konsultasjonsorgan med interessegrupper internasjonalt (EI, BIAC, TUAC, ESU, ENQA, EUA..) Undersøkelsen gjennomføres av internasjonale konsortier i samarbeid med deltagende land

9 Scope of tendered work Initial work: Tuning AHELO Sole supplier Generic Skills Strand (GenS) Economics Strand (EcoS) Engineering Strand (EngS) Value-Added Measurement Strand (VamS) Development of the GenS intrument Development of the EcoS intrument Development of the EngS intrument VamS Resarch-based: Methodological/ Instrument development Initial work: Contextual Dimension (Module A) Contextual dimension surveys: (Module D) (Module B) (Module C) - student -faculty - programme leadership - institutional leadership No direct assessment involved Done through commissioned experts (Module F) Field implementation and analysis Project management, survey operations and analyses of results (Module E)

10 Two proposals received Proposal 1 Proposal 2 Consortium led by the National Foundation for Educational Research (NFER) in partnership with Cito B.V. National Institute for Educational Measurement, and one organisation included as subcontractor: The Gallup Organisation Europe (Gallup Europe) Consortium led by the Australian Council for Educational Research (ACER) with the following associates: Capstan Linguistic Quality Control Agency (Belgium) Indiana University Center for Postsecondary Research (USA) University of Twente Centre for Higher Education Policy Studies (Netherlands) Educational Testing Service (USA) International Association for the Evaluation of Educational Achievement (Netherlands) National Institute for Educational Policy Research (Japan) Statistics Canada (Canada) University of Florence (Italy) Westat Inc (USA)

11 The European Classification of Higher Education Institutions. Welcome to U Map. The European Classification of Higher Education Institutions. U Map is the third phase of a research project to develop a European Classification of Higher Education Institutions. The classification maps the European higher education landscape and will make the institutional diversity of that landscape more transparent. U Map will not rank the institutions league table style, but will position institutions on a number of dimensions, each representing an aspect of the function and performance of higher education institutions. Click here to go to the finder. About the Project. The concept of diversity has moved rapidly up the political agenda over the past decade. The development of the European Higher Education Area (EHEA) and the European Research Are The Methodology. U Map has been developed in a dynamic and interactive process in which a number of general guidelines and design principles were used. The consultation of stakeholders was a corners Watch the demo.

12 Arbeid i Norge Tre norske institusjoner: NTNU, UMB, HiVe Generic skills strand Tilbud fra CAE om å gi seminarer om CLAundersøkelsen også til andre institusjoner Planlagt gjennomført i februar/mars 2011 Rapporter foreløpig planlagt til høsten 2011 Nasjonal projektleder (NPM) NIFU STEP i samarbeid med ILS/EKVA Prosjektorganisert i departementet prosjektgruppe, styringsgruppe, informasjonsmøter

13 Hvilke typer resultater for Norge? Bidrag til den internasjonale studien Internasjonale sammenligninger vil ikke bli laget. Utvalgene vil ikke være representative internasjonalt Data som kan identifisere respondent blir utilgjengelige Institusjonene må få rapport med sine resultater Departementet må få rapport om norske institusjoner som deltok Offentliggjøring av nasjonale resultater ikke avgjort Ikke klart om det vil bli gitt noen form for sammenligning med internasjonal benchmark

14 TASK DESCRIPTION Performance Task Each Performance Task requires students to use an integrated set of critical thinking, analytic reasoning, problem solving, and written communication skills to answer several openended questions about a hypothetical but realistic situation. In addition to directions and questions, each Performance Task also has its own document library that includes a range of information sources, such as letters, memos, summaries of research reports, newspaper articles, maps, photographs, diagrams, tables, charts, and interview notes or transcripts. Students are instructed to use these materials in preparing their answers to the Performance Task s questions within the allotted 90 minutes.

15 Eksempel på performance task

16