Learning Analytics Future possibilities and current limitations. Examples from Norway, Denmark and Singapore.

Transkript

1 Learning Analytics Future possibilities and current limitations. Examples from Norway, Denmark and Singapore.

2 Conexus - Release knowledge! Founded in employees' Software and content We work in: Norway Denmark Singapore Sweden from 2014

3 250 municipality costumers The Norwegian ministry of education The Norwegian Directorate for Education and Training Norwegian research consul Innovation Norway Funded by:

4 Concept partners Norway IMTEC Gyldendal Kommuneforlaget (KS) myoutime Microsoft International KMD Education (Denmark) Conexus Orient (Singapore) Orient Software (Vietnam) Knewton (New York) IST (Sweden)

5 In Norway The national government, all students and teachers. 80% marked All big publishers

6 Evidence Informed Quality Work (EIQW) The main purpose of the project is to develop and test processes and technology that supports quality work based on updated evidence. Over 30 school and preschool owners participate, together with 100 organizations (preschools, primary schools, upper secondary schools).

7 Content analyzed in the examples, form: Norwegian government All SAS providers in Norway SSB Gyldendal (publisher) Cappelen (publisher) Aschehoug (publisher) KIKORA National Center for Mathematics National Center for Reading 20 Other content providers

8 Academic feedback from: Erling Lars Dale Knut Roald Peter Mortimore Louise Stoll Kjell B. Hjertø Andy Hargreaves Viviane Robinson

9 Learning analytics has to develop: Learning for all students Professional Capital Assessment for learning Professional learning communities 21 CENTURY capacities Evidence informed processes Student Centered leadership Open To Learn Communication

10 The strategy to implement Learning analytics must: 1. Not only focus on the easy counted information 2. Build capacity and evidence informed processes 3. Improve students and school performance by assessment for learning 4. Be based on trust in the relation between government, school leadership, teachers, students and the community.

11 Learning Analytics possibilities

12

13

14

15 Learning Analytics ecosystem

16 A tool is only as good as the mindset using it (Fullan 2013)

17

18

19 1. Vision, values and direction 1.1 Uplifting leadership 1.2 Ethical leadership 1.3 Establishing goals 2. Evidence-informed practise 2.1 Reflective enquiry 2.2 Research-informed practice 2.3 Data-informed practice 3. Professional learning and development 3.1 Learning environment 3.2 Feedback culture 3.3 Collaborative learning 3.4 Collective responsibility 4 Innovation, improvement and change 4.1 Change processes 4.2 Improvement 4.3 Innovation 4.4 Continuous learning and development 5 Local and global responsibility 5.1 Collaboration with the municipality 5.2 Collaboration between school and home 5.3 Collaboration with other schools 5.4 Global awareness 6 Management 6.1 Information Literacy 6.2 Using School Administration Systems (SAS) 6.3 Communicating and sharing information (LMS) 6.4 Evaluating/monitoring (PULS) 6.5 Tracking and follow up of students (VOKAL) 6.6 Capacity assessment and appraisal systems (KAN) 1. Establishing goals and expectations 1.1 Clear Purpose 1.2 Raising expectations 2. Fostering relationships 2.1Relationships with students 2.2Relationships between students 2.3Relationships with parents 2.4Relationships with community? 3. Engaging students 3.1 Promote passion for the subject 3.2 Recognising different starting points 3.3 Designing challenging learning programs 3.4 Developing students responsibility for learning 3.5 Encouraging well-organized cooperative learning 3.6 Enhancing learning through technology 4. Assessing learning 4.1 Creating success criteria 4.2 Closely monitoring students understanding 4.3 Giving and receiving feedback 4.4 Promoting self-assessment 4.5 Promoting peer assessment 4.6 Tracking progress over time 5. Enquiring into your practice 5.1 Reflective enquiry 5.2 Research-informed practice 5.3 Data-informed practice 6. Committing to collective responsibility 6.1 Collaborative learning 6.2 Collective responsibility 7. Stepping up to leadership 8. Innovating and changing 8.1 Experiencing change 8.2 Learning about your learning

20 School leaders need to know Professional Capital Assessment for learning 21 CENTURY capacities Evidence informed processes Professional learning communities Student Centered leadership OTL

21

22 Teachers need to know 1. Establishing goals and expectations 1.1 Clear Purpose 1.2 Raising expectations 2. Fostering relationships 2.1Relationships with students 2.2Relationships between students 2.3Relationships with parents 2.4Relationships with community? 3. Engaging students 3.1 Promote passion for the subject 3.2 Recognising different starting points 3.3 Designing challenging learning programs 3.4 Developing students responsibility for learning 3.5 Encouraging well-organized cooperative learning 3.6 Enhancing learning through technology 4. Assessing learning 4.1 Creating success criteria 4.2 Closely monitoring students understanding 4.3 Giving and receiving feedback 4.4 Promoting self-assessment 4.5 Promoting peer assessment 4.6 Tracking progress over time 5. Enquiring into your practice 5.1 Reflective enquiry 5.2 Research-informed practice 5.3 Data-informed practice 6. Committing to collective responsibility 6.1 Collaborative learning 6.2 Collective responsibility 8. Innovating and changing 8.1 Experiencing change 8.2 Learning about your learning

23

24

25 Modell 1 Relation between data MAP Results MAV Support from home Class climat Teacher Self-eff Motivation

26 Learning analytics in Norway Final estimation of fixed effects Fixed Effect Coefficient t -ratio d.f. p -value For INTRCPT1, β 0 INTRCPT2, γ 00 3,78 169, <0.001 TRINN_ME, γ 01 0,03 0, ,52 K_MILJ4, γ 02 0,13 2, ,01 LAERER17, γ 03 0,33 4, <0.001 For KJONN slope, β 1 INTRCPT2, γ 10 0,09 2, ,01 For KARAKT6 slope, β 2 INTRCPT2, γ 20 0,08 4, <0.001 For VEIL3_11 slope, β 3 INTRCPT2, γ 30 0,19 11, <0.001 For HJEM3_09 slope, β 4 INTRCPT2, γ 40 0,14 8, <0.001 Final estimation of fixed effects Fixed Effect Coefficient t -ratio d.f. p -value For INTRCPT1, β 0 INTRCPT2, γ 00 3,78 166, <0.001 TRINN_ME, γ 01-0,02-0, ,70 K_MILJ4, γ 02 0,20 3, <0.001 LAERER17, γ 03 0,42 5, <0.001 For KJONN slope, β 1 INTRCPT2, γ 10 0,10 2, ,01 For KARAKT6 slope, β 2 INTRCPT2, γ 20 0,08 4, <0.001 For VEIL3_11 slope, β 3 INTRCPT2, γ 30 0,20 12, <0.001 TRINN_ME, γ 31 0,02 0, ,58 LAERER17, γ 32 0,10 1, ,05 For HJEM3_09 slope, β 4 INTRCPT2, γ 40 0,14 8, <0.001 Individnivå (N=1757/55) Nr.2. Den nest viktigste faktoren for motivasjonen er vellykkede veiledningssamtaler med rådgiver (t=11.97, β=.19***). Jo bedre veiledning, desto bedre motivasjon. Rådgivningen er den mest signifikante faktoren, men ikke den sterkeste. Hvis man tar med interaksjonen mellom trinn og rådgiver, er det en signifikant interaksjonseffekt mellom rådgivers veiledning og tilfredshet med lærerne på motivasjonen (t=1.97; β=.10; p<.05). Jo bedre lærere, desto bedre effekt av individuell veiledning, ut over den direkte effekten mellom veiledning og motivasjon. Nr.3: Støtte fra hjemmet. Jo bedre hjemmestøtte, desto høyere motivasjon (t=8.59; β=.14***). Nr.5: Kjønn (t=2.58; β=.09**). Kontrollfaktor. Nr.6: Karakterer (t=4.43; β=.08***). Kontrollfaktor. Gruppenivå (N=115) Nr.1: Viktigste faktor: Lærerne (t=4.21; β=.33***). Lærerne er de som påvirker elevenes motivasjon mest. NB! Elevene ble ikke spurt om lærernes faglige dyktighet (naturlig nok?). Nr.4: Klassemiljø (t=2.53; β=.13**). Nr.7: Klassetrinn betyr ingen ting for motivasjonen (kontrollert for de andre faktorene i modellen). Konklusjon:

27 Learning analytics in Norway Final estimation of fixed effects Fixed Effect Coefficient t -ratio d.f. p -value For INTRCPT1, β 0 INTRCPT2, γ 00 3,77 218, <0.001 TRINN_ME, γ 01 0,01 0, ,77 K_MILJ4, γ 02 0,17 4, <0.001 LAERER17, γ 03 0,23 3, <0.001 For KJONN slope, β 1 INTRCPT2, γ 10 0,10 3, ,00 For SE8_01 slope, β 2 INTRCPT2, γ 20 0,31 11, <0.001 For MAV6_03 slope, β 3 INTRCPT2, γ 30 0,11 4, <0.001 For MAP3_12 slope, β 4 INTRCPT2, γ 40 0,22 10, <0.001 Final estimation of fixed effects Fixed Effect Coefficient t -ratio d.f. p -value For INTRCPT1, β 0 INTRCPT2, γ 00 3,78 167, <0.001 TRINN_ME, γ 01-0,02-0, ,65 K_MILJ4, γ 02 0,20 3, <0.001 LAERER17, γ 03 0,42 5, <0.001 For KJONN slope, β 1 INTRCPT2, γ 10 0,13 3, <0.001 For SE8_01 slope, β 2 INTRCPT2, γ 20 0,31 11, <0.001 K_MILJ4, γ 21 0,10 1, ,09 LAERER17, γ 22-0,03-0, ,67 For MAV6_03 slope, β 3 INTRCPT2, γ 30 0,11 3, <0.001 For MAP3_12 slope, β 4 INTRCPT2, γ 40 0,22 10, <0.001 Individnivå (N=1757/55) Nr.1. Mestringstro (self-efficacy). Den viktigste faktoren for motivasjonen er mestringstro (t=11.96, β=.31***). Mestringstro er både den sterkest faktoren og den mest signifikante faktoren. Hvis man tar med interaksjonen mellom lærer og mestringstro, er det en marginalt signifikant interaksjonseffekt mellom mestringstro og klassemiljø på motivasjonen (t=1.68; β=.10; p<.10). Jo bedre klasseklima, desto sterkere effekt har mestringstroen på motivasjonen, ut over den direkte effekten av på motivasjon. Nr.3: Aktiv mestring mot mål (MAP) (t=10.42; β=.22***). Nr.5: Perfeksjonistisk mestring (MAV) (t=4.05; β=.11***). Nr.6: Kjønn (t=3.19; β=.10**). Kontrollfaktor. Gruppenivå (N=115) Nr.2: Lærerne (t=3.92; β=.23***). Lærerne er de som påvirker elevenes motivasjon mest etter mestringstro. Nr.4: Klassemiljø (t=4.51; β=.17**). Nr.6: Klassetrinn betyr ingen ting for motivasjonen (kontrollert for de andre faktorene i modellen). Konklusjon:

28 Results and progression

29 Learning analytics is about: Selection and focus

30

31

32

33 1 3 Input Proses Actions Profil

34

35

36

37 Multi 3rd grade Thegraph shows the amount of money four children spent in a week. Use the graph to answer the questions. How much did Ben spend? Dollar $ Who spent over $ 500? John Eve Alan Kate What is the difference between the one who spent the most and the one that spent the least? 40$ 60$ 80$ 100$ John Eve Kate Alan Previous Next

38 Math <header> Numbers Algebra Data Stats % Meassurement Geometry

39 <header> <name> <updated> <header> The result of a test is measured as a percentage of the norm (national average) for each function or each subtest. A score of 100% means, that the result is identical to the national average. A score of 130% would mean that the student has 30% more right than the national average <header> <level> <esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på 100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett> <header> <level> <esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på 100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett> <header> <level> <esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på 100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett enn landsgjennomsnittet, mens et resultat på 70 % vil tilsvarende bety at eleven har svart rett på 70 % av det en gjennomsnittselev har gjort. Kategori 1: > 80 % av norm. Resultatet indikerer at eleven trolig ikke har behov for tilrettelagt opplæringstilbud på dette> <header> <text> <text> <header> <header> <header> <header> <page> <page> <page> <chapter link> <text> <text>

40

41

42

43

44 Math New data in cooperation with Knewton Work load Task level Goal % Progression Test correlation

45

46 Students development tool Assessment for learning 21 CENTURY capacities

47 21 CENTURY capacities What it means The term 21 st century capacities refers to a broad set of knowledge, capacities, work habits, and character traits that are believed to be critically important to success in today s world, particularly in collegiate programs and modern careers. Generally speaking, 21 st century capacities can be applied in all academic subject areas, and in all educational, career, and civic settings throughout a student s life. Creative thinking 80% Creative thinking is the process we use when we come up with new ideas. creative thinking process can be accidental or deliberate Do it again

48

49 Students overview Students overview Numbers Algebra Data Stats % Meassurement Geometry Math ENG Reading Science Sport Test-correlation Task level Progression Self-efficacy Self-assessment

50

51 Logged in Start Page Supervision report Management areas Teaching evaluation Management targets Quality description School owners report Display Selection Export School report Selection Year Dotted Updated My reports NYC Total report Information Total report Key information Indicators School 1 School 2 School 3 School 4 Parents education level Class room management Motivation and coping National tests, 5th grade National tests, 8th grade National tests, 9th grade Primary points Deviation achievement- and examination mark Proportion who have successfylly completed first year of college Grade development from middle school to first year of college

52

53

54

55

56 Level Quality characteristics - Student Assessment Level Students find that feedback from the teachers is adapted to their 4 prior knowledge. Students know what they have achieved and what to strive for. They utilize the feedback so the learning environment is characterized by motivation and student participation. The teachers have a common understanding of what competence the various characters express. Level 3 Level 2 Level 1 Students and teachers have a common understanding of the strategies that should be used to achieve good learning. Teachers engage students in assessing their own work. Final assessment is based on the total competence goals. The school has a clear focus on assessment, but the feedback is often general and self-assessment is not set in system. Final assessment is to some extent based on the overall competence goals. The teachers' practice of continuous assessment is unsystematic. Final assessment is just to a small extent based on the overall competence goals. Indicators Student survey - Motivation - learning and assessment - professional guidance - criteria for self assessment

57 Level Quality characteristics - Organizational Learning Indicators Level 4 The school is involved in learning at all levels. There is a culture of Personnel survey sharing knowledge and exchanging good practice. Colleagues support - Collegial support - Common goals each other and make each other better. The discussions are - Learning Press characterized by confidence and openness. - Extra role behavior The school's development is characterized by collaboration and reflection on own practice and analysis of their results. The school is in the forefront of academic and educational development, uses new research and learn from good examples from other schools. Level 3 The school's educational development follow central planning and control documents. The school's goals and priorities are known to the staff. The school analyzes its own performance and set targets. Level 2 The school tries to systematize the educational development in relation to central planning and control documents. Individual results are presented for the staff. Level 1 The school's educational development is characterized by randomness, and the effort of individual initiative.

58

59 Learning Analytics current limitations Not all information in the same system (the ecosystem is incomplete) Not enough standards in the ed-tec industry The ed-tec industry is not cooperating The target is not the students learning but the government control need, or researchers joy in finding correlations Researchers focus on algorithms or teaching, not the processes in a ecosystem The governments focus on content and not structures Professional capital and capacity building is not the target for the tools and reports To much data driven control or personalized tasks, not enough evidence based processes and learning to learn. Not enough focus on the markers of future performance and formative assessment (Improve school performance by assessment for learning) Not enough trust in the relation between government, school leadership, teachers, students and the community.

60 Learning Analytics ecosystem

61 Yngve Lindvig Chief Research and Development Officer Conexus Linkedin: