Description of the research project:

Supporting personalised learning in secondary schools through the use of specific mathematics assessments that reveal thinking

This page describes the research project that accompanies the development of the SMART tests. This project is conducted at the University of Melbourne by Professor Kaye Stacey, Dr Helen Chick and Dr Vicki Steinle. It is funded by the Australian Research Council Linkage Projects Program LP0882176 in partnership with the Department of Education and Early Childhood Development, Victoria (2008 - 2010). Our project officers are Ms Beth Price and Mr Eugene Gvozdenko.

Research Goals

The effectiveness of the smart-tests will be thoroughly tested, for impact on student achievement and on the mathematics teaching skills of teachers. The smart-tests form an expert system for teaching mathematics, which takes the research knowledge of experts and puts it in a useable form for teachers.
The project therefore has 3 objectives:

• to create research-based smart-tests, smart diagnosis and smart teaching delivered with intelligent technology;
• to evaluate quantitatively and qualitatively the effect of long-term use of these resources on student achievement and teachers' mathematics pedagogical content knowledge; and
• to examine the processes through which teachers in secondary schools make the transition to personalised learning for mathematics using these resources.

The project will test two hypotheses:

• if teachers have easy access to information about their students' thinking and they make use of resources that target the development of conceptual understanding, then student achievement will increase;
• teachers' mathematics pedagogical content knowledge will increase if they have easy access to information about their students' thinking.

Figure 1. How teachers use smart resources to bring about higher student achievement, and how this encourages teacher growth.

Personalised learning and MPCK

The underlying goal of this project is to support the use of personalised learning in mathematics, an ambitious approach to teaching that is being advocated by our Partner Organisation, the Victorian Department of Education and Early Childhood Development (DEECD). The intent of personalised learning is to maximise the outcomes of schooling for each student, by tailoring teaching to each student's needs as identified by high-quality assessment. As the UK Department for Education and Skills (HREF1) points out, personalised learning considers the whole child, and goes beyond teaching in individual subjects to include curriculum entitlement, schools organised around teams supporting learning, and welfare initiatives beyond the classroom. Because this proposal focuses on the role of the classroom teacher, we will use the phrase 'personalised learning' to refer to just the classroom core of this 'whole child' view. The phrase thus describes an approach to maximise the effectiveness of teaching by:

• specifying curriculum and learning standards that clearly describe what students should learn and the stages through which they will generally pass;
• providing teachers with good data on each student's current knowledge;
• equipping teachers with skills to design teaching that will move students to the next stage.

Accompanying the move towards personalised learning is a new vision of a teacher as one who has the specialised pedagogical expertise to use data about the current knowledge and skills of each of their students to implement a personalised learning plan which will move that student forward. In this proposal's context of teaching mathematics and numeracy, we call this specialised expertise 'mathematical pedagogical content knowledge' (MPCK).
Personalised learning, in the above sense, has been shown to improve educational outcomes in some very successful projects, starting with Cognitively Guided Instruction (Carpenter, Fennema and Franke, 1996) and including several major local initiatives, most notably the NSW government's seminal Count Me In program (HREF2; Wright, Martland, Stafford & Stanger, 2002), Victoria's Early Numeracy Research Project (Clarke, 2001) and New Zealand's Numeracy Development Projects (HREF7), all of which have worked primarily in early years' numeracy. The intention is to create resources for implementing personalised learning in secondary school mathematics, and to evaluate the outcomes of the approach. Working in secondary schools calls for many innovative features, since the established early numeracy practices cannot be used in such a different environment.

Smart-tests are an expert system for teaching

There is a substantial body of research into students' mathematical thinking that has been accumulated internationally, and to which we have contributed ourselves (Stacey: number, algebra, working mathematically; Chick: chance and data; Steinle: number). However, such research needs considerable transformation for maximum effectiveness for teaching. One example of research that was ready to use involved decimal numeration, where Steinle and Stacey's work (Steinle & Stacey, 2003) on the Decimal Comparison Test provided an example of what is needed: a test that is quick to administer to a whole group, easy for teachers to interpret, and which reveals the underlying conceptual development of students, so highlighting areas for teaching to target. The usefulness of Steinle's approach has been widely acclaimed by teachers. This gave us the prototype smart-test, which is central to this project. The common features of smart-tests are that they can be administered to a group in a short time, focus on conceptual understanding of key ideas, and can lead to insightful diagnosis at a level of detail that can inform follow-up teaching.

The creation of smart resources embeds advanced research into artefacts that are easy for practitioners to use, creating what Pea (1993) calls 'distributed intelligence' in tools. When planning the teaching of a new topic, the smart-tests and diagnosis provide teachers with better knowledge of the mathematical thinking of their current students. Incidentally, yet importantly, teachers will also learn about the mathematical knowledge of students more generally.

Using the recommended smart teaching strategies provides enhanced instruction for the current students, and this also builds teachers' capacity for future teaching. Testing whether this growth of teachers' MPCK actually occurs is a key research aim; there is now international consensus in the research literature that teacher quality and especially their discipline-specific teaching expertise is a major determinant of overall student learning outcomes (see, for example, Darling-Hammond, 2000). Chief Investigator Chick found that some teachers lacked the necessary MPCK to maximise learning or address misconceptions, but also noted that the provision of robustly designed teaching activities could enhance the effectiveness of lessons and strengthen the MPCK of teachers (Chick & Baker, 2005).
In addition, there will be a key knowledge transfer outcome, of transforming the results of research into a form useful for teachers, and making them highly accessible through a website with adaptive capabilities, making smart use of technology to assist teaching. We intend that teachers' MPCK will improve by using the resources. However we also intend that the distributed intelligence in the technology tools we develop will provide expert advice for teachers whose MPCK is not well advanced, and will thereby make a personalised learning approach easier and more practical for all teachers.

Innovative features

The significance and innovation of the project arise from the challenge of extending the personalised learning approach, which has proved successful in early years' numeracy, to secondary school mathematics. To do this, this project has to create a solution that will operate well in the very different environment of the secondary school, make the best use of the strong but incomplete research base available, extending it at key points. Pragmatically, the solution also has to be sustainable with a lower budget than has been provided for the early years' interventions. As is evident in several of the points of innovation and significance below, we aim to find the right level of detail of information (e.g. about research or students' understanding) to support action. The project is also innovative in its attempt to provide a new level of support for teaching through adaptive programming in the online tools.

Powerful resources built on research into students' thinking.

The well-researched 'SMART tests' will probe conceptual understanding and be simple to use. The SMART diagnosis will allow teachers to learn about the thinking of their students, without the need to be an expert in all aspects of research into mathematical thinking. This is a key point of innovation for the project. While there are many tests already available to teachers, most assess the ability of students to follow learned procedures, rather than the present priority of revealing conceptual understanding. We see this as a difference between measuring learning (finding out how much students know in a summative assessment) and mapping learning (showing where students are in the conceptual field and pointing out their path to improvement). Mapping learning requires a different approach to assessment and standard measurement techniques such as Rasch analysis are not appropriate (Stacey & Steinle, 2006).

Using SMART resources has great potential to encourage teachers to consider their students' thinking more deeply. In the early numeracy programs, an individual interview was accepted as the only way to obtain reliable data on students' thinking. We understand the value of interviewing but, with the Decimal Comparison Test as evidence, we assert that information that is good enough for action can often be obtained more efficiently. Using SMART resources will also allow information gathering close to the time of teaching a topic, rather than just at one time in the year.

While students' raw responses will provide some information for teachers, the deeper analysis available in the SMART diagnosis will provide greater insight into the understanding of individuals and groups. Supporting documentation will assist teachers to recognize the known idiosyncrasies of the conceptual field; however, a well-designed SMART test will not require the user to have detailed specialist knowledge. A bank of SMART tests will be available for a wide range of topics from the lower secondary mathematics curriculum, covering the conceptual issues that research has shown to be stumbling blocks for students.

With detailed data-driven diagnoses of student understanding, it will be possible to make explicit suggestions of relevant activities, materials, and teaching strategies that focus on the conceptual growth desired in students. This will allow teachers to clearly determine how to provide students with personalised learning opportunities. This will extend the Mathematics Developmental Curriculum significantly, by providing a research base for such materials and a measure of their effectiveness. It is anticipated that these resources (the SMART tests, diagnoses, and teaching ideas) will enhance teachers' MPCK on the specific topics included in the resources and more generally.

Using intelligent technology

We see great potential in increasing the range of automated and adaptive services available to support teaching. There has been considerable growth in this area, but it focuses on systems to provide and manage student learning (e.g. student activities automatically graded and reported to teachers). We will instead use the automated features possible on a website with a well-designed database and strong meta-data to assist teachers to select resources. An adaptive service will help teachers select appropriate SMART tests, provide SMART diagnosis and suggest SMART teaching approaches sourced from the growing range of materials online. This will make teachers' work more efficient. Various processes will also be trialled to establish the best methods of providing diagnosis of individuals and groups. For example, we have created adaptive diagnosis of student misconceptions about decimal numbers using Bayesian net software (Stacey, et al., 2003).

Research on data-support systems is just beginning in the Education field, according to Breiter and Light (2006). They comment that the needs of the end-user must be built in at the start, whereas often systems simply present data that has been collected. A system designed to support teachers' decision making should begin with the source of the data (i.e. the students and their learning needs) and be tailored to decisions that the teacher needs to make.

Managing personalised learning in secondary schools

Focussing this project on secondary schools provides significant challenges and opportunities for distinctive research and strategic benefit. Personalised learning will not be easy to implement in secondary schools where teaching is normally directed to the whole class, in contrast to the individual and small group instruction common in primary schools. In primary schools, teachers teach only one class, know their students very well, and work on a restricted range of foundation topics with slowly accumulating learning. In secondary schools, in contrast, teachers often teach around 120 students per week, and the pace at which topics are covered is much greater. Moreover mathematics at the secondary level has different strands, intricately inter-connected, which renders inapplicable the Early Numeracy model of one framework encapsulating mathematical progress.
We have planned our SMART tests to work well in this environment, by being easy and quick to use with groups of students, giving just enough information for action. In the case study schools, we can observe how features of the SMART resources affect their usability. As an example, one issue is whether SMART diagnosis should be at the student level, the small group level or the class level. It is likely that teachers may not be able to act on detailed information about each student, given the time constraints of classroom teaching. Instead, information that highlights two or three common responses from the class may prove more useful. The intelligent website could deliver this.

Fostering and tracking teacher development

It has been amply demonstrated that improvements in student learning result from teachers' understanding the mathematical needs of their students. For example, Hill et al. (2005) found that in the early primary grades, teachers' MPCK was significantly related to student learning gains. All the early years' numeracy projects recommend that teachers individually interview each child each year, and this has proved to be a major vehicle for improving teachers' MPCK. However, such an approach demands too much teacher time to be feasible at the secondary level. Another standard model for improving teacher's professional knowledge is face-to-face professional development. However, this is an expensive approach and it is not available to all, most notably those in rural and regional areas. There is therefore an important need to research other ways of providing effective teacher professional development.
Our approach aims to provide resources for all teachers, to make their work easier and provide tangible benefits for students' learning. Guskey (1986) and many others have observed that the best stimulus for teacher change is seeing improvements in their own students' learning. Following this model, we intend that the SMART resources will not initially require major changes in teachers' behaviour, yet will lead to improvements in student learning that are obvious to the teacher. Noticing the change then stimulates the teacher to engage in deeper and longer-lasting personal change. A key factor in initiating the change will be the recommendation from DEECD, but the key factor in sustaining the process of change will be the capacity of our resources to increase the effectiveness of teaching. In this way, we intend that teacher professional development and the growth of MPCK can be supported sustainably. The project will test this hypothesis in 'Evaluation Schools' who deliberately have minimal contact with the research team.

Another key factor is that up to one-third of secondary mathematics teachers across Australia have had no mathematics method studies (Harris & Jensz, 2006). These teachers sometimes have a weak mathematical content knowledge and have had no systematic development of MPCK in their pre-service education. Researching methods of increasing teachers' MPCK at junior secondary level therefore has added significance.

Components of Research

There are four components to this research project:

• creation and evaluation of the SMART resources;
• case studies of how schools implement personalised learning using the SMART resources;
• observation of the effects on students' learning and on teachers' MPCK;
• concurrent design and evaluation of the SMART website with adaptive features.

Creation of SMART tests, SMART diagnosis and SMART teaching

Items in the published mathematics education literature that reveal conceptual understanding will provide the basis for creation of SMART tests. The items need adaptation to fulfil the useability criteria and diagnostic potential of SMART tests Approximately 1500 students in Years 7 - 9 of the 4 Development Schools will trial the items and their teachers will provide feedback on test items, and validity of responses. Careful analysis of student responses will be used to select the most revealing items, establish the range of responses of present-day students, and provide the foundation for the SMART diagnosis. Conceptualising the stages of student thinking underlying the SMART diagnosis is a major task, along with selection of the SMART teaching ideas. Carefully analysis of patterns in the data will reveal systematic behaviours characteristic of correct and incorrect conceptualisations.

Case studies of the adoption of personalised learning

As with other school improvement programs (Fullan, Hill & Crevola, 2006), personalised learning is most likely to be successful on a whole school basis. The 3 case studies therefore each consist of the set of all Year 7 - 9 mathematics teachers in each school Focus groups and interviews with teachers will provide information on:

• teachers' satisfaction with the SMART resources and how they can be improved;
• teachers' use of the SMART resources and changes in teaching practices;
• any effect on student learning as noted by the teachers;
• self-reported effect on teachers' MPCK;
• the adoption of a personalised learning philosophy in the school.

The hypothesis that using SMART resources will improve teacher's MPCK is explored by tracking teachers responses to questionnaires and interviews from Dr Chick's recent research on  teachers' MPCK. It will be analysed using the framework of Chick et al. (2006).

NAPLAN data frequently indicates a wider spread of student achievement within a year level than is indicated by teacher assessments. For this reason, another measure of strengthening of  teachers MPCK is the correspondence between the teacher judgements of students and NAPLAN results.  The hypothesis is that more extensive use of the SMART resources will improve the correspondence..

Observing the effects on student achievement and MPCK

The effects of adopting personalised learning through the use of SMART resources will be tested in the 4 Evaluation Schools. These schools will have access to the growing set of SMART resources, and provide feedback via electronic surveys. Conditions in this group of schools are intended as a fair test of the power of the SMART resources in everyday circumstances. In contrast to the Case Study Schools, the researchers will have minimal contact with the Evaluation Schools.The main purpose is to test the hypothesis that the use of personalised learning, stimulated by use of the SMART resources, improves student achievement. This will require an extension of the current project to get adequate longitudinal data.

Design of the SMART website

The research will gather data on how well the SMART website meets the needs of teachers.
This site and its services will be evaluated and improved with input from participating teachers, via electronic surveys. The major research question is to what extent automated and adaptive features are really useful for teachers.

References

Breiter, A., & Light, D. (2006). Data for school improvement: Factors for designing effective information systems to support decision-making in schools. Educational Technology & Society. 9 (3): 206-217.
Carpenter, T. P., Fennema, E., & Franke, M. L. (1996). Cognitively guided instruction: A knowledge base for reform in primary mathematics instruction, The Elementary School Journal, 97 (1), 3-20.
Chick, H. & Baker, M. (2005). Teaching of Elementary Probability: Not Leaving it to Chance. In Pierce R, Roche A, Gronn D, Clarkson P, Horne M, McDonough A & Downton A (Eds), Building Connections: Theory, Research and Practice (Proceedings of the Annual Conference of the Mathematics Education Research Group of Australasia). 1, 233-249. Sydney, Australia: MERGA.
Chick, H., Baker, M., Pham, T. & Cheng, H. (2006). Aspects of Teachers' Pedagogical Content Knowledge for Decimals. In Novotna J, Moraova H, Kratka M & Stehlikova N (Eds), Proceedings of the 30th Conference of the International Group for the Psychology of Mathematics Education. 2 2-297 - 2-304. Prague: PME.
Clarke, D. (2001). Understanding, assessing, and developing young children's mathematical thinking: Research as a powerful tool for professional growth. In Mitchelmore M, Perry R &Bobis J (Eds), Numeracy and Beyond. 1, 9-26. Turramurra, Australia: Mathematics Education Research Group of Australasia.
Darling-Hammond, L. (2000). Teacher Quality and Student Achievement: A review of state policy evidence. Education Policy Analysis Archives, 8(1), http://epaa.asu.edu/epaa/v8n1 (accessed 28/1/2002).
Fullan, M., P. Hill and C. Crevola (2006), Breakthrough, Corwin Press, California.
Guskey, T.R. (1986). Staff development and the process of teacher change. Educational Researcher, 15(5), 5-12.
Harris, K.-L., & Jensz, F. (2006). The preparation of mathematics teachers in Australia: Meeting the demand for suitably qualified mathematics teachers in secondary schools. Melbourne: CSHE, Univ of Melbourne.
Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effect of teachers' mathematical knowledge for teaching on student achievement. American Education Research Journal, 42(2), 371-406.
HREF1 Department for Education and Skills (UK). The Standards Site: Personalised Learning. http://www.standards.dfes.gov.uk/personalisedlearning/five/afl/ Accessed 2007-04-11
HREF2 Department of Education and Training (NSW). (no date). Count Me In Too.
http://www.curriculumsupport.education.nsw.gov.au/countmein/assessment.htm Accessed 2007-04-11
HREF3 Department of Education (Vic). Student Reports - Making Judgments and Assigning Scores. Accessed 2007-03-22. http://www.education.vic.gov.au/studentlearning/studentreports/schools/judgmentscores.htm
HREF4 Department of Education (Vic). (2007). Mathematics Domain - Assessment. Accessed 2007-04-11. http://www.education.vic.gov.au/studentlearning/teachingresources/maths/assessment.htm
HREF5 Department of Education (Vic). (2007). Mathematics Developmental Continuum Accessed 2007-04-11 http://www.education.vic.gov.au/studentlearning/teachingresources/maths/mathscontinuum/
HREF6 Department of Premier and Cabinet (Vic). (2007). Council of Australian Governments' National Reform Agenda: Victoria's plan to improve literacy and numeracy outcomes. Consultation Draft. February 2007.
HREF7 Ministry of Education (NZ). (2005). Findings from the New Zealand Numeracy Development Projects 2005. http://www.nzmaths.co.nz/Numeracy/References/compendium05.aspx
Pea, R. D. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed.) Distributed Cognitions: Psychological & Educational Considerations. Cambridge University Press.
Siemon, D., Virgona, J. & Corneille, K. (2001) Final Report of Middle Years Numeracy Research Project 1999-2001, Melbourne: RMIT University
Stacey, K. & Steinle, V. (2006). A case of the inapplicability of the Rasch Model: Mapping conceptual learning. Mathematics Education Research Journal, 18(2), 77 - 92.
Stacey K, Sonenberg EA, Nicholson AN, Boneh T & Steinle VA. (2003). A Teaching Model Exploiting Cognitive Conflict Driven by a Bayesian Network. In P Brusilovsky, A Corbett & F De Rosis (Eds), User Modelling 2003. XIV : 352-362. Berlin, Germany: Springer-Verlag.
Steinle, V. & Stacey, K. (2003). Grade-Related Trends in the Prevalence and Persistence of Decimal Misconceptions. In Zilliox JT, Dougherty BJ & Pateman NA (Eds), Proceedings of the 2003 Joint Meeting of PME and PMENA. 4 259-266. Honolulu, United States: College of Education, University of Hawai'i.
Thomson, S., & Fleming, N. (2004). Summing it up: Mathematics Achievement in Australian Schools in TIMSS 2002. Retrieved 2006-08-16. http://www.timss.acer.edu.au/documents/TIMSS_02_Mathsreport.pdf
Watson, A. (2000). Mathematics Teachers Acting as Informal Assessors: Practices, Problems and Recommendations. Educational Studies in Mathematics, 41(1): 69-91.
Wright, R., Martland, J., Stafford, A., Stanger, G. (2002). Teaching Number: Advancing Children's Skills and Strategies. Paul Chapman Educational Publications: London

Publications