Research on smart tests

Research Goals

One of the major challenges in teaching mathematics is to provide personalised learning opportunities that meet the needs of individual students. Many students have misconceptions or gaps in their knowledge which are a barrier for further learning but often go unaddressed. The smart tests aim to give teachers readily useable information about each student's understandings to help the teachers select instruction that will bring about specific conceptual changes.

"Specific mathematics assessments that reveal thinking", which we call smart tests, consist of mathematical questions that probe students' mathematical thinking and knowledge of key ideas. Many of the smart tests were first developed in research studies, local and international, and one of the goals of this project is to make the substantial research knowledge about students' mathematical thinking readily available in classrooms.

Research on smart tests revolves around (i) learning and (2) teaching. The learning questions are related to how students learn the individual topics that are tested in smart tests - what are the stages which students go through, what are the cmmon errors and misconceptions, how can these be identified in a computer-based test, and what teaching strategies are most helpful? The teaching questions are about the use of formative assessment - how can formative assessment tools be designed for easy, practical use by teachers, what information do teachers find most useful for their teaching, and what are the outcomes for students of using this system of formative assessment.

The smart test project is based on two hypotheses:

Figure 1. How teachers use smart resources to bring about higher student achievement, and how this encourages teacher growth.

The first research phase was named "Supporting personalised learning in secondary schools through the use of specific mathematics assessments that reveal thinking". This project was conducted at the University of Melbourne by Professor Kaye Stacey, Dr Helen Chick and Dr Vicki Steinle. It was funded by the Australian Research Council Linkage Projects Program LP0882176 in partnership with the Department of Education and Early Childhood Development, Victoria (2008 - 2010). The project officers were Ms Beth Price and Dr Eugene Gvozdenko. Later research has been funded from a variety of sources.

Smart tests are an expert system for teaching

There is a substantial body of research into students' mathematical thinking that has been accumulated internationally, and to which we have contributed ourselves (Stacey: number, algebra, working mathematically; Steinle: number; Chick: chance and data). However, such research needs considerable transformation for maximum effectiveness for teaching. One example of research that was ready to use involved decimal numeration, where Steinle and Stacey's work (Steinle & Stacey, 2003) on the Decimal Comparison Test provided an example of what is needed: a test that is quick to administer to a whole group, easy for teachers to interpret, and which reveals the underlying conceptual development of students, so highlighting areas for teaching to target. The usefulness of Steinle's approach has been widely acclaimed by teachers. This gave us the prototype smart test, which was central to developing the idea of smart tests. The common features of smart tests are that they can be administered to a group in a short time, focus on conceptual understanding of key ideas, and can lead to insightful diagnosis at a level of detail that can inform follow-up teaching.

The creation of smart resources embeds advanced research into artefacts that are easy for practitioners to use, creating 'distributed intelligence' in tools. When planning the teaching of a new topic, the smart tests and diagnosis provide teachers with better knowledge of the mathematical thinking of their current students. Incidentally, yet importantly, teachers also learn about the mathematical knowledge of students more generally.

We hypothesised that using the recommended smart teaching strategies would provide enhanced instruction for the current students, and that this would also build teachers' capacity for future teaching. Testing whether this growth of teachers' Mathematical Pedagogical Content Knowledge (MPCK) actually occurred was a key research aim; there is now international consensus in the research literature that teacher quality and especially their discipline-specific teaching expertise is a major determinant of overall student learning outcomes.
In addition, there are two knowledge transfer outcomes. We intend that teachers' MPCK improves by using the resources. However we also intend that the distributed intelligence in the smart test system provide expert advice for teachers regardless of whether their MPCK is well advanced, and thereby make a personalised learning approach easier and more practical for all teachers.

 

Steinle, V. & Stacey, K. (2003). Grade-Related Trends in the Prevalence and Persistence of Decimal Misconceptions. In Zilliox JT, Dougherty BJ & Pateman NA (Eds), Proceedings of the 2003 Joint Meeting of PME and PMENA. 4 259-266. Honolulu, United States: College of Education, University of Hawai'i.

Publications

Akhtar, Z. & Steinle, V. (2013) Probing students' numerical misconceptions in school algebra. In V. Steinle, L. Ball & C. Bardini (Eds.) MERGA36-2013 Yesterday, today and tomorrow (Proceedings of 36th annual conference of the Mathematics Education Research Group of Australasia). (pp. 36 - 43) Melbourne, VIC: MERGA. (abstract)

Akhtar, Z. & Steinle, V. (2013) Letter as object misconception in junior secondary school algebra. In A. M. Lindmeier & A. Heinze (Eds.) PME37-2013 Proceedings of 37th Conference of the International Group for the Psychology of Mathematics Education. (Vol. 5, p. 4) Kiel, Germany: PME.

Baratta, W., Price, E., Stacey, K., Steinle, V., Gvozdenko, E. (2010) Percentages: The effect of problem structure, number complexity and calculation format. In L. Sparrow, B. Kissane, C. Hurst (Eds.) MERGA33-2010 Shaping the Future of Mathematics Education. Proceedings of 33rd annual conference of the Mathematics Education Research Group of Australasia . (pp. 61 - 68) Fremantle: MERGA. (abstract)

Chick, H. (2009). Teaching the Distributive Law: Is Fruit Salad Still on the Menu?  In R. Hunter, B. Bicknell, T. Burgess (Eds). Proceedings of the 32nd annual conference of the Mathematics Education Research Group of Australasia. (vol 1, pp. 121 - 128) Wellington: MERGA ( abstract )

Chick, H. L. (2010). Aspects of teachers' knowledge for helping students learn about ratio. In L. Sparrow, B. Kissane, & C. Hurst (Eds.) Shaping the Future of Mathematics Education. Proceedings of the 33rd annual conference of the Mathematics Education Research Group of Australasia . (pp. 145-152). Fremantle, WA: MERGA. (abstract)

Chick, H. L., & Baratta, W. (2011). Teachers' strategies for demonstrating fraction and percentage equivalence. In J. Clark, B. Kissane, J. Mousley, T. Spencer, & S. Thornton (Eds.), Curriculum in focus: research guided practice. Proceedings of the 23rd biennial conference of The Australian Association of Mathematics Teachers Inc. and the 34th annual conference of the Mathematics Education Research Group of Australasia Inc . (pp. 183-191). Adelaide: MERGA/AAMT. (abstract)

Price, B., Stacey, K., Steinle, V., Chick, H., Gvozdenko, E. (2009). Getting SMART about Assessment for Learning. In D. Martin, T. Fitzpatrick, R. Hunting, D. Itter, C. Lenard, T. Mills, L. Milne (Eds). Mathematics - Of Prime Importance. Proceedings of the 2009 Annual Conference of the Mathematical Association of Victoria.  (pp. 174 – 181) Mathematical Association of Victoria: Melbourne. (abstract)

Price, B., Stacey, K. Steinle, V., Chick, H., Gvozdenko, E. (2011) Getting SMART About Assessment for Learning in 2011. Reflections , 36 (3), 3 - 7. (abstract)

Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2013). SMART online assessments for teaching mathematics. Mathematics Teaching (235). (pp. 10-15). (abstract)

Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2014). Using percentages to describe and calculate change J. Anderson, M. Cavanagh, & A. Prescott (Eds.), Curriculum in focus: research guided practice. Proceedings of the 37th annual conference of the Mathematics Education Research Group of Australasia Inc . (pp. 517-524). Sydney: AAMT. (abstract)

Stacey, K., Price, B., Steinle, V., Chick, H., Gvozdenko, E. (2009). SMART Assessment for Learning. Paper presented at the Conference of the International Society for Design and Development in Education, Cairns, Australia, September 28 – October 1, 2009.
http://www.isdde.org/isdde/cairns/pdf/papers/isdde09_stacey.pdf (abstract)

Stacey, K., Steinle, V., Wu, M., Pierce, R. (2010) Evaluating automated processes for revealing students' mathematical thinking. Unpublished presentation. Australian Association of Research in Education Annual Conference. http://ocs.sfu.ca/aare/index.php/AARE/AARE_2010/paper/view/2258 (abstract)

Stacey, K., Price, B., Steinle, V (2012) Identifying stages in a learning hierarchy for use in formative assessment - the example of line graphs. In J. Dindyal, Cheng L-P, Ng, S-F (Eds) Mathematics Education: Expanding Horizons Proceedings of the 35th Annual Conference of Mathematics Education Group of Australasia . (pp. 393 - 400). Adelaide: MERGA. (abstract)

Stacey, K. (2013) Bringing research on students' understanding into the classroom through formative assessment In V. Steinle, L. Ball & C. Bardini (Eds.) MERGA36-2013 Yesterday, today and tomorrow (Proceedings of 36th annual conference of the Mathematics Education Research Group of Australasia). (pp. 13 - 20) Melbourne, VIC: MERGA. (abstract)

Stacey, K., Steinle, V., Gvozdenko, E., Price, B. (2013) SMART online formative assessments for teaching mathematics, Curriculum & Leadership Journal, 11(20).

Steinle, V., Gvozdenko, E., Price, B., Stacey, K., Pierce, R. (2009). Investigating students’ numerical misconceptions in algebra.  In R. Hunter, B. Bicknell, T. Burgess (Eds). Proceedings of the 32nd annual conference of the Mathematics Education Research Group of Australasia . (vol 2, pp. 491 - 498) Wellington: MERGA (abstract)

Steinle, V. & Stacey, K. (2012) Teachers' Views of using an on-line, formative assessment system for Mathematics. Pre-proceedings. 12th International Congress on Mathematical Education Topic Study Group 33 8 July - 15 July, 2012 , (pp 6721 - 6730). COEX, Seoul, Korea. (abstract)

Abstracts

Abstract of Chick, H. (2009)
Introducing algebra in high school is a teaching challenge, where appropriate explanations and representations are essential. This study examines teachers' reactions to a page of explanation about the distributive law from a textbook, and investigates their pedagogical content knowledge about the teaching of algebra. Teachers were aware that students have difficulties with algebra, but the study shows that some of their strategies for teaching it are mathematically unsound. In particular, "fruit salad" approaches are still prevalent.


Abstract of Price, B., Stacey, K., Steinle, V., Chick, H., Gvozdenko, E. (2009)
"Specific Mathematics Assessments that Reveal Thinking" - or smart tests - provide teachers with a quick and easy way to conduct assessment for learning. Using the internet, students in years 7, 8, and 9 undertake a short test that is focussed strongly on a topic selected by their teacher. Students' stages of development are diagnosed, and sent to the teacher within minutes. Many tests have been produced and are now being trialled in 7 Victorian schools. Where available, on-line teaching resources are linked to each diagnosis, to guide teachers in moving students to the next stage. This project is sponsored by the Australian Research Council and Victoria's Department of Education and Early Childhood Development.


Abstract of Stacey, K., Price, B., Steinle, V., Chick, H., Gvozdenko, E. (2009)
"Specific Mathematics Assessments that Reveal Thinking," which we abbreviate to "smart tests," provide teachers with a quick and easy way to conduct assessment for learning. Using the internet, students in years 7, 8, and 9 undertake a short test that is focused strongly on a topic selected by their teacher. Students' stages of development are diagnosed, and sent to the teacher immediately. Where available, on-line teaching resources are linked to each diagnosis, to guide teachers in moving students to the next stage. Many smart tests are now being trialled in schools and their impact on students' and teachers' learning is being evaluated. Design issues are discussed.


Abstract of Steinle, V., Gvozdenko, E., Price, B., Stacey, K., Pierce, R. (2009)
Details are provided of simple algebraic items which can be used to detect two particular ways students think about the numerical meanings of letters (in contrast to non-numerical thinking). The data from Year 7 students (n=228) and Year 8 students (n=139) on these items is analysed using response patterns to probe student thinking. Less than 10% of the students were correct on these items whilst the prevalence of the two most common error patterns is 20%-30% at each year level. New response patterns are detected, indicating that further improvements to the items will enable students' thinking to be investigated further.


Abstract of Baratta, W., Price, E., Stacey, K., Steinle, V., Gvozdenko, E. (2010)
This study reports how the difficulty of simple worded percentage problems is affected by the problem structure and the complexity of the numbers involved. We also investigate which methods students know. Results from 677 Year 8 and 9 students are reported. Overall the results indicate that more attention needs to be given to this important topic. Simple unit fraction equivalents seem to be emphasised, at the expense of fundamental definition ("out of a hundred") and apparently easy percentages such as 30%. The draft National Curriculum gives better guidance on the variation amongst percentage problems.


Abstract of Chick, H. L. (2010)
Ratio (and associated topics such as fractions and proportion) is known to be an area of mathematics that students find difficult. Multiplicative thinking is necessary, and students benefit from a wide range of strategies and representations for interpreting ratio. This study examined aspects of teachers' pedagogical content knowledge for teaching ratio, and investigated their knowledge of a typical misconception together with the strategies that they would use for dealing with such a misconception. The nature of the numerical examples that they suggested might be useful in teaching was also examined. Most teachers were able to recognise the misconception, but not all were able to generate examples that might help students to deal with it. Teachers also appeared to have only a limited repertoire of strategies to assist students.


Abstract of Stacey, K., Steinle, V., Wu, M., Pierce, R. (2010)
Understanding student thinking is key to planning effective teaching. Research over three decades has established good knowledge of stages of development and common misunderstandings in many school topics. However, in order for teachers to make formative assessment a normal part of their practice, mapping of student understanding needs to be available for a greater proportion of the school curriculum. Online testing of students is beginning to create substantial databases which could provide relevant information. The aim of this symposium is to present and discuss methods which move towards automation of data analysis.
The three symposium papers will evaluate different methods of analysing data on students' understanding of mathematics. Smart tests ("Specific Mathematics Assessments that Reveal Thinking") are online diagnostic tests with feedback related to student thinking and teaching suggestions made available to the teacher. A very extensive database of student responses is emerging from the smart test development. This presents the challenge of needing to find more efficient methods of identifying new clusters of responses and combinations of answers that will give insights into students' incorrect, incomplete and unexpected mathematical conceptions, as well as the stages of development which are typical in specific topic areas. This complex data set is therefore a useful test bed for data mining methods in educational research.
The characteristics and fitness for purpose of three methods will be discussed in this symposium. Item Response Theory, including


Abstract of Chick, H. L., & Baratta, W. (2011)
Ratio (and associated topics such as fractions and proportion) is known to be an area of mathematics that students find difficult. Multiplicative thinking is necessary, and students benefit from a wide range of strategies and representations for interpreting ratio. This study examined aspects of teachers' pedagogical content knowledge for teaching ratio, and investigated their knowledge of a typical misconception together w ith the strategies that they would use for dealing with such a misconception. The nature of the numerical examples that they suggested might be useful in teaching was also examined. Most teachers were able to recognise the misconception, but not all were able to generate examples that might help students to deal with it. Teachers also appeared to have only a limited repertoire of strategies to assist students.


Abstract of Price, B., Stacey, K. Steinle, V., Chick, H., Gvozdenko, E. (2011)
"Specific Mathematics Assessments that Reveal Thinking" - or smart tests - provide teachers with a quick and easy way to conduct assessment for learning. Using the internet, students in years 7, 8, and 9 undertake a short test that is focussed strongly on a topic selected by their teacher. Students' stages of development are diagnosed, and sent to the teacher within minutes. Many tests have been produced and are now being used in schools. Where available, on-line teaching resources are linked to each diagnosis, to guide teachers in moving students to the next stage. This project was sponsored by the Australian Research Council and Victoria's Department of Education and Early Childhood Development.


Abstract of Steinle, V. & Stacey, K. (2012)
In this paper we discuss some issues arising from teachers' use of an on-line system for formative assessment of their students' understanding of mathematics. The 'smart tests' cover many topics in secondary school mathematics, and are programmed to provide an automated diagnosis of students' stages of learning in specific topics, to inform teaching. In particular, we highlight teachers' views about: the desirability of formative assessment; the use that they make of this information; the provision of feedback; and the effect of using the system on their pedagogical content knowledge and subsequent changes to their practice .Overall, teachers report that use of the tests improves their knowledge of the achievement of individual students and students in general, and that they can use this information in several ways to improve their teaching.


Abstract of Stacey, K. Price, B., Steinle, V (2012)
This paper discusses issues arising in the design of questions to use in an on-line computerbased formative assessment system, focusing on how best to identify the stages of a learning hierarchy for reporting to teachers. Data from several hundred students is used to illustrate how design decisions have been made for a test on interpreting line graphs.


Abstract of Akhtar, Z. & Steinle, V. (2013)
This study was designed to probe students' thinking about which numerical values can be assigned to algebraic letters. The data from students in Year 7 (n=533), Year 8 (n=377) and Year 9 (n=172) was analysed using resonse patterns. The data confirmed that eacvh year contained students with two different misconceptions; Different Letter means Different Number and the Empty Box misconceptions. The findings provide support for the Steinle et al (2009) hypothesis that a previously identified response pattern is a subset of the Empty Box misconception.


Abstract of Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2013)
Assessment is 'part and parcel' of mathematics teaching. Teachers make assessments as they interact with students in many ways and in variety of formats and media. The best assessment has a diagnostic element within its functionality. It is important for assessment to be reported upon within the shortest possible timeframe. Technology can enable sophisticated assessment to be engaged with and reported upon swiftly, and in the classroom. Here is an example of sophisticated online assessment that is available to all teachers .. now.


Abstract of Stacey, K. (2013)
'Specific Mathematics Assessments that Reveal Thinking' (abbreviated as 'smart tests') provide on-line formative assessment of middle years students. They aim to put information from research on students' understanding directly into the hands of teachers, by providing quick automated diagnosis of learning for all students in a class. The Reflections test is used as an example to describe item presentation, evidence identification, and reporting to teachers, and highlight how pedagogical content knowledge can be built.


Abstract of Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2014)
This study reports on the use of formative, diagnostic online assessments for the topic percentages. Two new item formats (drag-drop and slider) are described. About one-third of the school students (Years 7 to 9) could, using a slider, estimate “80% more than” a given length, in contrast with over two-thirds who could estimate “90% of” a given length. While four-fifths of the school students could, using drag-drop cards, choose the 2-step calculation of a reduced price after a 35% discount, only one-third could choose the corresponding 1-step calculation.