Research on smart tests

Research Goals

One of the major challenges in teaching mathematics is to provide personalised learning opportunities that meet the needs of individual students. Many students have misconceptions or gaps in their knowledge which are a barrier for further learning but often go unaddressed. The smart tests aim to give teachers readily useable information about each student's understandings to help the teachers select instruction that will bring about specific conceptual changes.

"Specific mathematics assessments that reveal thinking", which we call smart tests, consist of mathematical questions that probe students' mathematical thinking and knowledge of key ideas. Many of the smart tests were first developed in research studies, local and international, and one of the goals of this project is to make the substantial research knowledge about students' mathematical thinking readily available in classrooms.

Research on smart tests revolves around (i) learning and (2) teaching. The learning questions are related to how students learn the individual topics that are tested in smart tests - what are the stages which students go through, what are the cmmon errors and misconceptions, how can these be identified in a computer-based test, and what teaching strategies are most helpful? The teaching questions are about the use of formative assessment - how can formative assessment tools be designed for easy, practical use by teachers, what information do teachers find most useful for their teaching, and what are the outcomes for students of using this system of formative assessment.

The smart test project is based on two hypotheses:

Figure 1. How teachers use smart resources to bring about higher student achievement, and how this encourages teacher growth.

The first research phase was named "Supporting personalised learning in secondary schools through the use of specific mathematics assessments that reveal thinking". This project was conducted at the University of Melbourne by Professor Kaye Stacey, Dr Helen Chick and Dr Vicki Steinle. It was funded by the Australian Research Council Linkage Projects Program LP0882176 in partnership with the Department of Education and Early Childhood Development, Victoria (2008 - 2010). The project officers were Ms Beth Price and Dr Eugene Gvozdenko. Later research has been funded from a variety of sources.

Smart tests are an expert system for teaching

There is a substantial body of research into students' mathematical thinking that has been accumulated internationally, and to which we have contributed ourselves (Stacey: number, algebra, working mathematically; Steinle: number; Chick: chance and data). However, such research needs considerable transformation for maximum effectiveness for teaching. One example of research that was ready to use involved decimal numeration, where Steinle and Stacey's work (Steinle & Stacey, 2003) on the Decimal Comparison Test provided an example of what is needed: a test that is quick to administer to a whole group, easy for teachers to interpret, and which reveals the underlying conceptual development of students, so highlighting areas for teaching to target. The usefulness of Steinle's approach has been widely acclaimed by teachers. This gave us the prototype smart test, which was central to developing the idea of smart tests. The common features of smart tests are that they can be administered to a group in a short time, focus on conceptual understanding of key ideas, and can lead to insightful diagnosis at a level of detail that can inform follow-up teaching.

The creation of smart resources embeds advanced research into artefacts that are easy for practitioners to use, creating 'distributed intelligence' in tools. When planning the teaching of a new topic, the smart tests and diagnosis provide teachers with better knowledge of the mathematical thinking of their current students. Incidentally, yet importantly, teachers also learn about the mathematical knowledge of students more generally.

We hypothesised that using the recommended smart teaching strategies would provide enhanced instruction for the current students, and that this would also build teachers' capacity for future teaching. Testing whether this growth of teachers' Mathematical Pedagogical Content Knowledge (MPCK) actually occurred was a key research aim; there is now international consensus in the research literature that teacher quality and especially their discipline-specific teaching expertise is a major determinant of overall student learning outcomes.
In addition, there are two knowledge transfer outcomes. We intend that teachers' MPCK improves by using the resources. However we also intend that the distributed intelligence in the smart test system provide expert advice for teachers regardless of whether their MPCK is well advanced, and thereby make a personalised learning approach easier and more practical for all teachers.

 

Steinle, V. & Stacey, K. (2003). Grade-Related Trends in the Prevalence and Persistence of Decimal Misconceptions. In Zilliox JT, Dougherty BJ & Pateman NA (Eds), Proceedings of the 2003 Joint Meeting of PME and PMENA. 4 259-266. Honolulu, United States: College of Education, University of Hawai'i.

Publications

Akhtar, Z. & Steinle, V. (2013) Probing students' numerical misconceptions in school algebra. In V. Steinle, L. Ball & C. Bardini (Eds.) MERGA36-2013 Yesterday, today and tomorrow (Proceedings of 36th annual conference of the Mathematics Education Research Group of Australasia). (pp. 36 - 43) Melbourne, VIC: MERGA. (abstract)

Akhtar, Z. & Steinle, V. (2013) Letter as object misconception in junior secondary school algebra. In A. M. Lindmeier & A. Heinze (Eds.) PME37-2013 Proceedings of 37th Conference of the International Group for the Psychology of Mathematics Education. (Vol. 5, p. 4) Kiel, Germany: PME.

Akhtar, Z. & Steinle, V. (2017) The Prevalence of the 'letter as object' misconception in junior secondary students. In A. Downton, S. Livy & J. Hall (Eds.) 40 years on: We are still learning (Proceedings of 40th annual conference of the Mathematics Education Research Group of Australasian. (pp.77-84) Melbourne, VIC: MERGA.(abstract)

Baratta, W., Price, E., Stacey, K., Steinle, V., Gvozdenko, E. (2010) Percentages: The effect of problem structure, number complexity and calculation format. In L. Sparrow, B. Kissane, C. Hurst (Eds.) MERGA33-2010 Shaping the Future of Mathematics Education. Proceedings of 33rd annual conference of the Mathematics Education Research Group of Australasia . (pp. 61 - 68) Fremantle: MERGA. (abstract)

Chick, H. (2009). Teaching the Distributive Law: Is Fruit Salad Still on the Menu?  In R. Hunter, B. Bicknell, T. Burgess (Eds). Proceedings of the 32nd annual conference of the Mathematics Education Research Group of Australasia. (vol 1, pp. 121 - 128) Wellington: MERGA ( abstract )

Chick, H. L. (2010). Aspects of teachers' knowledge for helping students learn about ratio. In L. Sparrow, B. Kissane, & C. Hurst (Eds.) Shaping the Future of Mathematics Education. Proceedings of the 33rd annual conference of the Mathematics Education Research Group of Australasia . (pp. 145-152). Fremantle, WA: MERGA. (abstract)

Chick, H. L., & Baratta, W. (2011). Teachers' strategies for demonstrating fraction and percentage equivalence. In J. Clark, B. Kissane, J. Mousley, T. Spencer, & S. Thornton (Eds.), Curriculum in focus: research guided practice. Proceedings of the 23rd biennial conference of The Australian Association of Mathematics Teachers Inc. and the 34th annual conference of the Mathematics Education Research Group of Australasia Inc . (pp. 183-191). Adelaide: MERGA/AAMT. (abstract)

Guzman, Maria Alejandra. (2014) The smart test system: teachers’ views about this formative assessment for mathematics. Masters Research thesis. https://minerva-access.unimelb.edu.au/handle/11343/44090 (abstract)

McKee, Sara Jane. (2017) Using teacher capacity to measure improvement in key elements of teachers’ mathematical pedagogical content knowledge. Doctorate thesis. https://minerva-access.unimelb.edu.au/handle/11343/123566 (abstract)

Price, B., Stacey, K., Steinle, V., Chick, H., Gvozdenko, E. (2009). Getting SMART about Assessment for Learning. In D. Martin, T. Fitzpatrick, R. Hunting, D. Itter, C. Lenard, T. Mills, L. Milne (Eds). Mathematics - Of Prime Importance. Proceedings of the 2009 Annual Conference of the Mathematical Association of Victoria.  (pp. 174 – 181) Mathematical Association of Victoria: Melbourne. (abstract)

Price, B., Stacey, K. Steinle, V., Chick, H., Gvozdenko, E. (2011) Getting SMART About Assessment for Learning in 2011. Reflections , 36 (3), 3 - 7. (abstract)

Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2013). SMART online assessments for teaching mathematics. Mathematics Teaching (235). (pp. 10-15). (abstract)

Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2014). Using percentages to describe and calculate change J. Anderson, M. Cavanagh, & A. Prescott (Eds.), Curriculum in focus: research guided practice. Proceedings of the 37th annual conference of the Mathematics Education Research Group of Australasia Inc . (pp. 517-524). Sydney: AAMT. (abstract)

Quenette, Jacqueline. (2014) Diagnostic testing and changes to teaching practice in Year 9 mathematics classes. Masters Research thesis. https://minerva-access.unimelb.edu.au/handle/11343/43027 (abstract)

Stacey, K., Price, B., Steinle, V., Chick, H., Gvozdenko, E. (2009). SMART Assessment for Learning. Paper presented at the Conference of the International Society for Design and Development in Education, Cairns, Australia, September 28 – October 1, 2009.
http://www.isdde.org/isdde/cairns/pdf/papers/isdde09_stacey.pdf (abstract)

Stacey, K., Steinle, V., Wu, M., Pierce, R. (2010) Evaluating automated processes for revealing students' mathematical thinking. Unpublished presentation. Australian Association of Research in Education Annual Conference. http://ocs.sfu.ca/aare/index.php/AARE/AARE_2010/paper/view/2258 (abstract)

Stacey, K., Price, B., Steinle, V (2012) Identifying stages in a learning hierarchy for use in formative assessment - the example of line graphs. In J. Dindyal, Cheng L-P, Ng, S-F (Eds) Mathematics Education: Expanding Horizons Proceedings of the 35th Annual Conference of Mathematics Education Group of Australasia . (pp. 393 - 400). Adelaide: MERGA. (abstract)

Stacey, K. (2013) Bringing research on students' understanding into the classroom through formative assessment In V. Steinle, L. Ball & C. Bardini (Eds.) MERGA36-2013 Yesterday, today and tomorrow (Proceedings of 36th annual conference of the Mathematics Education Research Group of Australasia). (pp. 13 - 20) Melbourne, VIC: MERGA. (abstract)

Stacey, K., Steinle, V., Gvozdenko, E., Price, B. (2013) SMART online formative assessments for teaching mathematics, Curriculum & Leadership Journal, 11(20).

Steinle, V., Gvozdenko, E., Price, B., Stacey, K., Pierce, R. (2009). Investigating students’ numerical misconceptions in algebra.  In R. Hunter, B. Bicknell, T. Burgess (Eds). Proceedings of the 32nd annual conference of the Mathematics Education Research Group of Australasia . (vol 2, pp. 491 - 498) Wellington: MERGA (abstract)

Steinle, V. & Stacey, K. (2012) Teachers' Views of using an on-line, formative assessment system for Mathematics. Pre-proceedings. 12th International Congress on Mathematical Education Topic Study Group 33 8 July - 15 July, 2012 , (pp 6721 - 6730). COEX, Seoul, Korea. (abstract)

Abstracts

Abstract of Chick, H. (2009)
Introducing algebra in high school is a teaching challenge, where appropriate explanations and representations are essential. This study examines teachers' reactions to a page of explanation about the distributive law from a textbook, and investigates their pedagogical content knowledge about the teaching of algebra. Teachers were aware that students have difficulties with algebra, but the study shows that some of their strategies for teaching it are mathematically unsound. In particular, "fruit salad" approaches are still prevalent.


Abstract of Price, B., Stacey, K., Steinle, V., Chick, H., Gvozdenko, E. (2009)
"Specific Mathematics Assessments that Reveal Thinking" - or smart tests - provide teachers with a quick and easy way to conduct assessment for learning. Using the internet, students in years 7, 8, and 9 undertake a short test that is focussed strongly on a topic selected by their teacher. Students' stages of development are diagnosed, and sent to the teacher within minutes. Many tests have been produced and are now being trialled in 7 Victorian schools. Where available, on-line teaching resources are linked to each diagnosis, to guide teachers in moving students to the next stage. This project is sponsored by the Australian Research Council and Victoria's Department of Education and Early Childhood Development.


Abstract of Stacey, K., Price, B., Steinle, V., Chick, H., Gvozdenko, E. (2009)
"Specific Mathematics Assessments that Reveal Thinking," which we abbreviate to "smart tests," provide teachers with a quick and easy way to conduct assessment for learning. Using the internet, students in years 7, 8, and 9 undertake a short test that is focused strongly on a topic selected by their teacher. Students' stages of development are diagnosed, and sent to the teacher immediately. Where available, on-line teaching resources are linked to each diagnosis, to guide teachers in moving students to the next stage. Many smart tests are now being trialled in schools and their impact on students' and teachers' learning is being evaluated. Design issues are discussed.


Abstract of Steinle, V., Gvozdenko, E., Price, B., Stacey, K., Pierce, R. (2009)
Details are provided of simple algebraic items which can be used to detect two particular ways students think about the numerical meanings of letters (in contrast to non-numerical thinking). The data from Year 7 students (n=228) and Year 8 students (n=139) on these items is analysed using response patterns to probe student thinking. Less than 10% of the students were correct on these items whilst the prevalence of the two most common error patterns is 20%-30% at each year level. New response patterns are detected, indicating that further improvements to the items will enable students' thinking to be investigated further.


Abstract of Baratta, W., Price, E., Stacey, K., Steinle, V., Gvozdenko, E. (2010)
This study reports how the difficulty of simple worded percentage problems is affected by the problem structure and the complexity of the numbers involved. We also investigate which methods students know. Results from 677 Year 8 and 9 students are reported. Overall the results indicate that more attention needs to be given to this important topic. Simple unit fraction equivalents seem to be emphasised, at the expense of fundamental definition ("out of a hundred") and apparently easy percentages such as 30%. The draft National Curriculum gives better guidance on the variation amongst percentage problems.


Abstract of Chick, H. L. (2010)
Ratio (and associated topics such as fractions and proportion) is known to be an area of mathematics that students find difficult. Multiplicative thinking is necessary, and students benefit from a wide range of strategies and representations for interpreting ratio. This study examined aspects of teachers' pedagogical content knowledge for teaching ratio, and investigated their knowledge of a typical misconception together with the strategies that they would use for dealing with such a misconception. The nature of the numerical examples that they suggested might be useful in teaching was also examined. Most teachers were able to recognise the misconception, but not all were able to generate examples that might help students to deal with it. Teachers also appeared to have only a limited repertoire of strategies to assist students.


Abstract of Stacey, K., Steinle, V., Wu, M., Pierce, R. (2010)
Understanding student thinking is key to planning effective teaching. Research over three decades has established good knowledge of stages of development and common misunderstandings in many school topics. However, in order for teachers to make formative assessment a normal part of their practice, mapping of student understanding needs to be available for a greater proportion of the school curriculum. Online testing of students is beginning to create substantial databases which could provide relevant information. The aim of this symposium is to present and discuss methods which move towards automation of data analysis.
The three symposium papers will evaluate different methods of analysing data on students' understanding of mathematics. Smart tests ("Specific Mathematics Assessments that Reveal Thinking") are online diagnostic tests with feedback related to student thinking and teaching suggestions made available to the teacher. A very extensive database of student responses is emerging from the smart test development. This presents the challenge of needing to find more efficient methods of identifying new clusters of responses and combinations of answers that will give insights into students' incorrect, incomplete and unexpected mathematical conceptions, as well as the stages of development which are typical in specific topic areas. This complex data set is therefore a useful test bed for data mining methods in educational research.
The characteristics and fitness for purpose of three methods will be discussed in this symposium. Item Response Theory, including


Abstract of Chick, H. L., & Baratta, W. (2011)
Ratio (and associated topics such as fractions and proportion) is known to be an area of mathematics that students find difficult. Multiplicative thinking is necessary, and students benefit from a wide range of strategies and representations for interpreting ratio. This study examined aspects of teachers' pedagogical content knowledge for teaching ratio, and investigated their knowledge of a typical misconception together w ith the strategies that they would use for dealing with such a misconception. The nature of the numerical examples that they suggested might be useful in teaching was also examined. Most teachers were able to recognise the misconception, but not all were able to generate examples that might help students to deal with it. Teachers also appeared to have only a limited repertoire of strategies to assist students.


Abstract of Price, B., Stacey, K. Steinle, V., Chick, H., Gvozdenko, E. (2011)
"Specific Mathematics Assessments that Reveal Thinking" - or smart tests - provide teachers with a quick and easy way to conduct assessment for learning. Using the internet, students in years 7, 8, and 9 undertake a short test that is focussed strongly on a topic selected by their teacher. Students' stages of development are diagnosed, and sent to the teacher within minutes. Many tests have been produced and are now being used in schools. Where available, on-line teaching resources are linked to each diagnosis, to guide teachers in moving students to the next stage. This project was sponsored by the Australian Research Council and Victoria's Department of Education and Early Childhood Development.


Abstract of Steinle, V. & Stacey, K. (2012)
In this paper we discuss some issues arising from teachers' use of an on-line system for formative assessment of their students' understanding of mathematics. The 'smart tests' cover many topics in secondary school mathematics, and are programmed to provide an automated diagnosis of students' stages of learning in specific topics, to inform teaching. In particular, we highlight teachers' views about: the desirability of formative assessment; the use that they make of this information; the provision of feedback; and the effect of using the system on their pedagogical content knowledge and subsequent changes to their practice .Overall, teachers report that use of the tests improves their knowledge of the achievement of individual students and students in general, and that they can use this information in several ways to improve their teaching.


Abstract of Stacey, K. Price, B., Steinle, V (2012)
This paper discusses issues arising in the design of questions to use in an on-line computerbased formative assessment system, focusing on how best to identify the stages of a learning hierarchy for reporting to teachers. Data from several hundred students is used to illustrate how design decisions have been made for a test on interpreting line graphs.


Abstract of Akhtar, Z. & Steinle, V. (2013)
This study was designed to probe students' thinking about which numerical values can be assigned to algebraic letters. The data from students in Year 7 (n=533), Year 8 (n=377) and Year 9 (n=172) was analysed using response patterns. The data confirmed that eacvh year contained students with two different misconceptions; Different Letter means Different Number and the Empty Box misconceptions. The findings provide support for the Steinle et al (2009) hypothesis that a previously identified response pattern is a subset of the Empty Box misconception.


Abstract of Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2013)
Assessment is 'part and parcel' of mathematics teaching. Teachers make assessments as they interact with students in many ways and in variety of formats and media. The best assessment has a diagnostic element within its functionality. It is important for assessment to be reported upon within the shortest possible timeframe. Technology can enable sophisticated assessment to be engaged with and reported upon swiftly, and in the classroom. Here is an example of sophisticated online assessment that is available to all teachers .. now.


Abstract of Stacey, K. (2013)
'Specific Mathematics Assessments that Reveal Thinking' (abbreviated as 'smart tests') provide on-line formative assessment of middle years students. They aim to put information from research on students' understanding directly into the hands of teachers, by providing quick automated diagnosis of learning for all students in a class. The Reflections test is used as an example to describe item presentation, evidence identification, and reporting to teachers, and highlight how pedagogical content knowledge can be built.


Abstract of Guzman, Maria Alejandra (2014)
Given the continuous challenge of improving the quality of students’ learning of mathematics at the school level, and the consensus on the impact that teaching practices have on student learning, several countries have incorporated the development of assessments to directly influence teaching practices, known as formative assessments. However, this response has faced challenges, such as the need for more evidence about which particular formative assessment strategies may be beneficial for the learning of both students and teachers. Accordingly, this research is focused on providing evidence about teachers’ opinions from their experience as users of a particular formative assessment, the ‘Specific Mathematics Assessments that Reveal Thinking’ also known as the Smart Test System, developed at The University of Melbourne. The research aims to answer three fundamental questions. First, how do teachers perceive the quality of the Smart Tests items and the diagnosis provided from the Smart Test System? Second, to what extent, and how, do teachers utilize the information provided by the Smart Test System to change their teaching? Thirdly, to what extent, and how, do teachers report on their learning from their use of the Smart Test System? A mixed methodology approach was utilized according to the research questions, and an on-line self-administered survey was used as the method of data collection. The findings suggest that a majority of teachers who participated in this study have had a positive experience as users of the Smart Test System. Along with reporting that the Smart Tests items and the diagnosis provided are of high quality, many teachers mention formative uses of the Smart Test System. These uses mainly correspond to adjusting their planning and differentiation of their practices according to students’ needs. Almost all teachers in this study stated that they have learnt something from the Smart Test System. The findings of this study provide some understanding about how the Smart Test System fulfils its formative purposes. The results contribute to determining teachers’ opinions about the scope and the ways that they use the Smart Test System to adjust their practices. The results also identify some challenges that could be attended to in order to maximize the potential benefit from the use of the Smart Test System, and which can be considered to develop further formative assessment initiatives.


Abstract of Price, B., Stacey, K., Steinle, V., Gvozdenko, E. (2014)
This study reports on the use of formative, diagnostic online assessments for the topic percentages. Two new item formats (drag-drop and slider) are described. About one-third of the school students (Years 7 to 9) could, using a slider, estimate “80% more than” a given length, in contrast with over two-thirds who could estimate “90% of” a given length. While four-fifths of the school students could, using drag-drop cards, choose the 2-step calculation of a reduced price after a 35% discount, only one-third could choose the corresponding 1-step calculation.


Abstract of Quenette, Jacqueline (2014)
Teachers can use various means, including diagnostic tests, to determine their students’ knowledge. It is of interest to know the ways in which teachers interpret and act upon such diagnostic information. The aim of this study was to examine the use of a particular diagnostic testing system by six teachers in Year 9 mathematics classrooms. The focus diagnostic system was the SMART system (Specific Mathematics Assessment that Reveal Thinking), which provides teachers with an online diagnostic test, diagnostic analysis and teaching advice. This study focused on the use of the SMART system in two topics, linear equations and linear graphs. The participant teachers were interviewed before each topic to ascertain the ways in which they gathered knowledge about student understanding, current and intended teaching practices, and how they met individual student’s learning needs. On conclusion of each topic, participants completed a questionnaire and an interview to determine if any changes had been made. The teachers found the SMART system gave them some useful data on their students. The diagnostic analysis revealed gaps or misunderstandings in some students’ knowledge, the teachers realised that they could no longer assume that all students had the requisite prior knowledge. Through this discovery, teaching practice changed in a number of ways. First, the teachers were able to decide on a better starting point for the particular topic. For example, if many students did not have the expected prior knowledge the teachers began the topic with earlier concepts. Second, teachers could identify groups of students with similar learning needs and these students could be provided with activities that supported their learning. Furthermore, for some teachers it changed their view of students mathematical ability from, ‘some students do not have the ability to learn maths’ to, ‘these students have gaps in their knowledge and if these gaps or misconceptions are addressed they could progress to more complex concepts’. Most significantly, teachers reported becoming more prepared with appropriate materials for either individual students or groups of students. Hence the SMART system supported teachers to cater for individual student needs by highlighting the learning needs of students.


Abstract of Akhtar, Z. & Steinle, V. (2017)
The study investigated students’ thinking about the use of letters in algebra. Responses of over 1400 Australian secondary school students to a set of three algebra items were analysed to determine the prevalence of the ‘letter as object’ misconception. We estimate that 50% to 80% of Year 7 students bring this misconception to their initial learning of algebra. Over 50% of Year 8 students and over 40% of Year 9 students in the sample also selected responses consistent with this misconception.


Abstract of McKee, Sara Jane. (2017)
School systems world-wide are investing increasing resources in assessment of students. The challenge is to gain value for teachers from this process. This study examined how we can use a construct of teacher capacity to identify improvements in teachers’ knowledge of Mathematics, their knowledge of the curriculum, their understanding of student’s mathematical thinking, and their ability to design and implement effective mathematics instruction as a result of using online diagnostic assessments (SMART tests- Specific Mathematical Assessments that Reveal Thinking.) Two principal challenges were addressed in this study: the first concerns how to translate a theoretical construct of teacher capacity in ways that truly reflect the professionally informed judgement and disposition to act. The second challenge was to design and use measures that would show improvement of teacher capacity over time as a result of using SMART tests. This study used innovative approaches involving teacher self-reports that were supported by evidence derived from a content specific questionnaire, related to the four elements of teacher capacity identified previously. The research study was carried out in the researcher’s school. 14 teachers used SMART tests over the course of one semester. All teachers showed improvement in teacher capacity as a result of implementing SMART tests, however improved teacher capacity was most evident amongst accomplished and expert teachers. The use of SMART tests also had a direct impact on teacher planning and informed classroom instruction. The study concludes with recommendations for future research in schools and in pre-service teacher education, utilising online diagnostic assessments of students. This study provides insight into what teacher capacity means in an educational setting, and how leaders in schools can effectively measure and improve teacher capacity in a school setting.