International Schools' Assessment (ISA)

Contact us

Australian Council for Educational Research

19 Prospect Hill Rd,
Camberwell, VIC 3124, Australia

T: +613 9277 5555
F: +613 9277 5757
E: isa@acer.edu.au

• What kind of descriptive and diagnostic information does the ISA provide in the reports and what kinds of statistical and quantitative information do the reports provide?

• What is the background to ISA norming?

• What about the comparability of cohorts?

• If our students gain low marks on the ISA, is it going to affect our position or they are just diagnostic tests?

• How are the writing assessments scored, what rubric is used?

• How do we inform parents about the content of the ISA?

• Why use scale scores?

• How do we use the scale scores to add value and measure growth?

• What does a scale score of 500 represent?

• Why is there an artificial floor and ceiling for the individual student results?

• Why are there no age comparisons?

• Can you provide a ranking of our school against other ISA participating schools?

What kind of descriptive and diagnostic information does the ISA provide in the reports and what kinds of statistical and quantitative information do the reports provide?

All the reports use quantitative data, though some have qualitative data as well (the individual student reports). Viewt Sample Reports to see the type of quantitative data that we provide. The ISA Interactive Diagnostic Report gives more detailed analysis in quantitative and descriptive form.

The kinds of quantitative comparisons we provide may be different to those you have seen on other standardised tests: we report in terms of a 'scale score' (a standardised score) which is applicable across grade levels and from one year to the next. We provide comparisons of the individual student with all others in their grade level; comparisons of an individual class with all other students at a grade level, and school-wide comparisons of each grade with all other participating schools, other 'like' schools, boys, girls and English- and non-English-speaking subgroups.

What is the background to ISA norming?

Norming groups are usually constructed for tests that are "off the shelf": that is, the test is produced and norming group is established so that, whenever you want to do the test, you can compare your school's results to a wider group. The norming of the ISA consists in the results we get from each administration. Each school’s aggregated results are compared on the school report with the results of all the other schools that took the ISA on the same occasion (and with sub-groups from the same large group). So when you look at the list of participating schools on our website, that is, in effect, the norming group (minus the small number of schools who, for one reason or another, don't want their school's name published at this stage). In addition, our methodology allows us to provide standardised scale scores for individuals, classes and schools, which are stable over time and across year levels, regardless of which other schools take part in a particular year. While the same test forms will be used in October and the following February, each group of test-takers is treated as a separate norming group. For most schools following a northern hemisphere timetable, one would expect the performance of students in February to be, on average, stronger than that in October, though for the older students the differences are likely to be quite small.

In some other assessments - for example, the TORCH tests of reading comprehension and the IOWAs - the test is administered initially and then at intervals (usually every few years) to a large sample of students, and normative results generated, so that subsequent users of the test have a standard reference point. This is necessary when different users of the test can administer it at different times, and in small numbers. However, such a norming is not necessary for a test like the ISA, where thousands of students take the test within the same two-week period: in fact what we have in the ISA is regular, annual norming against a population as close to each school as one could hope to get.

The exception to this in the ISA reporting suite is the Grade 10 National norms reports that are provided to schools with Grade 8, 9 or 10 participants. Here, the reference group is a random sample of 4,500 15-year-old students from each of the OECD Programme for International Student Assessment (PISA)-participating countries. The samples in each country are very carefully drawn to represent the entire school population of 15-year-olds in that country, in all school sectors (except international schools). Grade 8, 9 and 10 cohorts from international schools taking the ISA are compared in their performance in reading and mathematical literacy with these large national cohorts. PISA is administered every three years. In ISA 2002 and 2003, the norming population on the National norms report came from the PISA 2000 administration. In ISA 2004, 2005-6 and 2006-7, the National norms report was based on the PISA 2003 administration. In ISA 2007-8, 2008-9 and 2009-10, the National norms report was based on the PISA 2006 administration. From ISA 2010-11, the National Norms report has been based on the PISA 2009 administration. 65 countries participated in the 2009 PISA data collection with a total of around 470,000 students, making up close to 90% of the world economy.

Unfortunately (from the ISA's point of view) only 15-year-olds are assessed in PISA. While we know that the Grade 9s and 10s (and possibly 8s) doing the ISA are roughly comparable in age to this group, it is not valid to make comparisons with the other Grades that take the ISA (3, 4, 5, 6 and 7). There's more detail about the Grade 10 Norm reports in the Guide to Reports that schools receive with their results. View Sample Reports

What about the comparability of cohorts?

On our school-level reports we provide comparisons of your school (mean and standard deviations) with 'All other schools' and with 'Other like schools' (on All, Boys, Girls, English speaking background students and Non-English speaking background students).

The 'Other like schools' definition is currently based on the proportion of English speaking background and non-English speaking background students in the (aggregated) tested group in each school.

In the 2006-7 administration of the ISA, the proportion of students from English-speaking and non-English-speaking backgrounds was analysed, and "like school" groups were formed to give an optimum balance of students and numbers of schools across an optimum number of groups. The same "like school" groups have been used in each ISA Administration since then. Each school has been assigned to one of four groups:

•Group 1: 25% of students or less in the school from an English speaking background;

•Group 2: between 26% and 40% of students in the school from an English speaking background;

• Group 3: between 41% and 55% of students in the school from an English speaking background;

• Group 4: more than 55% of students in the school from an English speaking background.

If our students gain low marks on the ISA, is it going to affect our position or are they just diagnostic tests?

The ISA is not conceived of as a high stakes assessment for students. Major decisions about students' futures will not be made on the basis of the results. Nor would the program be 'high stakes' for schools. No league tables will ever be published by ACER. ACER would not release the test results to any central agency that could use them to hold schools accountable in any way, far less to threaten or punish schools, in the way that standardised testing programs are used in some jurisdictions.

The ISA is an achievement test for students in international schools. It is designed to serve a number of purposes for these schools, enabling them to:

* evaluate instructional programs against objective evidence of student performance, to identify gaps in student understanding, and to measure growth in learning, between grade levels and from year to year within one grade level;
* provide normative data in relation to selected populations;
* compare subgroup performance (for example, girls and boys; students from different language backgrounds) to see where there may be unexpected results and try to understand them;
* measure individual students' achievement in order to reflect on and address strengths and weaknesses; and
* monitor an individual's or a cohort's progress over time.

It is important to interpret the results of this assessment together with other information about students' achievement provided by on-going classroom assessment and perhaps other external assessments. It is not the purpose of the ISA to provide a final grade for a student at the conclusion of their academic year.

Student results are confidential and are released to the school only.

How are the writing assessments scored, what rubric is used?

We do not publish marking rubrics as such. ISA markers work with a highly detailed rubric and accompanying sample scripts and the scoring guides are not available for individual schools. The report forms carry summary descriptors that relate exactly to the scoring rubric. The results are reported with descriptive information about what the figures mean, both in terms of a 'described proficiency scale' (a general description of the level of achievement for each of mathematical literacy, reading and writing), and a detailed description of the skills tested and achieved for each question in math and reading, and for each of the component scores in writing.

The marking guides/rubrics/rating scales developed for the ISA writing are the result of over 20 years of theoretical and empirical research into writing assessment at ACER and elsewhere. They are based on the principles of developmental assessment, which conceives of the student as developing in each different area of learning along an underlying continuum: assessments, in this model, are conceived of as occasions to gather evidence about the point (or more likely the region) on a continuum that the student has reached.

Regarding the choice of criteria for the ISA, there were several considerations, including coverage of the key features of writing (a balance between technical and ideational features) and providing some diagnostic information. Our aim was to give a reasonably broad yet diagnostically useful picture of a student's achievement.

The rubrics and guides used in operational marking are significantly more detailed than those that are published on the class reports. Each score - in a domain as complex as writing - is inevitably an on-balance judgment; and the descriptors are deliberately designed NOT to give a sense of 'checking off' features of writing. Every student receives scores from two markers: one for narrative and one for expository writing. Then the accuracy of the marker is checked against standard scripts by leading markers.

How do we inform parents about the content of the ISA?

The best way to inform parents and others about the content of the ISA tests is to show sample items that give a flavour of the testing focus and format. Another means of informing parents is to use the descriptors that are attached to the class report, which describe the skills and understandings assessed in each item. We realise that this is not as immediate as being able to see the actual tasks; on the other hand, the focus is less on the individual task than the kind of skills and understandings it represents. What we are interested in measuring is students' proficiency in a domain, represented by KINDS of questions, rather than the particular question that happened to be on this test.

Why use scale scores?

The advantage of using scale scores rather than raw scores or percentage reporting is that the scale makes it possible to compare results from different tests, as long as they are measuring the same variable (that is, the same collection of skills, knowledge and understanding). Using scale scores, we can compare the performance of students in different grades in the same year, of a particular grade from one year to the next, or of the same students from one year to the next.

How do we use the scale scores to add value and measure growth?

The ISA scale scores allow you to make direct comparisons between grade levels and over time. The class reports show the scale scores for each student for reading, mathematical literacy and writing. You can use the Excel spreadsheet to group the students in different ways. You can directly compare the scale scores of students from one ISA testing year to the next to measure individual growth and you can calculate the mean scale scores separately for whatever subgroups you wish to create.

The ISA school report shows the mean scale score for all other schools and for other like schools for all students and broken down into sub-groups by gender and English Language background. The average growth in scale scores between grade levels can be obtained by comparing the mean scale scores for different grade levels for all students and by the subgroups given. You can then see what normal progress between 3rd and 5th grade looks like in scale scores. You can then identify students who may have made average progress of an increase of, for example, 200 score points but still sit below the mean scale score for their grade and students with scores above the mean who have possibly made little progress since they previously sat the ISA.

With an increasing number of international schools using the ISA, you may be able to obtain the ISA scores students received at previous schools. You can add these scale scores to your data.

Download Documents for assistance in interpreting ISA results:

• Benchmark Interpretations of the ISA data for classroom teachers
• Diagnostic Interpretations of the ISA data for classroom teachers
• ISA School Leaders and Administrators Guide

What does a scale score of 500 represent?

The scale score of 500 represents the average score for the 15 year olds who sat PISA . The ISA has directly linked its tests to PISA at Grades 7 and 9/10 by using some of the same items in the ISA as were used in PISA . The ISA tests for all the grade levels are also linked through the use of some common items, allowing us to put all the ISA items on the same scale as PISA . We have created our own ISA/PISA scale. Therefore, lower grades have lower average scale scores on the PISA/ISA scale. It does not matter which ISA test a student takes, their scale score will show their performance relative to the 500 mean for PISA 15 year olds.

Any student who has a scale score of 500 is performing at the same level as the average PISA 15 year old. If a grade 7 student and a grade 10 student both have a scale score of 500, regardless of the fact that they sat different tests, they have the same level of ability in relation to the skills that ISA measures. Scale scores can be directly compared across all the ISA tests in one year and over time.

Why is there an artificial floor and ceiling for the individual student results?

We have an artificial floor to deal with students who answered almost no questions correctly on the ISA. The problem with allocating scale scores in these cases is that we have very little information (only any questions answered correctly) on which to base the scale score. This makes the measurement error very large for these scores and can cause some scale scores to fall below zero. Because this is a very difficult idea for parents to understand we impose an artificial floor of zero. This makes sense to the parents and also allows them to correctly interpret that their student has done very badly. In a similar fashion a student who has answered almost all the items in the test correctly cannot be given an accurate scale score because we do not know how much more they could have done had they been given harder items. Again, the measurement error for students receiving near perfect scores on a test is large and can cause highly inflated scale scores. We impose an artificial ceiling to show these students are clearly achieving well above their peers rather than report extraordinarily high scale scores that contain a large degree of measurement error.

Why are there no age comparisons?

Our research to date has suggested minimal differences in the scores of students of different ages within a grade cohort, except that the students in the oldest 6 month bracket have a consistently lower mean score and this becomes a progressively greater drop in performance at higher grade levels. The youngest students in a cohort have lower mean score in grade 3 but quickly catch up by grade 4 or grade 5.

Can you provide a ranking of our school against other ISA participating schools?

Rankings don't necessarily tell you much, which is why we avoid them. Your school could be 9th, for example, and actually a lot behind the others in terms of score points or 9th with a difference of only 9 score points between the first 9 schools.