I. Modelling the Domain

The first fundamental step was developing a model of the economic domain. By means of an analysis of the relevant literature of the domain, a model was developed that depicts the specifics and guiding ideas of the domain - such as content-related sub-areas, domain-specific approaches or targeted knowledge acquisition processes.

The model forms the basis for the development of an assessment in alignment with the curriculum, for measuring competencies in economic education. Within the framework of analyzing the curricula, the model of the domain  was validated in order to be transferred into a psychometric model. In this phase of the project, a scientific advisory board was convened, which was consulted at regular intervals throughout the project.

 II. Analyzing Curricula

Selected curricula from different states and school types in Germany were studied using a comparative content analysis. Using a code book developed on the basis of the domain model, the curricula were analyzed with regard to subject and learning process categories. In this way, it was possible to examine the extent to which state-specific and school-type-specific differences emerged. With regard to the domain model, it could be shown that it adequately represents the domain of economic education, since the elements of the model were found in all curricula, albeit in different forms.

III. Classifiying existing Measurement Instruments

The development of the instrument was preceded by intensive research of existing German and international questionnaires and testing instruments regarding Economic Literacy, as well as the classification of items for the measurement of Economic Literacy. In the process, content-related focal points, learning process-related categories such as the targeted types of knowledge and learning taxonomy levels, domain-specific approaches, and the real life relevance of eighth-grade students of the instruments were identified and compared. This resulted in a systematic overview that revealed strengths and gaps of existing instruments, which served as a guideline for the generation of questionnaire and test items for the ECON-2022 assessment. 

IV. Developing the Instruments

Based on the previous steps, the authentic ECON 2022 assessment was developed using the empirically proven difficulty-generating features of different degrees of specificity of the economic content, the modeling of the items, as well as the cognitive demand level of the 36 test items. 

In order to meet the requirement of a high level of relevance for eighth-grade students, authentic life situations of adolescents were modeled as a framework. The accompanying questionnaire included scales on economic information sources and learning opportunities, on attitudes and experiences as a consumer, on economic self-efficacy and on the perceived quality of teaching in economic school lessons.

V. Field Test

The field test took place, with a slight delay due to the pandemic, in spring 2021. In close cooperation with the IEA Hamburg, manuals and scripts were developed for those responsible for coordination and test administration at the schools that were drawn.

Using the data collected from the field test with 817 students from North Rhine-Westphalia, the empirical difficulties of the test items, the reliability, and the general fit of the test were determined using the IRT-based partial credit model. The results of the validation showed that the test reliably distinguishes between low and high ability overall. The scales of the questionnaire were subjected to factor and reliability analyses and modified in some places based on the results obtained.  

VI. Validating the Instrument

The second part of the validation of the test instrument was carried out by means of an expert survey. For this purpose, 25 people with expertise in the fields of economics, school and teaching, or test development were acquired and asked to evaluate each test item on the basis of the difficulty-generating features. In addition, the authenticity and usability of the entire instrument were rated. The ratings were then examined for discrepancies among themselves as well as discrepancies with the ratings of the test development team. Items with higher deviations from the ratings were then reexamined and revised as necessary with reference to the results of the empirical item analyses. 

 VII. Main Survey

From March to June 2022, the revised assessment with 31 test items was used at 143 schools in North Rhine-Westphalia. Despite the pandemic, there were only a few dropouts, so that a satisfactory amount of data could be collected. Random quality checks by the project team and the IEA on the test days verified that the tests were conducted properly. Until the delivery of the data by the IEA is completed, the open text answers of the students were coded. For each item, 30 percent of the cases were coded by three independent persons with the aim of achieving high intercoder reliability. This ensured that the open answers could be adequately translated into numerical values.

VIII. Generating the Dataset and Analysis

Data processing and analysis with regard to answering the research questions of the project are expected to be completed at the beginning of 2024 with the publication of the project report. Subsequently, the findings of the assessment will be processed at the level of the individual schools so that school-specific characteristics of the test results can be identified and reported back. This will allow schools to identify and address their development potential. Finally, the project will result in the development of an empirically validated competency model for economic education.