ChatGPT outperforms students in PAU math exams

The results of a study carried out by the teaching innovation group DSLAB-TI reveal that this artificial intelligence tool obtains better results in areas such as probability and statistics.

Irene Vega

The DSLAB-TI teaching innovation group has conducted a study to investigate the performance of ChatGPT in the EvAU exams, now called PAU (University Access Test), which give access to university throughout Spain. The study has focused on the exams for the subject of Mathematics Applied to Social Sciences in the Community of Madrid.

The main objective of this research was to compare the performance of ChatGPT against students in key areas of mathematics addressed in the test: algebra, mathematical analysis, probability and statistics.

The results obtained, recently published in the scientific journal International Electronic Journal of Mathematics Education, show that ChatGPT outperforms students in areas such as probability and statistics. “The work supports the existing trend of using artificial intelligence as support tools in the classroom. However, as the study shows, they should be used with caution and under teacher supervision because artificial intelligence has limitations when it comes to solving certain types of problems,” says Antonio Alonso, member of the DSLAB-TI group and co-author of the study.

Test evaluation and AI improvements

To carry out the research, EvAU exams taken between 2009 and 2012 in the Community of Madrid were used. The answers provided by ChatGPT to these exams were evaluated by the teaching staff of the DSLAB-TI teaching innovation group. Finally, these evaluations were compared with the students' real grades using statistical techniques and it was observed that artificial intelligence (AI) obtained a lower performance in the algebra and calculations tests. “This is important because it allows us to better understand the strengths and limitations of artificial intelligence in solving mathematical problems and to use these tools with this in mind,” concludes Antonio Alonso.

This study is part of the ongoing work of the DSLAB-TI teaching innovation group at the Rey Juan Carlos University, which seeks to apply Data Science tools to improve teaching.