Welcome to the Comparative Judgement Research Consortium!

Comparative Judgement Research Consortium

An interdisciplinary consortium of researchers who use and develop comparative judgement methods.

This international group brings together researchers and practitioners from diverse fields including mathematics, education, psychology, and computer science. We are dedicated to advancing the understanding and application of comparative judgement through collaborative research and practice. The group was founded in 2023 by Ian Jones, Marie-Josee Bisson, and Rowland Seymour.

Join us for regular meetings, take part in our online reading group, access our comprehensive resource lists, and connect with a vibrant community committed to innovation and excellence in comparative judgement. Explore our site to learn more about our work and how you can get involved!

Joining the Group

What is Comparative Judgement

Comparative judgment is a method used in assessment and evaluation to compare and rank different items or performances based on their perceived quality or merit. Instead of assigning absolute scores or grades to individual items, comparative judgment involves comparing pairs of items and determining which is better or of higher quality.

Here’s a simplified explanation of how comparative judgment works:

How comparative judgement works

Assessors are presented with pairs to compare (e.g., essays, project proposals, artwork).
Assessors choose which item is better based on criteria or professional judgement.
Many pairwise comparisons are collected across assessors and items.
The collected judgments are analysed to produce a ranked ordering of items.

One advantage of comparative judgment is that it allows for a more nuanced and reliable assessment by leveraging the human ability to make qualitative distinctions. It can be especially useful when evaluating complex or subjective tasks where assigning numerical scores may be challenging.

There are both manual and automated ways to implement comparative judgment. Manual methods involve people making the comparisons, while automated systems use algorithms to analyze the data and derive rankings. Automated systems can efficiently handle large-scale assessments, making comparative judgment a versatile approach in various fields, including education, art, and professional evaluations.

Acknowledgements

This group was set up through a National Centre for Research Methods Special Interest Group grant and then sustained with support from the London Mathematical Society, Bath Spa University, an EPSRC Mathematical Sciences Discipline Hopping Grant [UKRI2389], and a UKRI Future Leaders Fellowship [MR/X034992/1].