ChatEval is a scientific framework for evaluating open domain chatbots. Researchers can submit their trained models to effortlessly receive comparisons with baselines and prior work. Since all evaluation code is open source, we ensure evaluation is performed in a standardized and transparent way. Additionally, open source baseline models and an ever growing groups public evaluation sets are available for public use.

Upload Model

How much does ChatEval cost?

ChatEval is currently free for academic researchers. It is actively developed by the NLP Group of the University of Pennyslvania.

Is there an online demo video?

You can find a video tutorial for ChatEval here.

How is automatic chatbot model assesment and evaluation performed?

Read more about how automatic model assessment and evaluation is done here.

How was ChatEval built?

The ChatEval webapp is built using Django and React (front-end) using Magnitude word embeddings format for evaluation. Our source code is available on Github.