In (Vinyals and Le 2015), human evaluation is conducted on a set of 200 hand-picked prompts. We make this dataset available for comparison.

This evaluation dataset contains a random subset of 200 prompts from the English OpenSubtitles 2009 dataset (Tiedemann 2009).

This is an English as a second language conversational learning from ESL, Inc (https://eslfast.com). The ChatEval team selected 3 turns of 200 conversations. 

ESL 3-turn

The DBDC dataset consists of a series of text-based conversations between a human and a chatbot where the human was aware they were chatting with a computer (Higashinaka et al. 2016).

DBDC

The random Twitter test set is a random subset of 200 prompts from the ParlAi Twitter derived test set.

twitter

The Cornell Movie Dialogue Corpus (Danescu-Niculescu-Mizil and Lee, 2011) contains accurate speaker annotations. We use 1000 prompts selected by Baheti et al., 2018.

Cornell Movie Dialogue Corpus

This evaluation dataset consists of 360 model responses, where six different models each respond to 60 prompts. The prompts are sourced from TopicalChat (Gopalakrishnan et al. 2019). Three humans annotated for each model response, based on 6 different scoring categories. (Mehri & Eskenazi 2020)

Topical-USR

This evaluation dataset consists of 300 model responses, where five different models each respond to 60 prompts. The prompts are sourced from PersonaChat (Zhang et al. 2018). Three humans annotated for each model response, based on 6 different scoring categories. (Mehri & Eskenazi 2020)

Persona-USR

This evaluation dataset provides model responses and multiple references to the DailyDialog dataset(Li et. al., 2017), open-sourced by Gupta et. al., 2019

DailyDialog-Gupta

This evaluation dataset provides model responses and multiple references to the DailyDialog dataset(Li et. al., 2017), open-sourced by Zhao et. al., 2020

DailyDialog-Zhao

This evaluation dataset provides model responses and multiple references to the PersonaChat dataset (Zhang et. al., 2018), model responses and annotations open-sourced by Zhao et. al., 2020

Systems