In (Vinyals and Le 2015), human evaluation is conducted on a set of 200 hand-picked prompts. We make this dataset available for comparison.

This evaluation dataset contains a random subset of 200 prompts from the English OpenSubtitles 2009 dataset (Tiedemann 2009).

The DBDC dataset consists of a series of text-based conversations between a human and a chatbot where the human was aware they were chatting with a computer (Higashinaka et al. 2016).

DBDC

The random Twitter test set is a random subset of 200 prompts from the ParlAi Twitter derived test set.

twitter

The Cornell Movie Dialogue Corpus (Danescu-Niculescu-Mizil and Lee, 2011) contains accurate speaker annotations. We use 1000 prompts selected by Baheti et al., 2018.

Systems