DSTC11: Dialogue System Technology Challenge 11

Track 4: Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems

Click here to register for DSTC11.T4. (now available)

Click here to download DSTC11.T4 data. (now available)

Click here to submit your results. (now available)

Click here to use the baseline model. (now available)

Track Overview

• Track Details

This track consists of two tasks which are explained in more detail below:

1. Participants will develop effective automatic open-ended and multilingual dialogue evaluation metrics that perform similarly when evaluated over a new language.
2. Participants will develop effective automatic open-ended dialogue evaluation metrics that perform robustly when evaluated over paraphrased/back-translated sentences in English.

• Correlated to human judgments - the metrics should produce evaluation scores that well correlate to human judgments (scores) across multiple languages or alternative responses (i.e., back-translated or paraphrased).
• Explainable - the metrics should provide constructive and explicit feedback to the generative models in terms of the quality of their generated responses. For instance, if a generative model is contradicting itself, the evaluation metrics should signal such behavior to the generative models.

• Task 1: Multilingual Automatic Evaluation Metrics

In this task, the goal for participants is to propose effective automatic dialogue evaluation metrics that exhibit previously mentioned properties (Track Details section) and perform well on a multilingual setup (English, Spanish and Chinese). In concrete, participants will propose a single multilingual model obtaining high correlations with human-annotations when evaluated on multilingual dialogues (development set in Provided Datasets section) and perform well on the hidden multilingual test set. Participants are expected to use pre-trained multilingual models and train them to predict multidimensional quality metrics by using self-supervised techniques and optionally fine-tune their system over a subset of the development data.

Finally, participants will then evaluate their models over the development and test sets, and expect to show similar performance, in terms of correlations with human-annotations on the English, Spanish and Chinese utterances. (Note: only dev and test sets will have human-annotations, and only test sets will be manually translated or paraphrased/back-translated to guarantee the correlations with the original human-annotations on the English data).

• Task 2: Robust Automatic Evaluation Metrics

In this task, the goal for participants is to propose robust metrics for automatic evaluation of just English dialogues that exhibit previously mentioned properties (Track Details section) while being robust when dealing with paraphrased/back-translated English sentences. The expected performance must be on par with the correlations with human-annotations obtained over the original sentences. As robustness criteria proposed, paraphrased/back-translated sentences should have the same semantic meaning as the original sentence, but different wording.

Additionally, participants will have the opportunity of testing robustness over alternative machine translations that the organizers will provide. Finally, the influence on the metric will be also evaluated when providing the paraphrased/back-translated current turn sentences instead of the original ones, always along with their respective paraphrased/back-translated context.

During the test phase, hidden and manually curated back-translated test data will be provided to participants to evaluate their proposed metrics.

• Provided Datasets

After the organizers' participation in the CHANEL@JSALT2020 workshop (Rudnicky et al., 2020) at Johns Hopkins University, they have automatically translated back-and-forth (using the same MS Azure translation service) a total of 18 well-known human-human dialogue datasets. These data sets will be used as training data. The total amount of dialogues is 393k (approx. 3M turns).

• DBDC (Higashinaka et al., 2016)
• CMU_DoG (Zhou et al., 2018)
• Cornell Movie-Dialogs (Danescu-Niculescu-Mizil & Lee, 2011)
• DailyDialog (Li et al., 2017)
• DECODE (Nie et al., 2020)
• EmotionLines (Chen et al., 2018)
• EmpathicDialogues (Rashkin et al., 2018)
• Holl-E (Moghe et al., 2018)
• MEENA (Adiwardana et al., 2020)
• MELD (Poria et al., 2019)
• MetalWOz (Lee et al., 2019)
• Movie-DiC (Banchs, 2012)
• PersonaChat (Zhang et al., 2018)
• SentimentLIAR (Upadhayay & Behzadan, 2020)
• Switchboard Coherence (Cervone & Riccardi, 2020)
• Topical-Chat (Gopalakrishnan et al., 2019)
• Wizard of Wikipedia (Dinan et al., 2019)
• Wochat (D'Haro et al., 2016)

• CONVAI2-GRADE (CG) (Huang et al., 2020)
• DAILYDIALOG-GRADE (DH) (Huang et al., 2020)
• DAILYDIALOG-GUPTA (DG) (Gupta et al., 2019)
• DAILYDIALOG-ZHAO (DZ) (Zhao et al., 2020)
• DSTC7 (D7) (Galley et al., 2019)
• EMPATHETIC-GRADE (EG) (Huang et al., 2020)
• FED-DIAL (FD) (Mehri & Eskenazi, 2020a)
• FED-TURN (FT) (Mehri & Eskenazi, 2020a)
• HUMOD (HM) (Merdivan et al., 2020)
• PERSONA-SEE (PS) (See et al., 2019)
• PERSONA-USR (PU) (Mehri & Eskenazi, 2020b)
• PERSONA-ZHAO (PZ) (Zhao et al., 2020)
• TOPICAL-USR (TU) (Mehri & Eskenazi, 2020b)
• JSALT (JS) (Rudnicky et al., 2020)
• CHATEVAL (CS) (Sedoc et al., 2019)
• DSTC10 (D10) (Zhang et al., 2021)

• ECM (Zhou et al., 2018)
• KdConv (Zhou et al., 2020)
• LCCC (Wang et al., 2020)

• Datasets Summary

• Datasets Statistics

• Datasets Information

• Data Format

All data given follows the Data Formats which provides guidelines on how to store, maintain and handle dialogue corpora.

• Baseline Model

The default choice is Deep AM-FM (Zhang et al, 2020) (used for DSTC-10 and previously). This model has been adapted to be able to evaluate multilingual datasets, as well as to work with paraphrased and back-translated sentences.

This project has investigated more recent approaches, based on fine-tuned large language models. Zhang et al note that their approach may be limited due to domain specificity. On the other hand, LLMs are trained from large corpora that in principle are less domain-dependent. This is an empirical question.

All information related to the baseline model, such as code and data, can be found in this GitHub repository.

• Dimensions Evaluation

Considering the annotations available in the development data, the test data will have the following dimensions (annotations) to evaluate in both Task 1 (English, Chinese and Spanish) and Task 2:

• Turn-level: Appropriateness, Content Richness, Grammatical Correctness and Relevance.
• Dialogue-level: Coherence, Engageness/Likeability, Informativeness and Overall.

• Appropriateness - The response is appropriate given the preceding dialogue.
• Content Richness - The response is informative, with long sentences including multiple entities and conceptual or emotional words.
• Grammatical Correctness - Responses are free of grammatical and semantic errors.
• Relevance - Responses are on-topic with the immediate dialogue history.

• Coherence - Throughout the dialogue, is the system maintaining a good conversation flow.
• Engageness/Likeability - Throughout the dialogue, the system displays a likeable personality.
• Informativeness - Throughout the dialogue, the system provides unique and non-generic information.
• Overall - The overall quality of and satisfaction with the dialogue.

• How will we rank all submitted metrics in the test leaderboard?

Each participant's submission will be evaluated separately. For each submission, Spearmen's correlations at dimension-level will be calculated separately for each task. Then, the Spearman correlation scores obtained will be averaged. Finally, the Spearman correlation scores will be ranked.

To calculate the Spearman correlation, in the hidden test data, it will be done between the annotations provided by the participant and the hidden human judgment annotations. In addition to high correlation with human judgments, we also encourage explainability of the metrics.

Without influencing the ranking, results obtained using Pearson's correlation will also be published. Moreover, Spearman and Pearson correlations at language and dataset level will be published.

Hidden test data will be published after the presentation of the results. Additionally, a script will be shared to allow participants to evaluate their own models at different granularity-levels.

• Automatic Evaluation Leaderboard

The leaderboard shows names of submissions and their corresponding Spearman Correlation Coefficients for each development dataset. The name of each column corresponds to an abbreviation of the development datasets respectively.

All the results obtained by the baseline model are very similar, proving that the metric is multilingually adequate, as well as robust when working with paraphrases or back-translations.

Registration Details

To become an official DSTC11 Track 4 participant, you must be registered at this Microsoft Form. Once registered, you will be able to download the datasets and readme documents as well as submit your results at https://chateval.org/dstc11.

There must be only one team per laboratory or research group. The members of the same team must be under a single registration, that is, the team leader must register his entire team by giving their e-mail addresses in addition to his own.

Any updates and information about the tracks will be posted on the DSTC11 official website, or check the DSTC Mailing List.

Submission Details

Before submitting your results, do not forget to Sign Up on the ChatEval website. Only the team leader must register on ChatEval, with the same name and email address entered in the Microsoft Form. Once you have signed up, you can Log In and Submit your evaluations.

There are four different evaluations to test the models, namely:

• Task 1 - Turn-Level
• Task 1 - Dialogue-Level
• Task 2 - Turn-Level
• Task 2 - Dialogue-Level

Each task has annotations at turn-level and dialogue-level, so the models will be evaluated separately at turn-level and dialogue-level independently for each task, they will not be taken into account together at any level. That is, for Task 1 the models at turn-level and at dialogue-level will be evaluated separately, likewise, for Task 2 the models at turn-level and at dialogue-level will be evaluated separately.

If you want, you can participate in as many evaluations as you want. Whether you only want to participate in one, several or all of the evaluations, the scores obtained will be independent, unrelated to the other scores, and will not be combined for the final score. There will be a table with the scores obtained for each of the 4 different evaluations.

You can submit as many score files as you want for each evaluation, but only the last 5 files submitted for each type of evaluation in ChatEval will be valid and will count in the ranking to participate in the competition. Moreover, only the evaluations submitted by the team leader registered in the Microsoft form will be considered and count towards the competition.

In order to submit test data evaluations, they must be named appropriately. Below is the correct way to name the test files that should be sent correctly annotated:

• <team_name>_task1_turn_v<x>.csv
• <team_name>_task1_dial_v<x>.csv
• <team_name>_task2_turn_v<x>.csv
• <team_name>_task2_dial_v<x>.csv

Please specify clearly in the submission name which evaluation it is intended for, the team name in <team_name> and the submission version <x> to identify the submission.

Schedule

• Training/Validation data release: Dec 14, 2022
• Test data release: Mar 29, 2023
• Entry submission deadline: Apr 3, 2023 (23:59 Anywhere on Earth (AoE), UTC-12)
• Final result announcement: Apr 14, 2023
• Paper submission: June 2nd, 2023
• Workshop: September 11 or 12, at SIGDIAL x INLG 2023 in Prague, Czech Republic

Organizers

• Mario Rodríguez-Cantelar (Universidad Politécnica de Madrid, Spain)
• Chen Zhang (National University of Singapore, Singapore)
• Chengguang Tang (Tencent AI Lab, China)
• Ke Shi (Tencent AI Lab, China)
• Sarik Ghazarian (University of Southern California, USA)
• João Sedoc (New York University, USA)
• Luis F. D'Haro (Universidad Politécnica de Madrid, Spain)
• Alexander Rudnicky (Carnegie Mellon University, USA)

Contact

If you have further questions regarding the data, please let us know by the following email address at dstc11-robust-multilingual-automatic-evaluation@googlegroups.com.

Citation

Please cite the paper, code or data from DSTC 11 Track 4:

@inproceedings{rodriguezcantelar2023dstc11t4,

• author = "Mario Rodríguez-Cantelar and Chen Zhang and Chengguang Tang and Ke Shi and Sarik Ghazarian and João Sedoc and Luis Fernando D'Haro and Alexander Rudnicky",
• title = "Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4",
• booktitle = "DSTC11: The Eleventh Dialog System Technology Challenge",
• series = "24th Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)",
• year = 2023,
• month = "September",
• address = "Prague, Czechia"
}

Acknowledgement

This research project is supported by the Comunidad de Madrid through the call Research Grants for Young Investigators from Universidad Politécnica de Madrid (GENIUS:APOYO-JOVENES-21-TAXTYC-32-K61X37).

This work is supported by project BEWORD (PID2021-126061OB-C43) funded by MCIN/AEI/10.13039/501100011033 and, as appropriate, by “ERDF A way of making Europe”, by the “European Union”, and by Programa Propio - Proyectos Semilla: Universidad Politécnica de Madrid (VSEMILLA22LFHE).

We gratefully acknowledge valuable efforts from Tencent AI Lab who supports Chinese translation and annotation of datasets by funding and infrastructure.

Thanks to THU-CoAI (Conversational AI groups from Tsinghua University) for providing their Chinese datasets as part of the challenge data.

Thanks to Unbabel for providing the COMET MTQE scores annotations as part of the challenge data. This contribution was supported by national funds through *Fundação para a Ciência e a Tecnologia* (FCT) with references PRT/BD/152198/2021 and UIDB/50021/2020, and by the P2020 program MAIA led by Unbabel (LISBOA-01-0247-FEDER-045909).

We also want to give thanks to MS Azure services (especially to Irving Kwong) for their sponsorship to continue processing new datasets that could be interesting for the dialogue community.

This research project is supported by the NYU ChatEval Team led by João Sedoc.

This research project is supported in part by a grant from Amazon to Alexander Rudnicky, Carnegie Mellon University.

Thanks to Karthik Ganesan, Sarik Ghazarian, James Hagerty, Zhang Chen and Alex Rudnicky for developing the baseline model as part of the challenge tasks.

This work is supported by the European Commission through Project ASTOUND (101071191 — HORIZON-EIC-2021-PATHFINDERCHALLENGES-01).

FAQ

How much does participate in this Track cost?

This Track is currently free for everyone.

References

• Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2020). Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 1-56.
• Zhang, C., D'Haro, L. F., Friedrichs, T., & Li, H. (2021). MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation. arXiv preprint arXiv:2112.07194.
• Zhang, C., D'Haro, L. F., Banchs, R. E., Friedrichs, T., & Li, H. (2020). Deep AM-FM: Toolkit for Automatic Dialogue Evaluation. In Conversational Dialogue Systems for the Next Decade (pp. 53-69). Springer, Singapore.
• Zhang, C., Sadoc, J., D'Haro, L. F., Banchs, R., & Rudnicky, A. (2021). Automatic Evaluation and Moderation of Open-domain Dialogue Systems. arXiv preprint arXiv:2111.02110.
• Hori, C., & Hori, T. (2017). End-to-end conversation modeling track in DSTC6. arXiv preprint arXiv:1706.07440.
• Huang, L., Ye, Z., Qin, J., Lin, L., & Liang, X. (2020, November). GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 9230-9240).
• Gupta, P., Mehri, S., Zhao, T., Pavel, A., Eskenazi, M., & Bigham, J. P. (2019, September). Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue (pp. 379-391).
• Zhao, T., Lala, D., & Kawahara, T. (2020, July). Designing Precise and Robust Dialogue Response Evaluators. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 26-33).
• Galley, M., Brockett, C., Gao, X., Gao, J., & Dolan, B. (2019). Grounded response generation task at dstc7. In AAAI Dialog System Technology Challenges Workshop.
• Mehri, S., & Eskenazi, M. (2020, July). Unsupervised Evaluation of Interactive Dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue (pp. 225-235).
• Merdivan, E., Singh, D., Hanke, S., Kropf, J., Holzinger, A., & Geist, M. (2020). Human annotated dialogues dataset for natural conversational agents. Applied Sciences, 10(3), 762.
• See, A., Roller, S., Kiela, D., & Weston, J. (2019, June). What makes a good conversation? How controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 1702-1723).
• Mehri, S., & Eskenazi, M. (2020). USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation. arXiv preprint arXiv:2005.00456.
• Rudnicky, A., Banchs, R., D'Haro, L. F., Sedoc, J., Chen, Z., Rodríguez-Cantelar, M., Koh, A., & others. (2020). CHANEL-Metrics: Chat/Dialogue Modeling and Evaluation report. In 2020 Seventh Frederick Jelinek Memorial Summer Workshop.
• Sedoc, J., Ippolito, D., Kirubarajan, A., Thirani, J., Ungar, L., & Callison-Burch, C. (2019, June). Chateval: A tool for chatbot evaluation. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (demonstrations) (pp. 60-65).
• Vinyals, O., & Le, Q. (2015). A neural conversational model. arXiv preprint arXiv:1506.05869.
• Lee, S., Lim, H., & Sedoc, J. (2020). An evaluation protocol for generative conversational systems. arXiv preprint arXiv:2010.12741.
• Higashinaka, R., Funakoshi, K., Kobayashi, Y., & Inaba, M. (2016, May). The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) (pp. 3146-3150).
• Zhou, K., Prabhumoye, S., & Black, A. W. (2018). A dataset for document grounded conversations. arXiv preprint arXiv:1809.07358.
• Danescu-Niculescu-Mizil, C., & Lee, L. (2011). Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. arXiv preprint arXiv:1106.3077.
• Li, Y., Su, H., Shen, X., Li, W., Cao, Z., & Niu, S. (2017). Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
• Nie, Y., Williamson, M., Bansal, M., Kiela, D., & Weston, J. (2020). I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling. arXiv preprint arXiv:2012.13391.
• Chen, S. Y., Hsu, C. C., Kuo, C. C., & Ku, L. W. (2018). Emotionlines: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379.
• Rashkin, H., Smith, E. M., Li, M., & Boureau, Y. L. (2018). Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207.
• Moghe, N., Arora, S., Banerjee, S., & Khapra, M. M. (2018). Towards exploiting background knowledge for building conversation systems. arXiv preprint arXiv:1809.08205.
• Adiwardana, D., Luong, M. T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., ... & Le, Q. V. (2020). Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
• Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2018). Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508.
• Lee, S., Schulz, H., Atkinson, A., Gao, J., Suleman, K., El Asri, L., ... & Li, X. (2019). Multi-domain task-completion dialog challenge. Dialog system technology challenges, 8(9).
• Banchs, R. E. (2012, July). Movie-DiC: a movie dialogue corpus for research and development. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 203-207).
• Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018). Personalizing dialogue agents: I have a dog, do you have pets too?. arXiv preprint arXiv:1801.07243.
• Upadhayay, B., & Behzadan, V. (2020, November). Sentimental LIAR: Extended Corpus and Deep Learning Models for Fake Claim Classification. In 2020 IEEE International Conference on Intelligence and Security Informatics (ISI) (pp. 1-6). IEEE.
• Cervone, A., & Riccardi, G. (2020). Is this dialogue coherent? learning from dialogue acts and entities. arXiv preprint arXiv:2006.10157.
• Gopalakrishnan, K., Hedayatnia, B., Chen, Q., Gottardi, A., Kwatra, S., Venkatesh, A., ... & AI, A. A. (2019, January). Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In INTERSPEECH (pp. 1891-1895).
• Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., & Weston, J. (2018). Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241.
• D'Haro, L. F., Shawar, B. A., & Yu, Z. (2016). REWOCHAT 2016–Shared task description report. In Proceedings of the workshop on collecting and generating resources for chatbots and conversational agents-development and evaluation (RE-WOCHAT) (p. 39).
• Zhou, H., Huang, M., Zhang, T., Zhu, X., & Liu, B. (2018, April). Emotional chatting machine: Emotional conversation generation with internal and external memory. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).
• Zhou, H., Zheng, C., Huang, K., Huang, M., & Zhu, X. (2020). Kdconv: A chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. arXiv preprint arXiv:2004.04100.
• Wang, Y., Ke, P., Zheng, Y., Huang, K., Jiang, Y., Zhu, X., & Huang, M. (2020, October). A large-scale chinese short-text conversation dataset. In CCF International Conference on Natural Language Processing and Chinese Computing (pp. 91-103). Springer, Cham.
• Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A neural framework for MT evaluation. arXiv preprint arXiv:2009.09025.
• Damodaran, P. (2021). Parrot: Paraphrase generation for NLU.
• Zhou, L., Gao, J., Li, D., & Shum, H. Y. (2020). The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1), 53-93.
• Bao, S., He, H., Wang, F., Wu, H., & Wang, H. (2019). Plato: Pre-trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931.
• Mehri, S., Choi, J., D'Haro, L. F., Deriu, J., Eskenazi, M., Gasic, M., ... & Zhang, C. (2022). Report from the nsf future directions workshop on automatic evaluation of dialog: Research directions and challenges. arXiv preprint arXiv:2203.10012.