I started my Ph.D. in April 2023, at the Center for Language and Cognition (CLCG), University of Groningen.
My research mainly focuses on the low-resource conversational tasks, (multilingual) retrieval augmented generation, and the cross-lingual consistency of LLMs.
Some of my papers were published at international NLP conferences. See or for details.
🔥 News
- 2024.09.20: Â Our paper is accepted by EMNLP 2024 Main Conference! Meanwhile, the collaboration work SIFo benchmark for evaluating LLM sequential-instruction following ability is accepted as the Findings of EMNLP 2024!
- 2024.06.21: Â New preprint paper released: Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation.
- 2023.12.10: Â The paper was also selected for Outstanding Paper award in the Multilinguality and Linguistic Diversity track at EMNLP 2023!
- 2023.12.06: Â We received Best Data Award for our EMNLP paper from The 1st GenBench Workshop!
- 2023.10.06: Â Paper accepted by EMNLP 2023! Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
- 2023.04.01:  Ph.D. journey started at the University of Groningen, supervised by Arianna Bisazza and Raquel Fernández.
đź“ť Publications
Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation
Jirui Qi*, Gabriele Sarti*, Raquel Fernández, Arianna Bisazza
Ensuring the verifiability of model answers is a fundamental challenge for retrieval-augmented generation (RAG) in the question answering (QA) domain. Recently, self-citation prompting was proposed to make large language models (LLMs) generate citations to supporting documents along with their answers. However, self-citing LLMs often struggle to match the required format, refer to non-existent sources, and fail to faithfully reflect LLMs’ context usage throughout the generation. In this work, we present MIRAGE –Model Internals-based RAG Explanations – a plug-and-play approach using model internals for faithful answer attribution in RAG applications. MIRAGE detects context-sensitive answer tokens and pairs them with retrieved documents contributing to their prediction via saliency methods. We evaluate our proposed approach on a multilingual extractive QA dataset, finding high agreement with human answer attribution. On open-ended QA, MIRAGE achieves citation quality and efficiency comparable to self-citation while also allowing for a finer-grained control of attribution parameters. Our qualitative evaluation highlights the faithfulness of MIRAGE’s attributions and underscores the promising application of model internals for RAG answer attribution.
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
Jirui Qi, Raquel Fernández, Arianna Bisazza
Multilingual large-scale Pretrained Language Models (PLMs) have been shown to store considerable amounts of factual knowledge, but large variations are observed across languages. With the ultimate goal of ensuring that users with different language backgrounds obtain consistent feedback from the same model, we study the cross-lingual consistency (CLC) of factual knowledge in various multilingual PLMs. To this end, we propose a Ranking-based Consistency (RankC) metric to evaluate knowledge consistency across languages independently from accuracy. Using this metric, we conduct an in-depth analysis of the determining factors for CLC, both at model level and at language-pair level. Among other results, we find that increasing model size leads to higher factual probing accuracy in most languages, but does not improve cross-lingual consistency. Finally, we conduct a case study on CLC when new factual associations are inserted in the PLMs via model editing. Results on a small sample of facts inserted in English reveal a clear pattern whereby the new piece of knowledge transfers only to languages with which English has a high RankC score.
- The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models
Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, Maarten de Rijke
Findings of EMNLP 2024 - Multi-mask label mapping for prompt-based learning
Jirui Qi, Richong Zhang, Jaein Kim, Junfan Chen, Wenyi Qin, Yongyi Mao
AAAI-23 - Parameter-free Automatically Prompting: A Latent Pseudo Label Mapping Model for Prompt-based Learning
Jirui Qi, Richong Zhang, Junfan Chen, Jaein Kim, Yongyi Mao
Findings of EMNLP 2022 - Cross Domain Few-Shot Learning via Meta Adversarial Training
Jirui Qi, Richong Zhang, Chune Li, Yongyi Mao
arXiv preprint
🎮 Demos
- Interested in having a competition against LMs? Try our demo here and see if you can beat them!
đź“– Educations
- 2023.04 - Current: Ph.D. Candidate, Center for Language and Cognition, Faculty of Art, University of Groningen
- 2020.09 - 2023.01: Master, Beihang University
- 2016.09 - 2020.06: Undergraduate, Beijing Jiaotong University
- 2013.09 - 2016.06: Senior High, The High School Affiliated to Renmin University of China (RDFZ)