Skip to Main Content
 

Global Search Box

 
 
 
 

Files

File List


Dissertation_JieZhao_0106.pdf (9.51 MB)

ETD Abstract Container

Abstract Header

Towards Building Trustworthy Automatic Question Answering Systems

Zhao, Jie
http://orcid.org/0000-0003-0848-0224
http://rave.ohiolink.edu/etdc/view?acc_num=osu1609944750779603

Abstract Details

2021, Doctor of Philosophy, Ohio State University, Computer Science and Engineering.
Automatic Question Answering (QA) is the problem of finding answers to natural language questions asked by human users. In recent years, research on QA has gone through tremendous progress with the emergence of new knowledge sources and the application of deep learning technologies. State-of-the-art QA systems are catching up with or even exceeding human-level accuracy on many benchmark datasets. Although it is of great importance to pursue high accuracy on these benchmarks, the fact is that in practice, a QA system will hardly ever be omniscient. Therefore, an ideal QA system should respect the social right of users to understand and verify automatically produced answers, and eventually earn their trust. However, trustworthiness has not received enough attention so far in the design of most current QA systems. Trustworthiness is a comprehensive goal that requires a combination of efforts. For example, in this dissertation, we identify and target three important aspects that are essential for building trustworthy QA: (1) Be aware of uncertainty: the system should be self-aware of its uncertainties, both during training and inference time, and have good confidence in the produced answers. (2) Be interpretable: the QA system’s cognition and reasoning should be understandable and verifiable by general users, and should be able to provide intermediate interpretations of how answers are produced, rather than behaving entirely like black-boxes. (3) Leverage user interaction: the system should also consider historical user interaction data and leverage user feedback to improve question answering performance or enrich QA-related user experiences. In this dissertation, we introduce some techniques we developed to address some of the above aspects as well as some promising ideas for future work towards trustworthy QA. First, we discuss how to improve QA uncertainty awareness and reduce false-positive answers in face of incomplete knowledge sources. Specifically, we design a novel group- level objective function to jointly train a QA system to predict whether a question is answerable and to score the answer candidates, which helps to improve the overall answer triggering performance of the QA system. In addition, we develop techniques to measure the uncertainties of negative training examples via a question-description relevance prediction task, and show that the estimated uncertainties can be combined with adversarial learning to improve the performance of a software-domain QA system. Second, we investigate how to enhance the interpretability of a Product Question Answering system that finds answer/review sentences given customer questions. Unlike most end-to-end QA systems that work as black-boxes, our designed framework mines rich keyword representations of a question , which are intuitive to human users and can be directly used as input to a transparent keyword-based search module, enabling the whole process to be effective while preserving good interpretability. We also study the problem of generating natural language questions from Knowledge Graph (KG) queries. This task can improve QA system interpretability by allowing users to understand QA reasoning over abstract KG semantics, and can also potentially facilitate QA interactivity since it is easy for users to provide verification and feedback on natural language questions. In particular, we design a model that can effectively utilize existing simple questions, including directly related sub-questions and distantly related pseudo sub-questions, to generate a more complex question. Third, we use two case studies to demonstrate that richer QA-related services can be enabled using historical user interaction data. In one case study, we extract distantly supervised data from historical emails to train a system, which can recommend sales content to sales Representatives based on question intents in customer emails and then collect feedback on how the customers engage with the sales content. In the second case study, we leverage historical interactions between community QA users to train a system, which routes newly asked questions to expert users who are most likely to provide answers. Building trustworthy QA systems is a long-term goal that still needs a lot of effort. In future work, we discuss the limitations of our developed approaches and point out promising directions for future improvements. First, we briefly review some recent uncertain estimation and calibration techniques and discuss possible directions to leverage them to further improve the uncertainty-awareness of a QA system. We also discuss how other natural language generation tasks can be designed to provide explanations and to enhance automatic QA interpretability. Another future direction of research is to study QA under an interactive setting, where a QA system can present the reasoning steps to users using intermediate simpler questions decomposed from a given complex question, and collect feedback from users that can help find the final answers.
Huan Sun, Dr (Advisor)
Srinivasan Parthasarathy, Dr (Committee Member)
Yu Su, Dr (Committee Member)
Emily Patterson, Dr (Committee Member)
205 p.
Computer Science

Recommended Citations

Citations

  • Zhao, J. (2021). Towards Building Trustworthy Automatic Question Answering Systems [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1609944750779603

    APA Style (7th edition)

  • Zhao, Jie. Towards Building Trustworthy Automatic Question Answering Systems. 2021. Ohio State University, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu1609944750779603.

    MLA Style (8th edition)

  • Zhao, Jie. "Towards Building Trustworthy Automatic Question Answering Systems." Doctoral dissertation, Ohio State University, 2021. http://rave.ohiolink.edu/etdc/view?acc_num=osu1609944750779603

    Chicago Manual of Style (17th edition)

osu1609944750779603
548
© 2020, all rights reserved.
This open access ETD is published by The Ohio State University and OhioLINK.