The field of medical diagnosis has undergone a significant transformation with the advent of large language models (LLMs), yet the challenges of interpretability within these models remain largely unaddressed. Although LLMs can offer rudimentary explanations for their decision, they lack a comprehensive process to explain why other potential diseases are excluded and to which extent of confidence it made such a decision. This highlights the need for an interpretable LLM solution for diagnosis.
To address this researchers have introduced Chain-of-Diagnosis (CoD) to enhance the interpretability of LLM-based medical diagnostics. CoD transforms the diagnostic process into a diagnostic chain that mirrors a physician’s thought process, providing a transparent reasoning pathway. Additionally, CoD outputs the disease confidence distribution to ensure transparency in decision making. This interpretability makes model diagnostics controllable and aids in identifying critical symptoms for inquiry through the entropy reduction of confidences.
CoD Transforms the opaque decision-making process into a five-step diagnostic chain that reflects a physician’s thought process. 1. Step 1: Symptom Abstraction:, The first step summarizes the symptoms S of the patient’s question 2. Step 2: Disease Recall & Knowledge Integration: Next, CoD identifies the top-K potential diseases based on a disease retriever 3. Step 3: Diagnostic Reasoning: Now CoD generates the diagnostic reasoning process. 4. Step 4: Confidence Assessment: After generating T, CoD generates a confidence distribution. 5. Step 5: Decision Making: In the last step, a confidence threshold τ is set to control the decision making. The diagnostic task involves two decision types: 1) making a diagnosis Adiag(d), where d is the diagnosed disease, and 2) inquiring about a symptom Ainq(s), where s represents the symptom under inquiry.
To evaluate COD, DiagnosisGPT was developed, capable of diagnosing 9,604 diseases. Experimental results demonstrate that DiagnosisGPT outperforms other LLMs on diagnostic benchmarks. Moreover, DiagnosisGPT provides interpretability while ensuring controllability in diagnostic rigor.
Paper : https://arxiv.org/pdf/2407.13301
Code : https://github.com/FreedomIntelligence/Chain-of-Diagnosis