Visualization of patient behavior from natural language recommendations

The visualization of procedural knowledge from textual documents using 3D animation may be a way to improve understanding. We are interested in applying this approach to documents relating to patient education for bariatric surgery: a domain with challenging textual documents describing behavior recommendations that contain few procedural steps and leave much commonsense knowledge unspecified. In this work we look at how to automatically capture knowledge from a range of differently phrased recommendations and use that with implicit knowledge about compliance and violation, such that the recommendations can be visualized using 3D animations. Our solution is an end-to-end system that automates this process via: analysis of input recommendations to uncover their conditional structure; the use of commonsense knowledge and deontic logic to generate compliance and violation rules; and mapping of this knowledge to update a default knowledge base, which is used to generate appropriate sequences of visualizations. In this paper we overview this approach and demonstrate its potential.