Real-Time Queries on Large Volumes of Safety Text

It is often necessary to parse large volumes of text in the process of carrying out Safety and Risk Management duties. One example of this is the Close Call system, operated in the UK to log safety related incidents on the GB railways. Approximately 300,000 unstructured text reports are added each year. Traditionally, locating and categorizing potential risk indicators in the Close Call text (and other systems like it) has been a human task. Though steps have been taken towards augmenting this with computer-based analysis, real-time feedback has not been possible. This paper will discuss a platform which allows real-time queries on large volumes of text. A novel application of Integer based hashing is applied to n-grams of the text. Using this method, in combination with search optimizations such as binary searching (which would be cumbersome or impossible to perform on unmodified text) it can be shown that pattern matching performance is improved by several orders of magnitude when compared to Brute force matching, or even more developed methods such as the Boyer-Moore algorithm.