Document Processing: Methods for Semantic Text Similarity Analysis

Abdul Wahab Qurashi; Violeta Holmes; Anju P. Johnson

doi:10.1109/INISTA49547.2020.9194665

Document Processing: Methods for Semantic Text Similarity Analysis

Abdul Wahab Qurashi, Violeta Holmes, Anju P. Johnson

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

47 Citations (Scopus)

Abstract

The document text similarity measurement and analysis is a growing application of Natural Language Processing. This paper presents the results of using different techniques for semantic text similarity measurements in documents used for safety-critical systems. The research objective of this work is to measure the degree of semantic equivalence of multi-word sentences for rules and procedures contained in the documents on railway safety. These documents, with unstructured data and different formats, need to be preprocessed and cleaned before the set of Natural Language Processing toolkits, and Jaccard and Cosine similarity metrics are applied. The results demonstrate that it is feasible to automate the process of identifying equivalent rules and procedures and measure similarity of disparate safety-critical documents using Natural language processing and similarity measurement techniques.

Original language	English
Title of host publication	2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings
Subtitle of host publication	INISTA 2020
Editors	Mirjana Ivanovic, Tulay Yildirim, Goce Trajcevski, Costin Badica, Ladjel Bellatreche, Igor Kotenko, Amelia Badica, Burcu Erkmen, Milos Savic
Publisher	Institute of Electrical and Electronics Engineers Inc.
Number of pages	6
ISBN (Electronic)	9781728167992
ISBN (Print)	9781728168005
DOIs	https://doi.org/10.1109/INISTA49547.2020.9194665
Publication status	Published - 11 Sep 2020
Event	2020 International Conference on INnovations in Intelligent SysTems and Applications - Novi Sad, Serbia Duration: 24 Aug 2020 → 26 Aug 2020

Conference

Conference	2020 International Conference on INnovations in Intelligent SysTems and Applications
Abbreviated title	INISTA 2020
Country/Territory	Serbia
City	Novi Sad
Period	24/08/20 → 26/08/20

Access to Document

10.1109/INISTA49547.2020.9194665Licence: Unspecified

Cite this

Qurashi, A. W., Holmes, V., & Johnson, A. P. (2020). Document Processing: Methods for Semantic Text Similarity Analysis. In M. Ivanovic, T. Yildirim, G. Trajcevski, C. Badica, L. Bellatreche, I. Kotenko, A. Badica, B. Erkmen, & M. Savic (Eds.), 2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings: INISTA 2020 [9194665] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/INISTA49547.2020.9194665

Qurashi, Abdul Wahab ; Holmes, Violeta ; Johnson, Anju P. / Document Processing : Methods for Semantic Text Similarity Analysis. 2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings: INISTA 2020. editor / Mirjana Ivanovic ; Tulay Yildirim ; Goce Trajcevski ; Costin Badica ; Ladjel Bellatreche ; Igor Kotenko ; Amelia Badica ; Burcu Erkmen ; Milos Savic. Institute of Electrical and Electronics Engineers Inc., 2020.

@inproceedings{a498ce04e23048ca8b0486ee69fa9e38,

title = "Document Processing: Methods for Semantic Text Similarity Analysis",

abstract = "The document text similarity measurement and analysis is a growing application of Natural Language Processing. This paper presents the results of using different techniques for semantic text similarity measurements in documents used for safety-critical systems. The research objective of this work is to measure the degree of semantic equivalence of multi-word sentences for rules and procedures contained in the documents on railway safety. These documents, with unstructured data and different formats, need to be preprocessed and cleaned before the set of Natural Language Processing toolkits, and Jaccard and Cosine similarity metrics are applied. The results demonstrate that it is feasible to automate the process of identifying equivalent rules and procedures and measure similarity of disparate safety-critical documents using Natural language processing and similarity measurement techniques. ",

keywords = "Cosine Similarity, Jaccard Similarity, Natural Language Processing, Semantic",

author = "Qurashi, {Abdul Wahab} and Violeta Holmes and Johnson, {Anju P.}",

year = "2020",

month = sep,

day = "11",

doi = "10.1109/INISTA49547.2020.9194665",

language = "English",

isbn = "9781728168005",

editor = "Mirjana Ivanovic and Tulay Yildirim and Goce Trajcevski and Costin Badica and Ladjel Bellatreche and Igor Kotenko and Amelia Badica and Burcu Erkmen and Milos Savic",

booktitle = "2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "2020 International Conference on INnovations in Intelligent SysTems and Applications, INISTA 2020 ; Conference date: 24-08-2020 Through 26-08-2020",

}

Qurashi, AW, Holmes, V & Johnson, AP 2020, Document Processing: Methods for Semantic Text Similarity Analysis. in M Ivanovic, T Yildirim, G Trajcevski, C Badica, L Bellatreche, I Kotenko, A Badica, B Erkmen & M Savic (eds), 2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings: INISTA 2020., 9194665, Institute of Electrical and Electronics Engineers Inc., 2020 International Conference on INnovations in Intelligent SysTems and Applications, Novi Sad, Serbia, 24/08/20. https://doi.org/10.1109/INISTA49547.2020.9194665

Document Processing : Methods for Semantic Text Similarity Analysis. / Qurashi, Abdul Wahab; Holmes, Violeta; Johnson, Anju P.

2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings: INISTA 2020. ed. / Mirjana Ivanovic; Tulay Yildirim; Goce Trajcevski; Costin Badica; Ladjel Bellatreche; Igor Kotenko; Amelia Badica; Burcu Erkmen; Milos Savic. Institute of Electrical and Electronics Engineers Inc., 2020. 9194665.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Document Processing

T2 - 2020 International Conference on INnovations in Intelligent SysTems and Applications

AU - Qurashi, Abdul Wahab

AU - Holmes, Violeta

AU - Johnson, Anju P.

PY - 2020/9/11

Y1 - 2020/9/11

N2 - The document text similarity measurement and analysis is a growing application of Natural Language Processing. This paper presents the results of using different techniques for semantic text similarity measurements in documents used for safety-critical systems. The research objective of this work is to measure the degree of semantic equivalence of multi-word sentences for rules and procedures contained in the documents on railway safety. These documents, with unstructured data and different formats, need to be preprocessed and cleaned before the set of Natural Language Processing toolkits, and Jaccard and Cosine similarity metrics are applied. The results demonstrate that it is feasible to automate the process of identifying equivalent rules and procedures and measure similarity of disparate safety-critical documents using Natural language processing and similarity measurement techniques.

AB - The document text similarity measurement and analysis is a growing application of Natural Language Processing. This paper presents the results of using different techniques for semantic text similarity measurements in documents used for safety-critical systems. The research objective of this work is to measure the degree of semantic equivalence of multi-word sentences for rules and procedures contained in the documents on railway safety. These documents, with unstructured data and different formats, need to be preprocessed and cleaned before the set of Natural Language Processing toolkits, and Jaccard and Cosine similarity metrics are applied. The results demonstrate that it is feasible to automate the process of identifying equivalent rules and procedures and measure similarity of disparate safety-critical documents using Natural language processing and similarity measurement techniques.

KW - Cosine Similarity

KW - Jaccard Similarity

KW - Natural Language Processing

KW - Semantic

UR - http://www.scopus.com/inward/record.url?scp=85091973792&partnerID=8YFLogxK

U2 - 10.1109/INISTA49547.2020.9194665

DO - 10.1109/INISTA49547.2020.9194665

M3 - Conference contribution

AN - SCOPUS:85091973792

SN - 9781728168005

BT - 2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings

A2 - Ivanovic, Mirjana

A2 - Yildirim, Tulay

A2 - Trajcevski, Goce

A2 - Badica, Costin

A2 - Bellatreche, Ladjel

A2 - Kotenko, Igor

A2 - Badica, Amelia

A2 - Erkmen, Burcu

A2 - Savic, Milos

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 24 August 2020 through 26 August 2020

ER -

Qurashi AW, Holmes V, Johnson AP. Document Processing: Methods for Semantic Text Similarity Analysis. In Ivanovic M, Yildirim T, Trajcevski G, Badica C, Bellatreche L, Kotenko I, Badica A, Erkmen B, Savic M, editors, 2020 International Conference on INnovations in Intelligent SysTems and Applications, Proceedings: INISTA 2020. Institute of Electrical and Electronics Engineers Inc. 2020. 9194665 doi: 10.1109/INISTA49547.2020.9194665

Document Processing: Methods for Semantic Text Similarity Analysis

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this