Qualitative Judgement of Research Impact: Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research

Fionn Murtagh, Michael Orlov, Boris Mirkin

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitative peer-based judgement. The goal of this article is to propose availing of such a well positioned construct as domain taxonomy as a tool for directly assessing the scope and quality of research. We first show how taxonomies can be used to analyze the scope and perspectives of a set of research projects or papers. Then we proceed to define a research team or researcher’s rank by those nodes in the hierarchy that have been created or significantly transformed by the results of the researcher. An experimental test of the approach in the data analysis domain is described. Although the concept of taxonomy seems rather simplistic to describe all the richness of a research domain, its changes and use can be made transparent and subject to open discussions.

LanguageEnglish
Pages5-28
Number of pages24
JournalJournal of Classification
Volume35
Issue number1
Early online date22 Mar 2018
DOIs
Publication statusPublished - Apr 2018
Externally publishedYes

Fingerprint

impact research
Qualitative Research
Taxonomy
taxonomy
Research Personnel
Research
Metric
Appeal
Data analysis
appeal
data analysis
research project
Evaluation
Vertex of a graph
Framework
Judgment
evaluation

Cite this

@article{bba96c93203d499db4e88a150a87f66b,
title = "Qualitative Judgement of Research Impact: Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research",
abstract = "The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitative peer-based judgement. The goal of this article is to propose availing of such a well positioned construct as domain taxonomy as a tool for directly assessing the scope and quality of research. We first show how taxonomies can be used to analyze the scope and perspectives of a set of research projects or papers. Then we proceed to define a research team or researcher’s rank by those nodes in the hierarchy that have been created or significantly transformed by the results of the researcher. An experimental test of the approach in the data analysis domain is described. Although the concept of taxonomy seems rather simplistic to describe all the richness of a research domain, its changes and use can be made transparent and subject to open discussions.",
keywords = "Multicriteria decision making, Rank aggregation, Research impact, Scientometrics, Semantic analysis, Taxonomy, Stratification",
author = "Fionn Murtagh and Michael Orlov and Boris Mirkin",
note = "This is a post-peer-review, pre-copyedit version of an article published in Journal of Classification. The final authenticated version is available online at: http://dx.doi.org/10.1007/s00357-018-9247-0",
year = "2018",
month = "4",
doi = "10.1007/s00357-018-9247-0",
language = "English",
volume = "35",
pages = "5--28",
journal = "Journal of Classification",
issn = "0176-4268",
publisher = "Springer New York",
number = "1",

}

Qualitative Judgement of Research Impact : Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research. / Murtagh, Fionn; Orlov, Michael; Mirkin, Boris.

In: Journal of Classification, Vol. 35, No. 1, 04.2018, p. 5-28.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Qualitative Judgement of Research Impact

T2 - Journal of Classification

AU - Murtagh, Fionn

AU - Orlov, Michael

AU - Mirkin, Boris

N1 - This is a post-peer-review, pre-copyedit version of an article published in Journal of Classification. The final authenticated version is available online at: http://dx.doi.org/10.1007/s00357-018-9247-0

PY - 2018/4

Y1 - 2018/4

N2 - The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitative peer-based judgement. The goal of this article is to propose availing of such a well positioned construct as domain taxonomy as a tool for directly assessing the scope and quality of research. We first show how taxonomies can be used to analyze the scope and perspectives of a set of research projects or papers. Then we proceed to define a research team or researcher’s rank by those nodes in the hierarchy that have been created or significantly transformed by the results of the researcher. An experimental test of the approach in the data analysis domain is described. Although the concept of taxonomy seems rather simplistic to describe all the richness of a research domain, its changes and use can be made transparent and subject to open discussions.

AB - The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitative peer-based judgement. The goal of this article is to propose availing of such a well positioned construct as domain taxonomy as a tool for directly assessing the scope and quality of research. We first show how taxonomies can be used to analyze the scope and perspectives of a set of research projects or papers. Then we proceed to define a research team or researcher’s rank by those nodes in the hierarchy that have been created or significantly transformed by the results of the researcher. An experimental test of the approach in the data analysis domain is described. Although the concept of taxonomy seems rather simplistic to describe all the richness of a research domain, its changes and use can be made transparent and subject to open discussions.

KW - Multicriteria decision making

KW - Rank aggregation

KW - Research impact

KW - Scientometrics

KW - Semantic analysis

KW - Taxonomy

KW - Stratification

UR - http://www.scopus.com/inward/record.url?scp=85044271156&partnerID=8YFLogxK

U2 - 10.1007/s00357-018-9247-0

DO - 10.1007/s00357-018-9247-0

M3 - Article

VL - 35

SP - 5

EP - 28

JO - Journal of Classification

JF - Journal of Classification

SN - 0176-4268

IS - 1

ER -