Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods

Andrew JR Simpson; Gerard Roma; Emad M Grais; Russell D Mason; Chris Hummersone; Antoine Liutkus; Mark D Plumbley

doi:10.1109/EUSIPCO.2016.7760551

Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods

Andrew JR Simpson, Gerard Roma, Emad M Grais, Russell D Mason, Chris Hummersone, Antoine Liutkus, Mark D Plumbley

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Citations (Scopus)

Abstract

Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.

Original language	English
Title of host publication	2016 24th European Signal Processing Conference (EUSIPCO)
Publisher	IEEE
Pages	1763-1767
Number of pages	5
ISBN (Electronic)	9780992862657
DOIs	https://doi.org/10.1109/EUSIPCO.2016.7760551
Publication status	Published - 1 Dec 2016
Externally published	Yes
Event	24th European Signal Processing Conference - Budapest, Hungary Duration: 29 Aug 2016 → 2 Sep 2016 Conference number: 24 http://www.eusipco2016.org/ (Link to Conference Website )

Publication series

Name
ISSN (Electronic)	2076-1465

Conference

Conference	24th European Signal Processing Conference
Abbreviated title	EUSIPCO 2016
Country/Territory	Hungary
City	Budapest
Period	29/08/16 → 2/09/16
Internet address	http://www.eusipco2016.org/ (Link to Conference Website )

Access to Document

10.1109/EUSIPCO.2016.7760551Licence: Unspecified

Cite this

@inproceedings{b5cba6ab82a743ea90015ae07309a101,

title = "Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods",

abstract = "Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.",

keywords = "Audio source separation, BSSeval, Hypothesis test, SiSEC",

author = "Simpson, {Andrew JR} and Gerard Roma and Grais, {Emad M} and Mason, {Russell D} and Chris Hummersone and Antoine Liutkus and Plumbley, {Mark D}",

year = "2016",

month = dec,

day = "1",

doi = "10.1109/EUSIPCO.2016.7760551",

language = "English",

publisher = "IEEE",

pages = "1763--1767",

booktitle = "2016 24th European Signal Processing Conference (EUSIPCO)",

address = "United States",

note = "24th European Signal Processing Conference, EUSIPCO 2016 ; Conference date: 29-08-2016 Through 02-09-2016",

url = "http://www.eusipco2016.org/",

}

Simpson, AJR, Roma, G, Grais, EM, Mason, RD, Hummersone, C, Liutkus, A & Plumbley, MD 2016, Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods. in 2016 24th European Signal Processing Conference (EUSIPCO). IEEE, pp. 1763-1767, 24th European Signal Processing Conference, Budapest, Hungary, 29/08/16. https://doi.org/10.1109/EUSIPCO.2016.7760551

TY - GEN

T1 - Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods

AU - Simpson, Andrew JR

AU - Roma, Gerard

AU - Grais, Emad M

AU - Mason, Russell D

AU - Hummersone, Chris

AU - Liutkus, Antoine

AU - Plumbley, Mark D

N1 - Conference code: 24

PY - 2016/12/1

Y1 - 2016/12/1

N2 - Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.

AB - Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.

KW - Audio source separation

KW - BSSeval

KW - Hypothesis test

KW - SiSEC

U2 - 10.1109/EUSIPCO.2016.7760551

DO - 10.1109/EUSIPCO.2016.7760551

M3 - Conference contribution

SP - 1763

EP - 1767

BT - 2016 24th European Signal Processing Conference (EUSIPCO)

PB - IEEE

T2 - 24th European Signal Processing Conference

Y2 - 29 August 2016 through 2 September 2016

ER -

Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this