Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods

Andrew JR Simpson, Gerard Roma, Emad M Grais, Russell D Mason, Chris Hummersone, Antoine Liutkus, Mark D Plumbley

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.
Original languageEnglish
Title of host publication2016 24th European Signal Processing Conference (EUSIPCO)
PublisherIEEE
Pages1763-1767
Number of pages5
ISBN (Electronic)9780992862657
DOIs
Publication statusPublished - 1 Dec 2016
Externally publishedYes
Event24th European Signal Processing Conference - Budapest, Hungary
Duration: 29 Aug 20162 Sep 2016
Conference number: 24
http://www.eusipco2016.org/ (Link to Conference Website )

Publication series

Name
ISSN (Electronic)2076-1465

Conference

Conference24th European Signal Processing Conference
Abbreviated titleEUSIPCO 2016
CountryHungary
CityBudapest
Period29/08/162/09/16
Internet address

Fingerprint

Source separation
Statistical methods
Statistics

Cite this

Simpson, A. JR., Roma, G., Grais, E. M., Mason, R. D., Hummersone, C., Liutkus, A., & Plumbley, M. D. (2016). Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods. In 2016 24th European Signal Processing Conference (EUSIPCO) (pp. 1763-1767). IEEE. https://doi.org/10.1109/EUSIPCO.2016.7760551
Simpson, Andrew JR ; Roma, Gerard ; Grais, Emad M ; Mason, Russell D ; Hummersone, Chris ; Liutkus, Antoine ; Plumbley, Mark D. / Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods. 2016 24th European Signal Processing Conference (EUSIPCO). IEEE, 2016. pp. 1763-1767
@inproceedings{b5cba6ab82a743ea90015ae07309a101,
title = "Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods",
abstract = "Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.",
keywords = "Audio source separation, BSSeval, Hypothesis test, SiSEC",
author = "Simpson, {Andrew JR} and Gerard Roma and Grais, {Emad M} and Mason, {Russell D} and Chris Hummersone and Antoine Liutkus and Plumbley, {Mark D}",
year = "2016",
month = "12",
day = "1",
doi = "10.1109/EUSIPCO.2016.7760551",
language = "English",
publisher = "IEEE",
pages = "1763--1767",
booktitle = "2016 24th European Signal Processing Conference (EUSIPCO)",

}

Simpson, AJR, Roma, G, Grais, EM, Mason, RD, Hummersone, C, Liutkus, A & Plumbley, MD 2016, Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods. in 2016 24th European Signal Processing Conference (EUSIPCO). IEEE, pp. 1763-1767, 24th European Signal Processing Conference, Budapest, Hungary, 29/08/16. https://doi.org/10.1109/EUSIPCO.2016.7760551

Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods. / Simpson, Andrew JR; Roma, Gerard; Grais, Emad M; Mason, Russell D; Hummersone, Chris; Liutkus, Antoine; Plumbley, Mark D.

2016 24th European Signal Processing Conference (EUSIPCO). IEEE, 2016. p. 1763-1767.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods

AU - Simpson, Andrew JR

AU - Roma, Gerard

AU - Grais, Emad M

AU - Mason, Russell D

AU - Hummersone, Chris

AU - Liutkus, Antoine

AU - Plumbley, Mark D

PY - 2016/12/1

Y1 - 2016/12/1

N2 - Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.

AB - Audio source separation models are typically evaluated using objective separation quality measures, but rigorous statistical methods have yet to be applied to the problem of model comparison. As a result, it can be difficult to establish whether or not reliable progress is being made during the development of new models. In this paper, we provide a hypothesis-driven statistical analysis of the results of the recent source separation SiSEC challenge involving twelve competing models tested on separation of voice and accompaniment from fifty pieces of “professionally produced” contemporary music. Using non-parametric statistics, we establish reliable evidence for meaningful conclusions about the performance of the various models.

KW - Audio source separation

KW - BSSeval

KW - Hypothesis test

KW - SiSEC

U2 - 10.1109/EUSIPCO.2016.7760551

DO - 10.1109/EUSIPCO.2016.7760551

M3 - Conference contribution

SP - 1763

EP - 1767

BT - 2016 24th European Signal Processing Conference (EUSIPCO)

PB - IEEE

ER -

Simpson AJR, Roma G, Grais EM, Mason RD, Hummersone C, Liutkus A et al. Evaluation of audio source separation models using hypothesis-driven non-parametric statistical methods. In 2016 24th European Signal Processing Conference (EUSIPCO). IEEE. 2016. p. 1763-1767 https://doi.org/10.1109/EUSIPCO.2016.7760551