An Augmented Treble Stream Deep Neural Network for Video Analysis

Chaolong Zhang; Yuanping Xu; Zhijie Xu; Mei Gong; Benjun Guo; Dengguo Yao

doi:10.1109/IV51561.2020.00056

An Augmented Treble Stream Deep Neural Network for Video Analysis

Chaolong Zhang, Yuanping Xu, Zhijie Xu, Mei Gong, Benjun Guo, Dengguo Yao

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Video analysis for human action recognition is one of the most important research areas in pattern recognition and computer vision due to its wide applications. Deep learning-based approaches have been proven more effective than conventional feature engineering-based models. However, the performance is still unreliable when facing real-world application scenarios. Inspired by the Convolutional Neural Network (CNN) and Recurrent Long-Short Term Model (LSTM), this paper presents an augmented treble-stream deep neural network architecture that supports direct extraction of spatial-Temporal features from video streams and their corresponding dense optical flows. This innovative approach assists effective detection of complex video event features that are annotated by rich event "appearance"and motion features. Substantially improved recognition accuracy is recorded during the experiments that are carried and benchmarked over public video event datasets, for example, UCF 101 and HMDB 51. Analytical evaluation approves the validity and effectiveness of the treble-stream neural network design.

Original language	English
Title of host publication	2020 24th International Conference Information Visualisation, IV 2020
Subtitle of host publication	AI & Analytics, Biomedical Visualization , Builtviz, and Geometric Modelling & Imaging
Editors	Ebad Banissi, Farzad Khosrow-Shahi, Anna Ursyn, Mark W. McK. Bannatyne, Joao Moura Pires, Nuno Datia, Kawa Nazemi, Boris Kovalerchuk, John Counsell, Andrew Agapiou, Zora Vrcelj, Hing-Wah Chau, Mengbi Li, Gehan Nagy, Richard Laing, Rita Francese, Muhammad Sarfraz, Fatma Bouali, Gilles Venturin, Marjan Trutschl, Urska Cvek, Heimo Muller, Minoru Nakayama, Marco Temperini, Tania Di Mascio, Filippo Sciarrone Veronica Rossano Rossano, Ralf Dorner, Loredana Caruccio, Autilia Vitiello, Weidong Huang, Michele Risi, Ugo Erra, Razvan Andonie, Muhammad Aurangzeb Ahmad, Ana Figueiras, Mabule Samuel Mabakane
Publisher	IEEE
Pages	301-306
Number of pages	6
ISBN (Electronic)	9781728191348
ISBN (Print)	9781728191355
DOIs	https://doi.org/10.1109/IV51561.2020.00056
Publication status	Published - 7 Sep 2020
Event	24th International Conference Information Visualisation - Melbourne, Australia Duration: 7 Sep 2020 → 11 Sep 2020 Conference number: 24

Publication series

Name	IEEE Symposium on Information Visualization
Publisher	IEEE
Volume	2020-September
ISSN (Print)	1093-9547
ISSN (Electronic)	2375-0138

Conference

Conference	24th International Conference Information Visualisation
Abbreviated title	IV 2020
Country/Territory	Australia
City	Melbourne
Period	7/09/20 → 11/09/20

Access to Document

10.1109/IV51561.2020.00056Licence: Unspecified

Cite this

Zhang, C., Xu, Y., Xu, Z., Gong, M., Guo, B., & Yao, D. (2020). An Augmented Treble Stream Deep Neural Network for Video Analysis. In E. Banissi, F. Khosrow-Shahi, A. Ursyn, M. W. McK. Bannatyne, J. M. Pires, N. Datia, K. Nazemi, B. Kovalerchuk, J. Counsell, A. Agapiou, Z. Vrcelj, H-W. Chau, M. Li, G. Nagy, R. Laing, R. Francese, M. Sarfraz, F. Bouali, G. Venturin, M. Trutschl, U. Cvek, H. Muller, M. Nakayama, M. Temperini, T. Di Mascio, F. S. V. R. Rossano, R. Dorner, L. Caruccio, A. Vitiello, W. Huang, M. Risi, U. Erra, R. Andonie, M. A. Ahmad, A. Figueiras, ... M. S. Mabakane (Eds.), 2020 24th International Conference Information Visualisation, IV 2020: AI & Analytics, Biomedical Visualization , Builtviz, and Geometric Modelling & Imaging (pp. 301-306). [9373098] (IEEE Symposium on Information Visualization; Vol. 2020-September). IEEE. https://doi.org/10.1109/IV51561.2020.00056

Zhang, Chaolong ; Xu, Yuanping ; Xu, Zhijie et al. / An Augmented Treble Stream Deep Neural Network for Video Analysis. 2020 24th International Conference Information Visualisation, IV 2020: AI & Analytics, Biomedical Visualization , Builtviz, and Geometric Modelling & Imaging. editor / Ebad Banissi ; Farzad Khosrow-Shahi ; Anna Ursyn ; Mark W. McK. Bannatyne ; Joao Moura Pires ; Nuno Datia ; Kawa Nazemi ; Boris Kovalerchuk ; John Counsell ; Andrew Agapiou ; Zora Vrcelj ; Hing-Wah Chau ; Mengbi Li ; Gehan Nagy ; Richard Laing ; Rita Francese ; Muhammad Sarfraz ; Fatma Bouali ; Gilles Venturin ; Marjan Trutschl ; Urska Cvek ; Heimo Muller ; Minoru Nakayama ; Marco Temperini ; Tania Di Mascio ; Filippo Sciarrone Veronica Rossano Rossano ; Ralf Dorner ; Loredana Caruccio ; Autilia Vitiello ; Weidong Huang ; Michele Risi ; Ugo Erra ; Razvan Andonie ; Muhammad Aurangzeb Ahmad ; Ana Figueiras ; Mabule Samuel Mabakane. IEEE, 2020. pp. 301-306 (IEEE Symposium on Information Visualization).

@inproceedings{4665c39293f341e899b4b60e7c0640a4,

title = "An Augmented Treble Stream Deep Neural Network for Video Analysis",

abstract = "Video analysis for human action recognition is one of the most important research areas in pattern recognition and computer vision due to its wide applications. Deep learning-based approaches have been proven more effective than conventional feature engineering-based models. However, the performance is still unreliable when facing real-world application scenarios. Inspired by the Convolutional Neural Network (CNN) and Recurrent Long-Short Term Model (LSTM), this paper presents an augmented treble-stream deep neural network architecture that supports direct extraction of spatial-Temporal features from video streams and their corresponding dense optical flows. This innovative approach assists effective detection of complex video event features that are annotated by rich event {"}appearance{"}and motion features. Substantially improved recognition accuracy is recorded during the experiments that are carried and benchmarked over public video event datasets, for example, UCF 101 and HMDB 51. Analytical evaluation approves the validity and effectiveness of the treble-stream neural network design.",

keywords = "CNN and LSTM, deep learning, human action recognition, treble-stream network, video analysis",

author = "Chaolong Zhang and Yuanping Xu and Zhijie Xu and Mei Gong and Benjun Guo and Dengguo Yao",

note = "Funding Information: ACKNOWLEDGMENT This research is supported by the National Natural Science Foundation of China (NSFC, 61203172), the Sichuan Science and Technology Programs (2018JY0146, 2019YFH0187ˈ and 2019104). Publisher Copyright: {\textcopyright} 2020 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; 24th International Conference Information Visualisation, IV 2020 ; Conference date: 07-09-2020 Through 11-09-2020",

year = "2020",

month = sep,

day = "7",

doi = "10.1109/IV51561.2020.00056",

language = "English",

isbn = "9781728191355",

series = "IEEE Symposium on Information Visualization",

publisher = "IEEE",

pages = "301--306",

editor = "Ebad Banissi and Farzad Khosrow-Shahi and Anna Ursyn and {McK. Bannatyne}, {Mark W.} and Pires, {Joao Moura} and Nuno Datia and Kawa Nazemi and Boris Kovalerchuk and John Counsell and Andrew Agapiou and Zora Vrcelj and Hing-Wah Chau and Mengbi Li and Gehan Nagy and Richard Laing and Rita Francese and Muhammad Sarfraz and Fatma Bouali and Gilles Venturin and Marjan Trutschl and Urska Cvek and Heimo Muller and Minoru Nakayama and Marco Temperini and {Di Mascio}, Tania and Rossano, {Filippo Sciarrone Veronica Rossano} and Ralf Dorner and Loredana Caruccio and Autilia Vitiello and Weidong Huang and Michele Risi and Ugo Erra and Razvan Andonie and Ahmad, {Muhammad Aurangzeb} and Ana Figueiras and Mabakane, {Mabule Samuel}",

booktitle = "2020 24th International Conference Information Visualisation, IV 2020",

address = "United States",

}

Zhang, C, Xu, Y, Xu, Z, Gong, M, Guo, B & Yao, D 2020, An Augmented Treble Stream Deep Neural Network for Video Analysis. in E Banissi, F Khosrow-Shahi, A Ursyn, MW McK. Bannatyne, JM Pires, N Datia, K Nazemi, B Kovalerchuk, J Counsell, A Agapiou, Z Vrcelj, H-W Chau, M Li, G Nagy, R Laing, R Francese, M Sarfraz, F Bouali, G Venturin, M Trutschl, U Cvek, H Muller, M Nakayama, M Temperini, T Di Mascio, FSVR Rossano, R Dorner, L Caruccio, A Vitiello, W Huang, M Risi, U Erra, R Andonie, MA Ahmad, A Figueiras & MS Mabakane (eds), 2020 24th International Conference Information Visualisation, IV 2020: AI & Analytics, Biomedical Visualization , Builtviz, and Geometric Modelling & Imaging., 9373098, IEEE Symposium on Information Visualization, vol. 2020-September, IEEE, pp. 301-306, 24th International Conference Information Visualisation, Melbourne, Victoria, Australia, 7/09/20. https://doi.org/10.1109/IV51561.2020.00056

An Augmented Treble Stream Deep Neural Network for Video Analysis. / Zhang, Chaolong; Xu, Yuanping; Xu, Zhijie et al.

2020 24th International Conference Information Visualisation, IV 2020: AI & Analytics, Biomedical Visualization , Builtviz, and Geometric Modelling & Imaging. ed. / Ebad Banissi; Farzad Khosrow-Shahi; Anna Ursyn; Mark W. McK. Bannatyne; Joao Moura Pires; Nuno Datia; Kawa Nazemi; Boris Kovalerchuk; John Counsell; Andrew Agapiou; Zora Vrcelj; Hing-Wah Chau; Mengbi Li; Gehan Nagy; Richard Laing; Rita Francese; Muhammad Sarfraz; Fatma Bouali; Gilles Venturin; Marjan Trutschl; Urska Cvek; Heimo Muller; Minoru Nakayama; Marco Temperini; Tania Di Mascio; Filippo Sciarrone Veronica Rossano Rossano; Ralf Dorner; Loredana Caruccio; Autilia Vitiello; Weidong Huang; Michele Risi; Ugo Erra; Razvan Andonie; Muhammad Aurangzeb Ahmad; Ana Figueiras; Mabule Samuel Mabakane. IEEE, 2020. p. 301-306 9373098 (IEEE Symposium on Information Visualization; Vol. 2020-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An Augmented Treble Stream Deep Neural Network for Video Analysis

AU - Zhang, Chaolong

AU - Xu, Yuanping

AU - Xu, Zhijie

AU - Gong, Mei

AU - Guo, Benjun

AU - Yao, Dengguo

N1 - Conference code: 24

PY - 2020/9/7

Y1 - 2020/9/7

N2 - Video analysis for human action recognition is one of the most important research areas in pattern recognition and computer vision due to its wide applications. Deep learning-based approaches have been proven more effective than conventional feature engineering-based models. However, the performance is still unreliable when facing real-world application scenarios. Inspired by the Convolutional Neural Network (CNN) and Recurrent Long-Short Term Model (LSTM), this paper presents an augmented treble-stream deep neural network architecture that supports direct extraction of spatial-Temporal features from video streams and their corresponding dense optical flows. This innovative approach assists effective detection of complex video event features that are annotated by rich event "appearance"and motion features. Substantially improved recognition accuracy is recorded during the experiments that are carried and benchmarked over public video event datasets, for example, UCF 101 and HMDB 51. Analytical evaluation approves the validity and effectiveness of the treble-stream neural network design.

AB - Video analysis for human action recognition is one of the most important research areas in pattern recognition and computer vision due to its wide applications. Deep learning-based approaches have been proven more effective than conventional feature engineering-based models. However, the performance is still unreliable when facing real-world application scenarios. Inspired by the Convolutional Neural Network (CNN) and Recurrent Long-Short Term Model (LSTM), this paper presents an augmented treble-stream deep neural network architecture that supports direct extraction of spatial-Temporal features from video streams and their corresponding dense optical flows. This innovative approach assists effective detection of complex video event features that are annotated by rich event "appearance"and motion features. Substantially improved recognition accuracy is recorded during the experiments that are carried and benchmarked over public video event datasets, for example, UCF 101 and HMDB 51. Analytical evaluation approves the validity and effectiveness of the treble-stream neural network design.

KW - CNN and LSTM

KW - deep learning

KW - human action recognition

KW - treble-stream network

KW - video analysis

UR - http://www.scopus.com/inward/record.url?scp=85102940003&partnerID=8YFLogxK

U2 - 10.1109/IV51561.2020.00056

DO - 10.1109/IV51561.2020.00056

M3 - Conference contribution

AN - SCOPUS:85102940003

SN - 9781728191355

T3 - IEEE Symposium on Information Visualization

SP - 301

EP - 306

BT - 2020 24th International Conference Information Visualisation, IV 2020

A2 - Banissi, Ebad

A2 - Khosrow-Shahi, Farzad

A2 - Ursyn, Anna

A2 - McK. Bannatyne, Mark W.

A2 - Pires, Joao Moura

A2 - Datia, Nuno

A2 - Nazemi, Kawa

A2 - Kovalerchuk, Boris

A2 - Counsell, John

A2 - Agapiou, Andrew

A2 - Vrcelj, Zora

A2 - Chau, Hing-Wah

A2 - Li, Mengbi

A2 - Nagy, Gehan

A2 - Laing, Richard

A2 - Francese, Rita

A2 - Sarfraz, Muhammad

A2 - Bouali, Fatma

A2 - Venturin, Gilles

A2 - Trutschl, Marjan

A2 - Cvek, Urska

A2 - Muller, Heimo

A2 - Nakayama, Minoru

A2 - Temperini, Marco

A2 - Di Mascio, Tania

A2 - Rossano, Filippo Sciarrone Veronica Rossano

A2 - Dorner, Ralf

A2 - Caruccio, Loredana

A2 - Vitiello, Autilia

A2 - Huang, Weidong

A2 - Risi, Michele

A2 - Erra, Ugo

A2 - Andonie, Razvan

A2 - Ahmad, Muhammad Aurangzeb

A2 - Figueiras, Ana

A2 - Mabakane, Mabule Samuel

PB - IEEE

T2 - 24th International Conference Information Visualisation

Y2 - 7 September 2020 through 11 September 2020

ER -

Zhang C, Xu Y, Xu Z, Gong M, Guo B, Yao D. An Augmented Treble Stream Deep Neural Network for Video Analysis. In Banissi E, Khosrow-Shahi F, Ursyn A, McK. Bannatyne MW, Pires JM, Datia N, Nazemi K, Kovalerchuk B, Counsell J, Agapiou A, Vrcelj Z, Chau H-W, Li M, Nagy G, Laing R, Francese R, Sarfraz M, Bouali F, Venturin G, Trutschl M, Cvek U, Muller H, Nakayama M, Temperini M, Di Mascio T, Rossano FSVR, Dorner R, Caruccio L, Vitiello A, Huang W, Risi M, Erra U, Andonie R, Ahmad MA, Figueiras A, Mabakane MS, editors, 2020 24th International Conference Information Visualisation, IV 2020: AI & Analytics, Biomedical Visualization , Builtviz, and Geometric Modelling & Imaging. IEEE. 2020. p. 301-306. 9373098. (IEEE Symposium on Information Visualization). doi: 10.1109/IV51561.2020.00056

An Augmented Treble Stream Deep Neural Network for Video Analysis

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this