Robust variable selection and distributed inference using t-based estimators for large-scale data
Loading...
Access rights
openAccess
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Date
2020
Major/Subject
Mcode
Degree programme
Language
en
Pages
5
Series
28th European Signal Processing Conference, EUSIPCO 2020 - Proceedings, pp. 2453-2457, European Signal Processing Conference
Abstract
In this paper, we address the problem of performing robust statistical inference for large-scale data sets whose volume and dimensionality maybe so high that distributed storage and processing is required. Here, the large-scale data are assumed to be contaminated by outliers and exhibit sparseness. We propose a distributed and robust two-stage statistical inference method. In the first stage, robust variable selection is done by exploiting t-Lasso to find the sparse basis in each node with distinct subset of data. The selected variables are communicated to a fusion center (FC) in which the variables for the complete data are chosen using a majority voting rule. In the second stage, confidence intervals and parameter estimates are found in each node using robust t-estimator combined with bootstrapping and then combined in FC. The simulation results demonstrate the validity and reliability of the algorithm in variable selection and constructing confidence intervals even if the estimation problem in the subsets is slightly underdetermined.Description
Keywords
Bootstrap, High-dimensional, Large-scale data, Robust, Sparse, Statistical inference, Variable selection
Other note
Citation
Mozafari-Majd, E & Koivunen, V 2020, Robust variable selection and distributed inference using t-based estimators for large-scale data . in 28th European Signal Processing Conference, EUSIPCO 2020 - Proceedings ., 9287773, European Signal Processing Conference, European Association For Signal and Image Processing, pp. 2453-2457, European Signal Processing Conference, Amsterdam, Netherlands, 24/08/2020 . https://doi.org/10.23919/Eusipco47968.2020.9287773