Title | Online updating method with new variables for big data streams. |
Publication Type | Journal Article |
Year of Publication | 2018 |
Authors | Wang, Chun, Ming-Hui Chen, Jing Wu, Jun Yan, Yuping Zhang, and Elizabeth Schifano |
Journal | Can J Stat |
Volume | 46 |
Issue | 1 |
Pagination | 123-146 |
Date Published | 2018 Mar |
ISSN | 0319-5724 |
Abstract | For big data arriving in streams, online updating is an important statistical method that breaks the storage barrier and the computational barrier under certain circumstances. In the regression context, online updating algorithms assume that the set of predictor variables does not change, and consequently cannot incorporate new variables that may become available midway through the data stream. A naive approach would be to discard all previous information and start updating with new variables from scratch. We propose a method that utilizes the information from earlier data in the online updating algorithm with bias corrections to improve efficiency. The method is developed for linear models first, and then extended to estimating equations for generalized linear models. Closed-form expressions for the efficiency gain over the naive approach are derived in a particular linear model setting. We compare the performance of our proposed bias-correcting approach and the naive approach in simulation studies with data generated from a normal linear model and a logistic regression model. The method is applied to a study on airline delay, where reasons for delays were only available more recently, starting in 2003. |
DOI | 10.1002/cjs.11330 |
Alternate Journal | Can J Stat |
Original Publication | Online updating method with new variables for big data streams. |
PubMed ID | 29662263 |
PubMed Central ID | PMC5898930 |
Grant List | P01 CA142538 / CA / NCI NIH HHS / United States R01 GM070335 / GM / NIGMS NIH HHS / United States |
Online updating method with new variables for big data streams.
Project: