Interactive model building for -learning.

TitleInteractive model building for -learning.
Publication TypeJournal Article
Year of Publication2014
AuthorsLaber, Eric B., Kristin A. Linn, and Leonard A. Stefanski
Date Published2014 Oct 20

Evidence-based rules for optimal treatment allocation are key components in the quest for efficient, effective health care delivery. Q-learning, an approximate dynamic programming algorithm, is a popular method for estimating optimal sequential decision rules from data. Q-learning requires the modeling of nonsmooth, nonmonotone transformations of the data, complicating the search for adequately expressive, yet parsimonious, statistical models. The default Q-learning working model is multiple linear regression, which is not only provably misspecified under most data-generating models, but also results in nonregular regression estimators, complicating inference. We propose an alternative strategy for estimating optimal sequential decision rules for which the requisite statistical modeling does not depend on nonsmooth, nonmonotone transformed data, does not result in nonregular regression estimators, is consistent under a broader array of data-generation models than Q-learning, results in estimated sequential decision rules that have better sampling properties, and is amenable to established statistical approaches for exploratory data analysis, model building, and validation. We derive the new method, IQ-learning, via an interchange in the order of certain steps in Q-learning. In simulated experiments IQ-learning improves on Q-learning in terms of integrated mean squared error and power. The method is illustrated using data from a study of major depressive disorder.

Alternate JournalBiometrika
Original PublicationInteractive model building for Q-learning.
PubMed ID25541562
PubMed Central IDPMC4274394
Grant ListP01 CA142538 / CA / NCI NIH HHS / United States
R01 CA085848 / CA / NCI NIH HHS / United States