Skip to Main Content
 

Global Search Box

 
 
 
 

Files

File List


dissertation_anirbanr_nov28.pdf (1.81 MB)

ETD Abstract Container

Abstract Header

Robust and Scalable Algorithms for Bayesian Nonparametric Machine Learning

Roychowdhury, Anirban
http://rave.ohiolink.edu/etdc/view?acc_num=osu1511901271093727

Abstract Details

2017, Doctor of Philosophy, Ohio State University, Computer Science and Engineering.
Bayesian nonparametric techniques provide a rich set of tools for modeling complex probabilistic machine learning problems. However the richness comes at the cost of significant complexity of learning and inference for large scale datasets, in addition to an orthogonal set of challenges related to algorithm convergence and correctness. In this dissertation we address these issues by developing a variety of novel methods using concepts from established machine learning methodologies as well as fields like statistical physics and differential geometry. First, we develop fast inference algorithms for sequential models with Bayesian nonparametric priors using small-variance asymptotics, an emerging technique for obtaining scalable combinatorial algorithms from rich probabilistic models. We derive a ``hard'' inference algorithm analogous to k-means for the standard hidden Markov model by making particular variances in the model tend to zero, and extend the analysis to the Bayesian nonparametric case, yielding a simple, scalable, and flexible algorithm for discrete-state sequence data with a non-fixed number of states that is also very robust to suboptimal initializations. %We demonstrate the advantages of the proposed framework by a number of results on synthetic and real datasets. We start the second section with a novel stick-breaking definition of a certain class of Bayesian nonparametric priors called gamma processes (GP), using its characterization as a completely random measure and attendant Poisson process machinery. We use it to derive a variational inference algorithm for the infinite Gamma-Poisson model, a particular Bayesian nonparametric latent structure formulation where the latent variables are drawn from a GP prior with Poisson likelihoods. We present results on probabilistic matrix factorization (PMF) tasks on document corpora, and show that we compare favorably to similar constructive methods from the literature. %sampling-based techniques as well as variational approaches based on beta-Bernoulli priors and direct constructions of the GP based on Dirichlet processes. In the third section, we use concepts from statistical physics to develop a robust Monte Carlo sampler that efficiently traverses the parameter space. Built on the Hamiltonian Monte Carlo framework, our sampler uses a modified Nos\'{e}-Poincar\'{e} Hamiltonian with a suitable symplectic leapfrog integrator to ensure correct sampling from the canonical ensemble, with structural cues from Riemannian preconditioning matrices. We follow it up with variant that uses minibatched stochastic gradients, to handle real-life machine learning scenarios with massive datasets, showing strong performance in the PMF scenario mentioned above. We continue with an L-BFGS optimization algorithm on Riemannian manifolds that uses stochastic variance reduction techniques for fast convergence with constant step sizes, without resorting to standard linesearch methods, and provide a new convergence proof for strongly convex functions without using curvature conditions on the manifold. Compared to state-of-the-art Euclidean methods like VR-PCA and first-order Riemannian techniques, our method performs strongly in computation of Karcher means for symmetric p.d. matrices and leading eigenvalues of large data matrices. We finish with a novel technique for learning the mass matrices in Monte Carlo samplers obtained from discretized dynamics that preserve some energy function, by using existing dynamics in the sampling step of a Monte Carlo EM framework, and learning the mass matrices in the M step with a simple but effective online technique. Along with a novel stochastic sampler based on Nos\'{e}-Poincar\'{e} dynamics, we use this framework with standard Hamiltonian Monte Carlo as well as newer stochastic algorithms such as SGHMC and SGNHT, achieving sampling accuracies comparable to existing adaptive samplers that use Riemannian preconditioning techniques, while being significantly faster on a variety of datasets.
Srinivasan Parthasarathy (Advisor)
Brian Kulis (Committee Member)
Mikhail Belkin (Committee Member)
Huan Sun (Committee Member)
187 p.
Computer Science

Recommended Citations

Citations

  • Roychowdhury, A. (2017). Robust and Scalable Algorithms for Bayesian Nonparametric Machine Learning [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1511901271093727

    APA Style (7th edition)

  • Roychowdhury, Anirban. Robust and Scalable Algorithms for Bayesian Nonparametric Machine Learning. 2017. Ohio State University, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu1511901271093727.

    MLA Style (8th edition)

  • Roychowdhury, Anirban. "Robust and Scalable Algorithms for Bayesian Nonparametric Machine Learning." Doctoral dissertation, Ohio State University, 2017. http://rave.ohiolink.edu/etdc/view?acc_num=osu1511901271093727

    Chicago Manual of Style (17th edition)

osu1511901271093727
374
© 2017, all rights reserved.
This open access ETD is published by The Ohio State University and OhioLINK.