About the talk
Most deep learning networks today rely on dense representations. This stands in stark contrast to our brains which are extremely sparse, both in connectivity and in activations. Implemented correctly, the potential performance benefits of sparsity in weights and activations is massive. Unfortunately, the benefits observed to date have been extremely limited. It is challenging to optimize training to achieve highly sparse and accurate networks. Hyperparameters and best practices that work for dense networks do not apply to sparse networks. In addition, it is difficult to implement sparse networks on hardware platforms designed for dense computations. In this talk we present novel sparse networks that achieve high accuracy and leverage sparsity to run 100X faster than their dense counterparts. We discuss the hyperparameter optimization strategies used to achieve high accuracy, and describe the hardware techniques developed to achieve this speedup. Our results show that a careful evaluation of the training process combined with an optimized architecture can dramatically scale deep learning networks in the future.
Subutai Ahmad is the VP of Research at Numenta, a research company that is applying neuroscience principles to machine intelligence research. Subutai brings experience in deep learning, neuroscience, and real time systems. His research interests lie in understanding and applying neuroscience insights from areas such as sparsity, dendrites, unsupervised learning, and sensorimotor prediction. Subutai holds a B.S. in Computer Science from Cornell University, and a Ph.D in Computer Science and Computational Neuroscience from the University of Illinois at Urbana-Champaign.View the profile
Buy this talk
Buy this video
Our other topics
With ConferenceCast.tv, you get access to our library of the world's best conference talks.