Events Add an event Speakers Talks Collections
 
SigOpt Summit 2021
November 16, 2021, Online
SigOpt Summit 2021
Request Q&A
SigOpt Summit 2021
From the conference
SigOpt Summit 2021
Request Q&A
Video
How Can We Be So Slow? Winning the Hardware Lottery by Accelerating Sparse Networks
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
89
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Discussion

About the talk

Most deep learning networks today rely on dense representations. This stands in stark contrast to our brains which are extremely sparse, both in connectivity and in activations. Implemented correctly, the potential performance benefits of sparsity in weights and activations is massive.  Unfortunately, the benefits observed to date have been extremely limited. It is challenging to optimize training to achieve highly sparse and accurate networks. Hyperparameters and best practices that work for dense networks do not apply to sparse networks. In addition, it is difficult to implement sparse networks on hardware platforms designed for dense computations. In this talk we present novel sparse networks that achieve high accuracy and leverage sparsity to run 100X faster than their dense counterparts. We discuss the hyperparameter optimization strategies used to achieve high accuracy, and describe the hardware techniques developed to achieve this speedup. Our results show that a careful evaluation of the training process combined with an optimized architecture can dramatically scale deep learning networks in the future.

About speaker

Subutai Ahmad
VP Research at Numenta

Subutai Ahmad is the VP of Research at Numenta, a research company that is applying neuroscience principles to machine intelligence research. Subutai brings experience in deep learning, neuroscience, and real time systems. His research interests lie in understanding and applying neuroscience insights from areas such as sparsity, dendrites, unsupervised learning, and sensorimotor prediction. Subutai holds a B.S. in Computer Science from Cornell University, and a Ph.D in Computer Science and Computational Neuroscience from the University of Illinois at Urbana-Champaign.

View the profile
Share
Cackle comments for the website

Buy this talk

Access to the talk “How Can We Be So Slow? Winning the Hardware Lottery by Accelerating Sparse Networks”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Full access

Get access to all videos “SigOpt Summit 2021”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Similar talks

Michael McCourt
Head of Engineering at SigOpt
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Scott Clark
Co-founder and CEO at SigOpt
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Jian Zhang
Senior Software Engineering Manager at Intel Corporation
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video
Access to the talk “How Can We Be So Slow? Winning the Hardware Lottery by Accelerating Sparse Networks”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
909 conferences
36259 speakers
13921 hours of content