Events Add an event Speakers Talks Collections
 
DataWorks Summit Barcelona 2019
March 19, 2019, Barcelona, Spain
DataWorks Summit Barcelona 2019
Request Q&A
DataWorks Summit Barcelona 2019
From the conference
DataWorks Summit Barcelona 2019
Request Q&A
Video
Better than Deep Learning: Gradient Boosting Machines (GBM)
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
9 K
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About the talk

With all the hype about deep learning and "AI", it is not well publicized that for structured/tabular data widely encountered in business applications it is actually another machine learning algorithm, the gradient boosting machine (GBM) that most often achieves the highest accuracy in supervised learning tasks. In this talk we'll review some of the main GBM implementations available as R and Python packages such as xgboost, h2o, lightgbm etc, we'll discuss some of their main features and characteristics, and we'll see how tuning GBMs and creating ensembles of the best models can achieve fantastic prediction accuracy for many business problems.

With all the hype about deep learning and "AI", it is not well publicized that for structured/tabular data widely encountered in business applications it is actually another machine learning algorithm, the gradient boosting machine (GBM) that most often achieves the highest accuracy in supervised learning tasks. In this talk we'll review some of the main GBM implementations available as R and Python packages such as xgboost, h2o, lightgbm etc, we'll discuss some of their main features and characteristics, and we'll see how tuning GBMs and creating ensembles of the best models can achieve fantastic prediction accuracy for many business problems.

About speaker

Szilard Pafka
Chief Scientist (Chief Data Scientist) at EP0CH

Szilard studied Physics in the 90s and obtained a PhD by using statistical methods to analyze the risk of financial portfolios. He worked in finance, then more than a decade ago moved to become the Chief Scientist of a tech company in Santa Monica, California doing everything data (analysis, modeling, data visualization, machine learning, data infrastructure etc). He is the founder/organizer of several meetups in the Los Angeles area (R, data science etc) and the data science community website datascience.la. He is the author of a well-known machine learning benchmark on github (1000+ stars), a frequent speaker at conferences (keynote/invited at KDD, R-finance, Crunch, eRum and contributed at useR!, PAW, EARL etc.), and he has developed and taught graduate data science and machine learning courses as a visiting professor at two universities (UCLA in California and CEU in Europe).

View the profile
Share

Don't thank you for having me. My name is too large and I have a slide a little bit more about myself so many many years ago almost dinner time. I was doing physics. That's why I was 20 plus years ago and like most of my colleagues I moved to finance and then I'm originally from Europe but twelve years ago. I moved to California to do what later has been cold ever since the data science. I'm also Meetup organizer. I've been teaching at two universities and I have so much fun learning benchmarks that are on GitHub and I'm going to talk a little bit about those as

this talk. Quick disclaimer everything I say here is special everything bad enough to use my own responsibilities. So don't blame my employer down really nice people. So find many of you have seen how many of you have seen the slide or I saw at least have so this is a great guy. He's founder of Coursera. He's been pitching machine learning for a juice at Stanford. So he really knows what he's doing is a little bit typing people are named Gracie. So this is a slight from maybe

3-4 years ago where he claims that deep learning is better than any other machine learning algorithm, especially if you have a lot of data, And we expect deep learning to give us self-driving cars and the claim is also that deep learning is already smart and then human since he beats human players and many many games. And of course we expect to have any minute away. I And what about dads will bring to us? So I think those claims a little bit overblown. So the reality is more nuanced. So indeed deep learning the has been very successful in

computer vision image classification. It has some success with sequence prediction time series text and combined with enforcement learning indeed. It has been very successful in many virtual environment. I like games where you can create as much data by playing the computer is against each other as much data as you want and in that case deep learning can indeed have very successful results However in real life I'm not sure deep learning will really give us a I or if it will be

very successful and most big business applications. And in fact, we've been having machine learning for at least 30 years and very successful analytics applications in many many domains and many many domains for example in fraud detection or credit scoring or marketing analytics people have been doing machine learning induced domains for 20-30 years. Many many other domains like manufacturing example of detection you guys if you working businesses here you already know this. So I try

and in a little GitHub repo on some public data to see deep learning in some of these domains gives better results and I've been trying many many architectural sent to ningun talk to. Top word experts in deep learning and in many cases I couldn't make as good results as with other traditional algorithms. And this was couple of years ago. I think a better even better proof of this is just a dog by tianqi Chen he's worth top researcher Bose and traditional machine learning algorithms and also in deep learning

and he had a talk with my meet up in Los Angeles where in 2016 he was saying that do stars of kaggle competitions where people really compete on accuracy and people invest a lot of time in finding the best solution to problem in 2/3 of the cases the winning algorithm was not deep learning was gradient boosting and in about one-third of cases, he was deep learning. So the pattern here is a little bit oversimplified is like if you have some kind of business traditional business problem with toddlers, Dan typically gradient boosting beats deep learning if you have

image or a speech or text a. Then you because I deep learning Keys is best. So yes kind of my answer to and yang slide. So I think in most applications in businesses where you have tabular data. Typically does Datacom relational databases and the typical problem is like for today show a customer turn or any of this kind of traditional business applications deep learning is not really the best. Therefore is my daughter of the stalk. But for those of us doing machine learning for a decade or more,

this is not really new and it's not really surprising so here to papers from 2006 icml Hell by overlapping daughters that I like. I've been coaching these for the last five years. I don't I think they're still not outdated. So these guys have taken 20 + data sets 10 + algorithms vs Mavericks and have done a huge comparison of the accuracy of different to algorithms and they came out that. That's why I swear it like random Forest boosting your network switch now deep learning is he wants support specs on machines and then logistic regression but like boosting random Forest came up the

top and this was a little bit smaller data sets that nowadays on candle siding the difficult. Size was maybe 10,000 records people go tacos sizes and business applications are like $100,000 or two million records and induce case boosting becomes better than random forest in general. But when you want to do you go for the accuracy and emotional learning project is not only the algorithm that matters. I choose several other things matter even more one of the most important things to use to

clean your data and understand your daytime the domain knowledge use it for building features and you can even you can improve even more deactivate CSU Ensemble many many models so important than a special that we need a then. Featuring extraction or feature engineering is sometimes more important than which algorithm you use weather is random Forest boosting or you want ads And ultimately this whole process of this used to be called a to Mining and this is just my picture inspired some what's called Chris which came from

the 1990s the document put together by industry Experts of what we are doing in our data mining now call you later science machine learning projects so we could play We like some days are we explore visualize clean the daytime than we do the modeling validate them all down with him and then deploy them and we try also to understand what's going on in a problem-oriented a Time. So this is the only important at least as important as more important then then we are going on you choose. So the algorithm you choose each other in the morning part 2 small biz.

So if I want to kind of summarize what I want to say in the stock and put it in very simple terms. I think if you have typical Business program problem with structured tabular data coming from relational databases, then I will try for its gradient boosting go random Forest do any of these items which are assembled self trees. If you have small days a couple of hundred maybe a thousand observations then everything else then probably some very simple linear models

will over fit so you don't do any good by using something very sophisticated. So go for the simple linear models if you have big day. Work or five days, I'm done. Okay gradient boosting random Forest. Even you are networks will be computationally infeasible. You would need to wait to train them for hours so days or even more so you want to use a game linear models typically served with not even solving the Matrix algebra part, but just doing stochastic gradient descent you have image speech or

even text then indeed do deep learning so you can see that the title of the stock was kind of misguided and just make you come here to listen to my talk but indeed so gradient boosting is not better than deep learning Ian neither is deep learning better than gradient boosting. So each of those are Method schools that are good at something and they are bad as something else. So in a typical business application with structures, have you heard a time you would want to go first and

use gradient boosting because that's going to be eating a curiosity deep learning but with like images speech and many other problems deep learning has the best soul using so it depends on on the problem you have to work on. So indeed Independence another way is to try them all so that I may be like for a gradients you if you do supervised learning trying to predict an outcome from input then. Maybe four or five items. You need to pair the mouse so you can try them

all. You can also play with hyperparameter tuning menu of this High Gardens have a lot of parameters. You can try to June them to get better accuracy. I mentioned already ensembles do many models and Ensemble them but simple ways on Sample that needs to take an average but there are more sophisticated ways featuring engineering mention either gained very important. So the Deep learning people will tell you that the blending is replacing feature engineering and eating you don't need to

do featuring Ginuwine, which is true for images and in some other than mines, but the gain in this business applications, it typically deep learning doesn't figure out domain knowledge. So you still want and have to do to change your name if you want to get good results and they the cleaning obviously And sometimes you don't sometimes accuracy is not really the most important thing you go after in a project. So if you have to explain your mother because of regulations are some of the reasons for you want to

understand your mother and then then you might not even want to use gradient boosting or deep learning. You might want to use something more explainable like additive models or even God forbid the linear regression. So with all this disclaimers that they are great is not only the algorithm is important but many other things and gradient boosting is not better than deep learning. But so that's how you want to hear a little bit more about gradient boosting. So this is going to be the rest of the stock.

Forget the gradient boosting is based on trees you build decision trees and then you can somebody Whoever has them. So what's the decision tree decision to you can think of it as a recursive partitioning of the input space and then you assign the slavers. You can so busy cleats on the data you build the structure where the whole structure has learned from the data and also like this which variables are in the splits and what are the splits points? You also it comes from the day that with the base

decision Tree Learning algorithm. So the outcome is something like this. This is an example from my favorite book. I'm going to talk a little bit later. So this is like a Spam detection problem and basic the year was that The email is Spam or not. And this is some denied these when the spam emails had a lot of dollar signs in the in the in the text. So basically if the percentage of dollar signs was like greater than 5% and then you would go in this direction

and basically the leaves of the trees severe degree to read a decision tree you have an input and then you go over the splits and then you end up in a leave and then believe will tell you if it's in case of a binary classification is 801 or in case of regression of a number so the first boosting algorithm comes not gradient boosting but Edibles comes from 1997 and the idea here is You don't need to read this. I just put it for a reference. So you build decision trees and you going to average them and the way you build them is that when I when you be

able to answer all season tree, you try to build it in such a way to get to write the responses that the previews average of the trees have missed. So basically you and you do this by by over waiting the examples that you missed. So physically in each step when you brought Saint Ann's decision tree you going to take the previews weighted-average and you cannot wait the examples that previously the previews this baby situation missed you going to overweight them and this is how you build more and more trees.

And a gradient boosting. You definitely don't need to look at this again just for a friend. So here's his book that explained it. I think the best so basically Friedman was the one that not only wrote this book, which is I think one of the best books on machine learning with special from a statistical point of view, but also invented the gradient boosting algorithm. So basically you do you do this with gradients. more more more importantly, how can I use this so well I tried to find some solution

that that work pretty well for what I needed for work a while ago and I wanted to look at which open source tools I can use and the was not really anything I could find with Google. So I kind of started to compare this option with chili carb item packages xgboost spark and a few others. So the first question is why open source, I'm sure I don't have to say you guys open sores. When I started doing this I stumbled upon this code sound does Dan kygo number 101 Bank. What's that mean? They asked him how he wins all this kind of competitions. He

developed his own algorithms and basically was saying now then I just use open-source implementations and then I Spy Most of my time with featuring Jhene during which is very sensible thing to do. So open-source is free do so has great Community meetups conferences documentation books back overflow and then most people do machine learning data science with open source, I using our acquaintance and many of those are Great Danes have R and python implementation has the most

implementing D&C cplusplus of java, and then the reason R and python package wrapper. So I did this Benchmark, which I don't really consider. It has like very complete but it's more than what ever existed before because there was no not really anything comparing any of those packages so you can see it all on GitHub and it will have a lot of Salem skating grand slams does example runtime vs. Size. This is on a logarithmic scale and the number for Fred Meyers in millions. So I'm like

10,000 to 10 million records and was Duran time between those and then was the accurity. So what kind of surprising is that? There is such a huge difference, even though everyone implements the algorithm in this book, basically. So you would think that everyone implements does and still you would find a lot of difference even in accuracy, but yeah even bigger and like run time. So NX E100 bags between the best end up where is implementation is so he's wanting blue imitation around for

1 minute and the other one for a hundred minutes. Then probably want to use the pass the one hour one hour versus $100 and the best ones. I'm going to spare you the time you boost H2O and now I'm relatively newer package that's being open source by Microsoft Flight GPM. And all of this have our invites on the rapper so you can use our appointment you can install them in like under a minute and then start using it. The question is well, this is a big data conference. So why not using spark so it seems simple

very slow. So unfortunately Sparky's lights at least 10 times slower and uses ten times more memory than the best implementation. So it's literally slow. And this has been recognized by the Dayton breaks game like many many years ago, and I'm sure they try to improve it and indeed with 2.0. So this was around 1.5 when I did this benchmarks and then like 2.0. I really like the new dataframe API. So instead of writing a hundred lines of code. Now you could do anything this fine with the fuel lines of code just like we did all the other packages

but unfortunately. Even slower than before so and also the ransom accuracy problems. So I think it's kind of like an art architecture bag because all those big data algorithms State try to do some approximation to see in order to to work on like big data in in a distributed fashion. So I think... Still it hasn't improved since so Yeah, but I have big data. So I need like a big data solution. So first many of you inspirational like business projects, you don't really have these days off work. If the data is big is the real daytime. Let's say the clicks but

once you want to do user analytics featuring Junior that and you would get much more reduce dementia bar in refined size survey say that couples million records a hundred yards is what covers I don't know ninety-five 99% of use cases. So and now that's kind of fit in Ram. So when I started its Benchmark Physical a the largest instant had already 250 gigs of RAM and you could buy a hundred gigs of RAM into your Sarver PA from a cheap and then seems that way we got much better and even be to be there as soon as he's done

does and I'll also look back as a service on the size of data for analytics. It seems like they've been growing at about 20% per year in the last decade and RAM has been growing at least 50% affordable easily accessible ramp has grown out at least 50% per year in the last decade. So a lot more data now fits in the ram of one server. So I always encourage people to use single server to see if those cards for machine learning applications. And here's a tweets from a while ago. I think by now everyone knows done for machine learning. You

have to try for a single Machine Tools. And unfortunately for machine learning the Big Data tools are still kind of cumbersome like the Dora bags that are slow and much much less features. So I Tom Thumb. Physics don't Finance in analytics and kind of like more sophisticated analytics and on the other hand. Most of the big data is mostly about processing SQL and some very simple mathematically speaking some simple Transformation Center for machine learning. You need more of the sophisticated ones and not necessary on the biggest data ever.

And by now most people have realized that at least those working in machine learning that really want they want is faster training and not just ink still work with big are they that because you want to see the gradient boosting is still very computation of a very demanding and it can run for a long time. So you you kind of wanted to be faster. So I took this Benchmark, which was not only for gradient boosting. But for random Forest even deep learning you are nads linear models,

and I didn't really wanted to maintain it especially with spark and any other tools that do them to improve for many years. So I made like them much leaner Benchmark just for gradient boosting just for the best tools that I made it in a way that people can easier reproduce it. So basically the result dockerfile and you can just run it and then you can run any ads on any hardware you want. So it's a longer table. So all you have to do is basically pretty much does. Mr.

Pretty much here. Also be kind of improve now incrementally. So this data from a year ago is not really very outdated. So here you have around time and accuracy on a million and 10 million records and most people are saying in the last few years that they work with hundred thousand acres a million records. So this is way enough for most used cases so you can see that so far he has extra boost was kind of the fastest. but now a lot like you beyond is going to faster on CPUs and Jeep use iron like be great because of deep learning.

So we're deep learning. It makes sense with gpus you improve 100 X on the speed. So no sane Foxwoods train you are and that's where is deep learning on CPUs because she abused are so much faster, but for gradient boosting is different so you can analyze it so much easier and basically what you can't paralyze is the computation of so-called histograms, but you can just gained so much so it appears that. You don't get the hundred. Thanks. If you are happy if you got to be faster than CPU and induced

queso a GBM to have completely different implementations from their city and permutation. So for some reason the xgboost implementation is right now is not only faster, but it's the one that's more stable and more feature-rich and this is one another's sorrow with recommend using it if you run a thunder GPU So you can't see the discounting is not linear also in need of those cases and it's you kind of have to try it on your own data to see which one is his best to try. It's really just a couple of lines of code. So this Racine are the same would be

inviting you spend some. Coach here with trying to do you have the most business applications you have categorical data and you have to transform them two numbers one hot encoding or something like that. And then the whole training is just goes he's really like one two lines of code and then here's do predictions. So it's super easy to to train and use a gradient boosting our them with any of those tools. It's also very easy to deploy in production. So you might ask why did I put H2O here because I still

kind of like it the best in the way that when you deploy Latina busy temptation is the tool that gives you the most so even though it's raining is a bit slower. They provide you a tool with which you can with only this many lines of command line or cold or whatever you call it. You already candy sport the model and Incorporated in a vassal. Maybe I am getting servers and you can scan and make requests for scoring so it's bad. So back to training gradient boosting has a lot of hyperparameters. Some of the most important ones are the number of Threes. I was explaining that we are

averaging trees. Typically you want a few hundred that's of the trees how deep they are typical you want. I don't know 10 or between 5 and 50 maybe and then W something cold learning rate. Which is similar to stochastic gradient descent that the Earth will prevent a tree to get to to high white and then Durham read something called early stopping at Indians Alexa light would explain so one problem with gradient boosting in a way similar to a neuron that if you train them too much they going to overfit. So if you just check the accuracy on your training set

it will look like if it goes up so you wouldn't want to train more more like you to really check the accuracy on validation hold outdoor test set, then you would see that it increases and then after that well when it starts over 50, the accuracy on a does independent holdouts and will decrease. So are you stopping would help you stop here? amazingly oldest 3 Implementations Bay you just give it a few Parmalee durrance and it will do already stopping so this has been implemented

later than when I started but not all the vegetables they have it. You can't spend a lot of time learning how to tune manually here to block calls. You can check for a reference and ultimately goes down to the singer from my the favorite book II Cote. Has that basically you were trying to find an Optima model complexity. So what do complex model with be great on a training set, but it will overfeed and it will not give you Optimal Performance or not separate the Satori not practical application like a simple

model such like maybe a linear model in this kind of when a more sophisticated one would be better would be here. So it was under feet. So how do we do the studenglass seven ways you can do so cool manual search when you try different values based on kind of your knowledge. You can do a grid search by combining all this time hyperparameters exam for example number of trees also the trees, but the Literature suggested random search when you poke random numbers in the Cypress Bayou Meto space is computation and more efficient than a grid search. So look at this paper.

This would explain why you want to do random search and the results of something called student gradient descent when you give it to your students to endure hyperparameters. and do the work I've been playing with tuning a random search in this beat up right before you can check it out. Another thing I want to mention is the scaling of those of this algorithms and implementations to multi-core is not trivial. so kind of ironically at this conference I was expressing my views against distributed computing for

machine learning but because service is the oldest implementations now they have they they also have the distributed version but in this case between the nose you have to exchange So Much Information Network that sometimes it slows down so even for a million 10 million records you might gas load slower training on five or ten thousand and one single mode but even worse but never never nobody's acted is that you can get this even on a single machine if you have multiple CPU sockets so

many of this Amazon is new season power from the servers have now I don't know a hundred twenty-eight course but don't know don't want to see if you die down like two or three. Star for Quad stock heads and each of the CD socket is connected on a fast bus to its own memory bank. But when one CPU has to access the day that I went on in Ram, which is it happens to be on another memory bank. This will create slowdowns. So you could see here with any special xgboost invite you be on the phone to this. So you would see here. This

is the wrong time depending on which course on the CPU I'm using and I am pinning them to different course. So which for course you got to spend time with a course you got this one time, but it's you who needs to course on different CPUs, then you would have the slowdowns from a tear to eight. You're onto City cars also like if you use a hyper threading, it will also create this kind of slow down. So you would get this with 16 course you would get the kind of the Des Des wise on the instance with for sockets each one having 16 real Car,

Simulator 16, hyper-threaded 32 * 428 quarts is 32 extra large. So and then if you run it don't even more course you get even slower and 128 course that wouldn't fit in this graph. So it was like 5 times slower on 128 courseden on 16 cars. So sometimes you see people bragging that series I used it takes before course 128th Court. Start training my machine learning team. Guess what? It might be slower than 16 course. So you kind of have to try it for yourself a couple of months ago these guys have

Open down guitarist you and they trying to work on making things better, but it's really hard to make just basically soccer than Uma. so Music reduces another grab that kind of shows you that well, what's the process of doing a machine learning projects that do we do the feature engineering training to name him if a relation the model and then we do the deployment and most of the time I was speaking here about Just DE Mota Lincoln, which auguri dumb and how we can choose the fastest one in the most accurate. But

many of these other things I was so important. So if you score you can score in the back to real time for real time. I was saying already that what you want because it has kind of the tools that make deployment into real-time score in the easiest. and here it would be some kind of brief comparison of what I think are the best free options and I made this for a Nar conference before so I included kind of like the default gradient boosting implementation

in our which is very outdated and nobody montanes it but you're still probably the first thing he would stumble upon a few look for gradient boosting in our well don't use the package use any of those three and I was kind of saying already that basically if you wanted the fastest implementation on CPU than then use lightgbm find if you have a GPU then try to use xgboost and then if you care about fast with time scoring then use a steel. So again, you have three tools and depending on you

use case and many others. Thanks. You you want to use one or the other? And then about a year ago I asked. People on Twitter and implementations are they using and xgboost is so popular because it has been used a lot in Canton NY GPM is a newer one. But now I think slide you be a muse is also raising and I think still more people you do kegels and use the gradient boosting for more like home group projects and not necessary production. So that's why I store is kind of solo. So

and you can see that nobody really use your Spark by this time. Everyone has to doubt it. You kind of want all the others and even the spark people. They're trying now to wrap the end of packages into smart. And then CPU vs GPU. So yeah, not a lot of people have gpus and Views don't offer this kind of tax advantage like for deep learning and then what kind of people are using in practice so I know supplies for my followers. They were one of the old business applications in Industry. So they use allowed to handle for a

gradient boosting and last on your own ads. And I have a lot more repos related to gradient boosting. I think the supplies will be available or just contact me if you cannot find them and then I will be happy to send them to you. I guess this is it and then we probably have a few minutes for a questions. Thank you. Yes question. Do I have a mic for the questions or falcon? So I come from an Academia background. So I was wondering if eaten in this realm there is any interesting algarin coming from high performance Computing and interesting Advanced of my because they usually know they

think about like they developed Nations down the bass because they've been done by people with HTC background. So and that's why the other one is is not because the other one Has been approved by more like those kind of Big Data crunching kind of background. So basically that are put in the store later. It comes down to how you read the data from Ram so that you can keep it in the registers as much as possible and you can take away take advantage of cash locality in those kind of things. So

is it greedy algorithms have been already optimized results for like all this I using like Alamo storage or kind of optimizing the 10x speed which is quite huge. So it's not like 10% but it's like 10 x comes from this kind of knowing really was going on in that processor and people doing this have or background in HPC. alright thank you so much I got a spray tan

Cackle comments for the website

Buy this talk

Access to the talk “Better than Deep Learning: Gradient Boosting Machines (GBM)”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Access to all the recordings of the event

Get access to all videos “DataWorks Summit Barcelona 2019”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “IT & Technology”?

You might be interested in videos from this event

November 9 - 17, 2020
Online
50
21
future of ux, behavioral science, design engineering, design systems, design thinking process, new product, partnership, product design, the global experience summit 2020, ux research

Similar talks

Hilary Mason
General Manager, Machine Learning at Cloudera
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Kerem Tomak
Executive VP, Divisional Board Member, Big Data-Advanced Analytics-AI at Commerzbank AG
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Justin Norman
Vice President, Data Science at Yelp
+ 1 speaker
Sagar Kewalramani
Customer Engineer at Google
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video
Access to the talk “Better than Deep Learning: Gradient Boosting Machines (GBM)”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
843 conferences
34172 speakers
12918 hours of content