Stephen J. Smith is the research director for data science at G7 Research. His unique perspective comes from his real-world experience in building the predictive analytics products Darwin, Discovery Server and Optas. These products were among the first to deliver machine learning on an MPP computer architecture, implement algorithms directly in SQL, and embed model results in an OLAP tool. He has written the best-selling business technology books: “Data Warehousing, Data Mining and OLAP” and “Building Data Mining Application for CRM” with McGraw-Hill. He received his undergraduate degree in engineering from MIT and his graduate degree from Harvard in machine learning and predictive analytics. His current research is on the limits of automation in data science.View the profile
About the talk
AI and data science teams suffer from all the same issue as a normal tech team: late delivery or models, changing requirements, sandbagging of goals and great models that are never used. OKRs is a goal setting process developed at Intel and perfected at Google that solves these problems. In this workshop you will learn how to apply them to your AI and data science teams to produce better models, faster and with more dedicated and enthusiastic team members.
I'm on the east coast in Boston here. So it's the sun has set. But I look forward to chatting about okay ours and AI for the next 30 minutes. Thanks for all coming. And my name is Steven Smith. My background is in building data science and AI infrastructure projects for probably last 30 years. I've also as a serial entrepreneur, I have built and sold several companies. And then also in my spare time, I did a lot of coaching of starter companies that MIT and Harvard last kind of I-90 companies and I'm
kind of that weird background of data science machine learning scientist coupled with looking at how companies run and how teams are managed. I came of this idea of you know, how to use this new management tool called an okr To help with the AI in data science teams, and I think if you have Ron of your on data science for AI team, I think you look kind of appreciate what okay ours can do for you. So he talking about that today and feel free to go ahead and jump in if you have questions or post a question on the chat trying to answer if I can or at
the end and let me see if I can get to the next slide here. Got that they were go. You can get this over here. The first question is do you actually set goals or objectives for your AI project seems like a basic thing you'd want to do in business, but I'm surprised that a lot of companies actually considered AI projects are they the same kind of special and sometimes they don't set goals, but it good good test for this is do you know what your goals are right now? You have some AI projects are running but you have specific objectives are goals for those projects or is the goal just to do a i
if you have goals you hit your goals and then really importantly if you know your goals do your subordinates in the people on the team know the goals. There was a recent study then they asked to managers what percentage of the employees they thought would know the goals for the company in the managers estimated probably about 60% So 6 out of 10 employees would probably know what the main goals objectives of the company were when they checked on it was actually less than 10% So if you have goals, it's also important
to communicate them and AI projects machine Learning Center our little bit of a different animal and here's some of the reasons I've heard some of the companies that work with a is super creative. We can't promise anything. Who knows what will discover it's all blue sky, it is different. But on the other hand is still part of business. Now, we have to set some goals. Did we use Advil and lean methodologies with a 2-week Time Horizon with are Sprint's all good. But the reality is that even if you're using an awesome Adria methodology, okay ours can
actually run on top of that. I'm actually working with a company right now where the really super agile process setup and they put okay hours on top of that in order to set some goals. So nothing cuz I hear from managers say I can't be rushed we can't set an arbitrary time limit for delivery again sort of true. But also we need to do better and probably set sometimes and then again, I'm doing something so cutting head that I just can't adhere to the best practices. So for using OK ours or lean methodology elsewhere in the
organization, I can't do that can fly that to a I partially true but it's also something that you need to get around because as Yogi Berra said if you don't know where you are going you will end up someplace else and in physical Yogi Bear fashion that is deeply true. If you think about it, so goals are important even if you are doing AI or machine learning So the point of this talk. is that Doe applications are creative and unpredictable. It can still be improved with
best practices and has best practices are all these to be. Okay r OCR stands for objectives and key results that was practicing evangelize behind John door and Google and a lot of other high-tech startups will show you some of those in a second. And it can accommodate the variability of a I kind of almost invented for things that have a lot of requirements for discipline delivery, but also have some variability and whether they can be delivered or not. It's
important recognize that okay our works. Well, if there's work well with lean development practices. It is a help to guide them. It's a great way to kind of at a high-level understand what's going on, but it does not replace them. But if all you're using is lean development is really not sufficient for AI projects and data science projects. Am I singing? But okay Ayrsley link the AI in data science directly to the goals and objectives of the company and align them so that the resulting products are valued by Senior Management, and I think if you are a data scientist Renee
our researcher you recognize importance of the statement. This is basically saying that your work will not be wasted that when you set up a goal you have alignment with Senior Management so that when you deliver some awesome amazing models, they will actually be used rather than ignored or executed but then not implemented in the real world. And that's a tremendous value of having okrs implemented in AI an AI team. So who uses okrs? Hopefully after the stock you will my current company is B7 research again any other companies in coaching out of MIT
or Harvard? I strongly encourage them to use it as such a lightweight process. Even if you are a tiny little company. It's a great way to get started there. Is this other not tiny little company called Google that will see you in a second has built themselves solely based on okay hours for that reason that reason alone. It's worth looking at them. It has been what Google is used to run the company from basically two employees up to the size. They are now Netflix also uses that other companies Amazon is used in various ways. And the last one there is a Banos company from the rock band
U2 believe it or not. They actually use okrs for some of his charity work. He's recognized that to solve poverty and reduce illness around the world that okay ours actually great match so she can imagine works for a rock band singer as well as Anthony's high tech companies is pretty flexible. So let's just take a quick background view of why do goals help? They help you to focus on the most important thing is lots of things to do lots of things to read. They help you focus on what's most important to John door actually call schools a vaccine
against fuzzy thinking a vaccine against fuzzy thinking and I think that's a pretty pretty good description. The number to lower stress this one might not be obvious. But I have seen a lot of companies where the employees are really stressed out because they're not quite sure what they're supposed to be working on. You think that would be kind of fun right go work on whatever you want, but it's really opposite people like to know that they're working on the important things and things that are significant and it may be hard.
But if they know that they're working on the right things, it'll actually lower stress. It also increases trust route the organization by making people accountable. They have particular deliverables and they and they actually build trust with the people around them because they have these deliverables and obviously if they are on the hook for them, they will be measured by them and they can improve their accountability. The other thing is also because
okay, there's a transparent they incur support from others and they create a feeling of success. I was at a game conference game developers conference and the game designer for very popular game Uncharted. I actually admitted that when he went home at night, he wasn't sure as of this is a very successful video game. He wasn't sure whether he had made progress or not. We'd actually go and play the Solitaire can one of these The Angry Birds or something like that because he felt
like he was making some progress and this is basically he probably would have done better if he had you been using Okay ours because they tell you what you're doing in the nice. You knock them down you build a real feeling of success, which is really helpful to building teamwork. So goals are awesome, but they also have a downside sometimes. They can have sent the wrong behaviors. So for instance, you may remember Wells Fargo a couple of years ago have a goal to obviously make revenue and open as many accounts as they possibly could
I could go but they missed out on customer satisfaction and also kind of missed out on not doing something illegal. They actually open accounts for customers that didn't know that they have the accounts. So be careful what you wish for you to be very careful and setting your goals another example for a long time ago was Ford Pinto had exploding gas tanks. They have an awesome goal of very lightweight car fuel efficient and it low cost but they also needed a goal of us safety. And
again challenges with Facebook and some of the other Tech Giants are monetizing customer data, but they are now also facing the additional goal of preserving personal privacy. Nicole scan and sent the wrong behaviors. And this is kind of one of the things that okay ours help with the other problem with I just regular goals is that they can incent sandbagging where sandbagging is basically the person who setting the goal knows that they are going to be evaluated gone with achieve that goal or not at the end of the quarter at the end of the year and they might actually
make or lose a lot of money based on whether they achieved the goal. So what's human nature people are going to basically set the goal lower than they might. Otherwise be able to achieve in order to get as much personal, fish compensation is that can result in lower goals goals of lower after expiration and also murky objectives, right? If you don't have a crisp objective, then you can always argue with you achieved it. Sogos have these current problems and Mana recognize that he's sums up the okay ours is of the you want an environment for risk and for trust and
we're failing is not a fireable offense and that is particularly true in Ai and data science projects where there is often no guarantee of success at the end. So successful company said only a few goals that is an important thing to do Larry Page agreed with this and it's been my experience to the most successful companies. I've mentored emphatically decide what they are not going to do and I heard a great little cash crazy of the day great companies do less and obsess and
this is very true and you'll see that a big part of okay ours is setting no more than three to five objectives. So the other part of a goal is the fact that you actually have to do stuff to achieve that you can't just come up with a big aspirational goal and dream of how to visualize that you have to actually do stuff and Andy Grove who actually invented okay hours. I didn't tell many many years ago. So it almost doesn't matter what you know execution is what matters the most is all leading up to it. You'll see for the oak ER structure.
And um, this is a picture of the Dead Sea Scrolls from way back when but on the left is actually one of the first okay ours. That was created by Grove in 1984 Intel and this was the objective kind of it is something that is meant to be aspirational establish this particular microprocessor as the highest performer in the industry. But then he said well, that's the aspiration but it's not very specific. Why don't we backed it up with how we're going to do it? And those of the key
results of these are the actual okay are from 1988. Perhaps the first official okr ever you see the key results would look very measurable develop and publish 5 benchmarks showing Superior performance repackage, the entire family products should be measurable and get the 8 megahertz part into production. So you can tell us a pretty far back in the in history 1/8 megahertz was considered an aspirational goal key result you'll notice also here we have these numbers 0.6 1.0 + 0.0. These are basically
showing after the quarter has ended what percentage of the goal was achieved 0.64 the first one a hundred percent for the second one and zero on the third one. How do we get to this point we get to Stonehenge over there by ancient history. I'll go over there. Not that ancient 1940s goals were pretty easy, which was make things fat run faster cheaper better make more money and more profits and we basically had process process optimization and that I'll goals
were defined by a Senior Management to tell everybody what to do. 1950s management by objectives came around that still very valuable. I've been more collaborative and setting goals in this was represented. The industry was changing. It wasn't clear exactly what you needed to do and you really needed more feedback from your managers and Year from your workers as to what particular goals and objectives should be in what's what's the Super Bowl and what's aspirational? Then in the 19 are the two thousand and there was other
goals in there as well. But there's a balanced scorecard where it said it's not just the financial goals, you know think back of our Wells Fargo example, but we also have to have soft goals that might be for instance employee satisfaction or ability to recruit or training for our for our employees and that actually works very well for a time. But then okay ours was developed to address some of these other issues that we had with goals. Mom that came from these other systems and it was particularly good match for high
technology. Again, Andy Grove invented it, but John Dora was kind of the Johnny Appleseed Pied Piper of letting everybody know about it, and it's certainly served a Sergey and Larry Page very well at Google. So let's look at an example. I'm up here in Boston. So we'll look at our football team. Oh, that's Robert picture of Robert Kraft there at the owner of the team and his objective might be to become the most valuable and most respected team in the NFL.
And that's kind of an aspirational goal. It's not particularly measurable. I guess you could sort of measurable but it's very aspirational. It's also very inspirational but the key results to get there would be to win the Super Bowl that would imply that you aren't the best at what you do to make money. You need to sell at the stadium but there might also be other key results that are measurable the Patriots have had a history of problems off. The field with deflategate were Tom Brady was deflated footballs or Spygate where the coach
was perhaps videotaping the other team incorrectly when Robert Kraft had some issues personally as well. And this would be important for having a team that's respected. So you can see the objective is very broad, but the key results are very specific. Now that would get trickle down to the head coaches of Bill Belichick. How much younger Bill Belichick over here on the right and his objective would be to win the Superbowl? I'm and his ski results might be results are on the offense having a strong offense that has a
lot of yards per game and a strong defense which doesn't allow a lot of points per game. He then also has coaches Josh McDaniels here. I'm who has some Bozo KRS Cascade down to him. He's the offensive coach. He takes text takes that objective from his his head coach 450 yards per game and he breaks that down into passing and rushing key results. Then Tom Brady works for him sort of and he might take just the passing results. But he's aware of all these other goals and objectives throughout the organization, but he might take just that result and then he might break that
down into specific heat results in terms of completion yards per game being able to play in all the games without getting injured low and receptions. And of course don't deflate the footballs cuz we know that's a for the owner of the sea used to not have any more bad press. So, why are okay there's good? Okay are separate out the objective is a single aspirational sentence from the key results which are very specific and measurable. This allows employees to dream big will be held accountable to measurable outcomes.
So it's sort of addresses this problem of sandbagging it allows people to shoot high, but it also makes it measure so it's not just a dream. It's actually something about how you will actually get there. And one of the other great things about okay ours is they are simple they do not take a lot to implement. Another cool thing about okay ours is that you don't need to get 100% on them. It's kind of this weird thing that if you end of the quarter and you got 100% on all of your key results
I'm your boss would probably should come to you and say hey, you didn't really set your jectors high enough. OK are generally expect about a 70% completion rate for the the KRS. And again that kind of takes the pressure off. It says look we want you to set goals. It really are hard to achieve and if you don't achieve them, that's okay and the benefits of management of that is that some that means that the people that are working for you are shooting high and they're on the same team. They're not fighting with you and not stand back and this is another one of the great things
about okrs. Another good thing about okay ours is that they are transparent. so we talked about printing out the Okie ours and putting them in office stores. Now, we're not all working in offices anymore. But if you do share them digitally you need to find ways that they don't just get buried away on some shared Drive someplace then have to be front-and-center and I always say that the CEO of the company should be able to walk down the hallway and stop any employee and ask them what the company's 325 objectives
are in the employee should be able to say that so transparency is very very important. Everybody knows everybody else is okay ours and ideally they would be there to help them achieve those okay hours and if a person is having trouble achieving their okay are they should ask for help and people should be willing to give that help in all of this comes from the fact that it's completely transparent. We got a picture of a transparent jellyfish here where you can really see the inner workings of the animal. So do the same thing for a business deal cars
expose expose the inner workings of the business and what that does is put everybody on the same team pulling in the same direction. Do you know what your boss is? Okay, are you know what your bosses bosses, you know what your score and as you know what your peers are and it really builds teamwork in a great way. I never get off this is teamwork. I'm okay or is it typically not tied directly to compensation? I would suggest that a small bonus. Maybe you want percent of total compensation tide directionally to the okay are you don't
have to high-a bonus because then it kind of crops the system. Okay hours are flexible. They are done every quarter which allows you to modify them and to match the rapidly changing business. Although if you could get through the quarter without modifying then that's obviously better. In the talk and you guys will get the talk after this. I have a couple of exercises you can go through but I'm going to skip over that for now just cuz I'm kind of running out of time but What you go through those you can go through those on your own or we can chat
afterwards couple of questions about. Okay ours when John doerr presented this the surgery and Larry he basically pitch the idea and they're like man, I don't know. It's obviously a big change for a company women organization. So when you pitch this internally just pictures they hey just like making pancakes in the first pancakes usual amount little bit malformed doesn't come off The Griddle that well, but let's try it and see what happens you strive for a quarter. And if you do that this kind of there's no pressure on you to do it, but you learn a lot
and if successful then you kind of started introduce. Okay Arc organization. So Mochi hours, you need to do better than 70% If you have a legal requirement by an SLA for 99.9999% up time and you got to do it. So 70% is not going to get enough. She really got to hit it. Show me common mistakes with okrs complexity is the enemy of okay ours. So picking more than five objectives and key results is typically where people get themselves into trouble. Making gnocchi are descriptions to complex. They should be literally like one liners. And again, if if they're too complex, they're not
going to pass the walking down the hall test that people can tell you what the objectives are. I getting too carried away and tying the okay are so big bonuses or changes in compensation. I'm assigning shared accountability. Like here's a you know, everybody should be there should be one person on the hook for each of the KRS because share responsibility. Basically that person is on the hook for the KR can ask for help from anybody. So the team is working but that's be one person who is responsible for making sure that that happens.
And obviously not taking two days. Seriously. That's a cultural thing. That's really important. So in summary okrs, the mission is why you're doing it Bona was there to save the world? That's his mission is a objective is lowering poverty in particular places in Africa IQ results might be how he's going to do it specifically what he's going to do that quarter to get it done. The benefits are helping you to focus on the most important lowering stress free
organization on your end of the year individual contributors. And I think that second one lower stress is a really big one cuz that's really a problem and then also Increase trust her accountability support from others and generally a feeling success when you say when you do what you say you're going to do. I might have some other slides here that you can take a look at when you get the deck. These are just particular. Apis that are good for AI projects and this is a
pretty simple way to actually get it launched kind of the recipe for getting a lunch within your company pretty simple to do. If you want to go online, you can actually take a I think it's like a 20-question. Okay AR quiz, just to see if you were staying awake during the stock at least their instructions to do it again will be sent out with the slides afterwards. And if you have any questions, I guess I've 30 seconds here follow up. This is part of a longer training course and implementation program that I have if you're interested in just contact me to see the jog
nada.com. Also email me if you want to come and get a copy of my paper. How do you say Okay hours for your AI team and then for sure take a look at John doors Ted Talk that's on YouTube or 10. And that is it.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.