Duration 39:24
16+
Play
Video

Future of Forecasting the Future in... By Arun Verma, Head of Quant Research Solutions, Bloomberg LP

Arun Verma
Head of Quantitative Research Solutions at Bloomberg LP
  • Video
  • Table of contents
  • Video
Request Q&A
Video
Future of Forecasting the Future in... By Arun Verma, Head of Quant Research Solutions, Bloomberg LP
Available
In cart
Free
Free
Free
Free
Free
Free
Add to favorites
81
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speaker

Arun Verma
Head of Quantitative Research Solutions at Bloomberg LP

Dr. Arun Verma joined the Bloomberg Quantitative Research group in 2003. Prior to that, he earned his Ph.D from Cornell University in the areas of computer science and applied mathematics. At Bloomberg, Arun's work initially focused on Stochastic Volatility Models for pricing & hedging Derivatives & Exotic financial Instruments. More recently, he has enjoyed working at the intersection of diverse areas such as data science, cross-asset quantitative finance models and machine learning & AI methods to help reveal embedded signals in traditional & alternative data.

View the profile

About the talk

In this talk, we cover various aspects of forecasting methods used in finance. A smart consensus of various economists, brokers, analysts and forecasters is a good way to think about forecasts and we illustrate the methodologies needed to score, rank and aggregate forecasts from multiple sources. Another important area for consensus forecasting is error & regime change detection. We also talk about the challenges of calibrating machine learning models to partial ground truth data on errors and how an ensemble methodology is needed to precisely identify different classes of errors accurately to allow for automation of the entire process from error detection & removal to scoring/ranking of forecasters to the final smart consensus forecast. We will demonstrate use cases in company financials estimates, economic indicator and FX/Commodity spot rates forecasting.

01:02 Bloomberg

06:25 Forecasting is complicated

11:40 Lucky and wrong forecasts

16:20 Median and mean absolute deviation

19:26 Sourcing truth

25:16 Regime flag

29:45 Factors and principles used for ranking

33:57 Scoring forecasts for tradable and non-tradable indicators

36:51 Being a consensus premium

Share

How's the weather nice introduction? Yes, as you said a lot of the folks in the Call of Duty finest came from you know, physics labs and physics background. I initially so, you know, it's the finance field is actually very very scientific institution because of that cancel the physicist from Bell labs and other places so nice to meet everybody. Good morning to those in the west coast. Good afternoon to those in East Coast or Europe or whatever else in the world. You are I'm going to speak

about future forecast in the future in finance and I work within the chief technology officer at Bloomberg as a team lead or head of the quantitative research Solutions team, and before we jump into the technical portion of my dog, let me just sore. background on our company I guess most of you folks probably no rumor guy same as a news company or a media company. Data company, but what I want to emphasize this that Bloomberg actually is formoterol technology company than anything else we could be you have a big news organization.

We have our own journalist. And we you know, we we have presents in almost every country, but about 30% of us top RX you technology people and it could a fraction of that is PST level scientists scientists or AI machine learning researchers and we have basically been putting out our product much more into a multi-platform landscape working with open source technology contributing back to the community efforts in in the technology space. for the financial industry just to

give you a bit more of what room is really all about is really taking in all this wealth of information which comes in through, you know news or analyst report Cindy from the Southside institutions all kinds of filing status from hedge funds and the owners of assets for social media and web data and Bloomberg Health survey process all the data in real-time and extract the right information out of it. And then we present that in in was our Flagship product called the Bloomberg terminal which you may have heard. So that's all I

mean product for portfolio managers and and Traders and analysts but for their scientists and cons of we are actually offering a much more flexible open platform call the broom broke on platform, which allows users to take Bloomberg data wire API. Bloomberg model is also available cloud services and really allow you to build your own models on ecosystems substantially not just taking Bluebird tomorrow as as as the default but bring your own assumptions bring your own data even

and then build a solution which is customized for you. Are you scared? And that's really the next generation of Bloomberg products and look like Of course all the things my dog is going to be focused on forecasting. So that's the rest of the rest of the content here. But you know, we want to emphasize that the investment into machine learning and AI at Bloomberg's was started with our our real-time news processing initiative. So how we tag ever seen the news item that

flows through an ecosystem with with the sentiments expressed in that news item also with entities, like people companies locations Brands as well as Copic goats. So we try to tell you as much as we can about every single document every single new story and then you can consume the data and and and build on top of it. Of course, we have taken in machine learning efforts beyond the news and the document analysis part of it to release her. I'm mad at the field. Country Finance with machine learning some

kannada Finance used to be some more prescriptive in this it is in syrup most vanilla farm at earlier. So lord of the clocks were off of the scientific man, you know, they will make assumptions about what kind of process is financial instruments prices follow and then we would be able to try to calibrate a particular parametric process that can explain that the data but of course as the amount of data and as the amount of coverage in terms of number of

companies number of ASL classes number of instruments have just gone after many states vs Olaf put that on his head and said that let let us let us not make assumptions about how we think you know, these prices are evolving all this process of behavior. Let us learn that from the diesel directly. So without making any strongest options, so that's sort of bringing the best of data science machine learning and marrying it with the corned finals discipline. So use the structure whatever you think is really present, but when it's not present use the best of the learning models

start to uncover the truth wire processes and one of the line that is in the forecasting and that's where Bloomberg is obviously synthesizing a lot of information that comes in through through our analysts. So we have become an economist on a platform. We have sell-side researchers were giving us estimates for various kinds of metrics. So, you know for sure or dividend or sales revenue for every company and Bloomberg is taking it that was some of the information and then combining it

or are you getting it? Add a company level to see this is what we think is the best forecast for IBM sales and I'm going forward knowing that all these forecasters are coming in into the single single a process and and we have to provide the best additive consensus using machine learning method of forecasting problem. And you know Paul Samuelson the female Nobel Laureate, you know, Duncan Sheik's has said like stock market since to 9 out of the last five receptions, which is essentially an overstatement about overfitting how we are sometimes build models which are which are all fit into

history and how can we avoid that, you know any more robust framework and if you need this has been said about forecasting methods in general and Does a book which bus Which which was a best-seller New York Times bestseller by the famous author of the book Philip tetlock soon Economist at Ivy League institution and you can and basically that the theory is that all forgot to say bad but there are there some forecasters which is better than the others. So this is notion of foxes vs. Hydrox. The foxes

are the kind of forecasters. We should know many things about about a certain topic so they will know they'll have a broad knowledge about a certain topic and hedgehogs are the so-called experts which we have knowledge as opposed to Broad knowledge about a certain special topics that we offered with a journalist and Gardner. Fez that combining many foxes that you better than using a single Hedgehog which means that if you just take many many people will know how broadly about the subject and just average

year did answer or come up with the consensus you would do better than a single expert. So, you know, this experiments that have been done in particular you want to submit some of the crowds. We're about a hundred years ago. There was his experiment which got the weight of a call within one point accuracy by just averaging Common People dresses. So that's the Best Buy game maybe cherry-picking example, but but but but the idea of the crowd at Bloomberg, we have this function call whisper where

anybody, you know ass with a cystogram of systems and Ashley, you know this but a number with visible means to submit the forecast for NE. Any Target indicator weather is foreign exchange rates are economic indicators or commodity prices and we will put all that information together and provide you with the wisdom of the crowd answer and of course, you know, this has been proven or art or art shown in other areas where apparently regular people are better than CIA.

Nothing. Nothing really new ideas is known that wisdom of crowds is definitely a good way to do forecasting than relying on Experts all the time. But of course as did Anquan sweet, we have to deal with the Practical aspects of that problem. So when we when we dive into this forecasting Use cases. We have to look at first and foremost what you call a. Checking so we know that forecasters come come with a lot of different types of errors in there in the Raw data. So we have to first make sure we clean the litter properly again using

data science and machine learning methods. And then the second phase is also scoring. So once you have clean data, then we can think about how can we go about scoring a political analyst a particular Economist? And once we have done the second stage of scoring very good handle Ranch order or equality order of this different forecasters, and then we can combine the forecast in do it instead of a smart consensus Speedman consensus that will presumably a better forecast in any single forecast itself.

Song talking about the other check-in, you know, she coming here again in the forecast for either lucky or wrong because we know forecasting is very hard. But but it do to to make it make it make it more practical sitting here to clearly distinguish different types of errors. It's very important. So not all the single variety receiver example address which are simply like fat fingered mistakes. So like sign letters number was supposed to be around in a 4 million - 4 million in this case some sort of a debt of a company but some analysts Ender 4 million,

assuming, you know, if it was it was cashed in the right sign, but of course the numbers flow into our systems big sometimes come with the wrong side. In other cases, we are scheduling errors which are you know, usually numbers which are which are you know mentioned in thousands or millions and maybe three ski gas rescheduling billion and will be will be off by it affect housing or capture a million. So be sometimes the forecast don't come in the same units near to correct for the scaling out automatically and then there are the hardest Beast of

General outliers because that's a very subjective class of Errors, you know outliers are not as precisely identifiable as finals getting letters is sometimes a little bit of subjective in an error or is it just say just bored forecast which is away from the other four other forecasters. And then finally, we need to also kind of take a step back and say that some of the others that we may have identified may actually have been incorrectly classified. So sometimes we see there's a genuine ship in the market. So we have no discharge here at the bottom right shows you

a genuine shift in the market YouTube covid which we are all so expensive right now. So every early this year starting in March April, we started to see analyst suddenly starting to downgrade companies and ship the forecast for their sales and revenues and we should not be taking those those ships as an indication of errors, but they should be correctly classified as ever. So we have to put a mayor of regime detection on top of this error checking methodology or technology do incorrect or

directly and flag. Directions to see some of the address we have incredibly flat. Basic data science Alberton 404 error checking is start some very simple USMC have to start with a basic building block, which is the central value which we can use the median value of the forecast and the division metric to define acceptable range. So you take a median value and you take + + + + -5 divisions around that meeting value that is shown in the green bar green vertical line.

And the Shaded region is the acceptable Range region to Define invalid range and that is outside of this world range example. What is 0 - 4 will be classified as an error because he's outside that bad. But if you flip that value if you flip the -4 value and if it then lies within the range then is a sign of her because clearly if if it came to the right fine, it would have been exactly in this apparent reason and then we can check, you know, if it's not a sign of her then we can

check for it's killing her if it's more than five times the median or less than one-fifth of the median then you know clearly is it Pacific 10 x 10 x 100 x 1,000 x or something like that, which is a skidding at her and everything else is actually an outlier. So this is so basic waterfall of production. But then this alarm bells and whistles that we need to worry about. So the division metric that I mentioned earlier. We do have a couple of choices, you know, what the music either a median absolute deviation or a mean absolute

deviation of the mean absolute deviation is very similar to the popular standard deviation concept. It's actually not very robust because it is sensitive to outliers themselves. So in the mean absolute deviation will will be will be higher so difficult for you to find the outliers because the division ourselves is too large. So that's too big on the on the other hand. The meaning of Civil Division is always less than the mean after division. So the band is much narrower. So so that

we have we have other kinds of problems with that which means that we may be and we may end up liking too many others with the median because band is too narrow to almost everything fall outside of the band. So so they have these basically these trade-offs between these two different metrics and and we will find a way to Wesley International combine boat in in in are in a very sordid handcrafted from the show you we find the mean because it's it's very large mean division can be very large is a very conservative starting point for us to

fast. Will we find out who really the really true egregious errors by using mean and then we can switch to medium medium is giving us a very narrow acceptance which witch is which is perfect for a signer. Do I need if I sign that are so so here we go. So what we do is we actually been in Ensemble of these different choices so fast, we build the first Fastpass I should say we use the median and the mean deviation which could be large but this is sufficient for us to find really big skating hours are really agrees of

liars. Once you're taking care of those outlets and turn them out of the data the first time out of the data, then we can follow that up with the median division pass which is really precisely is cemented to identify the signers and then we do it clean up 3rd Bass, which is again using a means deviation to find out of the regular outliers. If you didn't then we will have a very poor outcome in the charts that because it might be able to identify the regular outliers because the outliers are in presence of a

flash of killing a dose varieties of those types of Errors to be able to do a good job of regular pliers. So this will hopefully give you a taste of how you need to serve hand Travis algorithm to make sure it's doing the right things, of course challenge in this problem as an energy source in the truth because at the end of the day all these algorithms and morals they need to be calibrated to the ground truth. Right? And of course, we would like to have the truth the whole truth and nothing but the truth but

turns out is not always possible in particular for forecasting problems. We know that we have in a million simoleons forecast in the system and we possibly can ask or enter taters to tag almost as a big round trip cost to build a training sample example Excel will have to be corrected as a very small sample so we can only afford it. Automotive truck big enough for the whole truth because that was just too time-consuming given the nature of the data and and also the class imbalance really is very important play here because 19.9% of the data will be good. Only .1% of

data is especially likely to be errors. So you can you can ask us to to spend, you know, they're valuable time just tagging date of witches, you know, clearly not an error. So you have to make sure that you get some data points which are likely to be others or on the boundary or the edge cases for us to really learn learn better from the actress has to be carefully sourced. And the Second Challenge and this point often does not get appreciated very well is that true that cells can be multifaceted. So there's no, call Melinda Sue definition of

truck sometimes so as you can see in the picture of that object looks like a square in 110 projection and a circle in a different projection and that's true because we find that 444 cast as well. Sometimes it's very difficult to tell whether an error is a sign of an outliner because if a metric take both positive and negative values be outliers, or they could be sinus and only Adida expert order me an expert in tell you that I sometimes even domain experts agree on that so you can imagine

How how how complex is this can be even itself has an inherent error in it. And then do they send her in the end of gators version of the truth. So we are heading of this kind of different errors in the in the data and you can only imagine the machine learning models which are only as good as they are given, you know, very sensitive to do this kind of issues with the two data so we can throw at us is Monday when they come back to us and they want us to give us the primordials to be calibrated on there today. Can you handle my throat? And

you know how lucky me I think based on the beast on the on the Type of Albertson w we have crafted we were able to do a very good job. So this is just a summary of a tree but what I was I was I was talking about in the previous slide Harvey have two Souls the truth very carefully. I'll be able to be with inconsistency in the truth. And you know how we can bring more scientific rigor in this process of sourcing truth and the fine-tuning of the morals and directly but here is our answer to this challenge. Can we handle the

truth? And the answer was that? Yes, we are able to handle the truth. But at least to some degree of satisfaction here. I'm going to be at three different kinds of flags are are all data points. So that the Greens Greens are designers the rest of the outliers and the and the blues are the new address. So, you know, you see this so black lines, which I had our decision decision boundary is in our algorithm. So black lines are determined using using an algorithm and you can see the black lines divide the region of you know, this

two-dimensional region into 3 into three sub-regions. And those three subdivisions are doing a reasonable job off of classify angles cancel better soon. Each region gets, you know each type of error in in a majority are so if he follow those lying position boundaries, we will do a very good job of Flying Daggers correctly. Of course. This is a metal or version is a 2-D 2-D view of a full it damage the future space so you can imagine how complex this might be. If you were to you before I go to go to show you the

sinful Glory, but hopefully this gives you a taste of how you know, the challenges of the Outsourcing and fine-tuning and then finally coming over the model that does a good job on that and and of course at the end of this again, we have to add that region protection flag. So we we look at the trend of the data. So if you suddenly see a lot of Errors being being mocked our flag which all are clustered in time. So that's over the last last month or two months and suddenly we have

50% or more of the data points as I don't use that will signal to us. Something is going on with the shift in the data. Also this amount of cheap car dealership and be busy to take those clustered set of flags. And we we call them detective detective scenario. So giving a signal to are flags are possibly not errors because these are Planet dealerships. And of course, I'm going to be dry to use use full transparency in in in providing that so that are you done or teachers can do a very good job of getting the data because they have to go and correct the mistakes. So,

you know, we we don't want it want them to waste their time looking at regime Flags because most of the region Flags will be known as we really want to fuck them to focus on the big scaling ladders in the signers current time in the system. So all of this helps in the workflow And this is so final review of the confusion Matrix that we get again. You don't have to look into each and every number there but you know biscuit highlighting the fact that the most of the the numbers are along the diagonal which is good for the computer Matrix so busy that we are able to identify a

dress properly and was very interesting is the objective function that we have to follow for certain types of Errors we care about about recall for other kinds of weather we care about position. So for example for outliers, we care more about position because of this monomer of outliers General outlets in the system, so they cannot be manually eliminated because that's just too expensive. So we want them to be super precise so that we can just order me in that process. So no manual intervention is necessary for scale and sign as his needs a bigger errors. We were

Them to Emmanuel transformed and for that we care more about the recall because you know, if even if your 60% position we don't care but we want to make sure every scale and every sign that a catcher so be again here. We see actually get hundred percent of the scaling and 95% of the sign and we're out last week at 97th and high nineties for the mattress really care about her sign position as a side like we we get about 70% position in sign sign class, but that's less important because science

is the objective which is more important is a recall you want to make sure we have captured every kind of sign her in tomorrow. Anyhow, so that's the next step in this process is about once you've cleaned up the data. How do we rank the analyst and how to rebuild a consensus going forward Bloomberg be at room in Whitby capture data for Enloe FX forecast. So this is a screen sfc on Bloomberg an agency be a rank orders different forecasters. And also this is a different screen for a clown make

forecasts and we have again ranks hear that the app without using our models. Of course, they have you only give out the top ranks and not all the rights because we don't want to you know, you know, you know just highlight the best ones and making the top 20% disability and leave the rest of the rest of the way rankings as not available so that we can take advantage. Healthy out the best in class without beating beating up the rest of the analyst. So in order to

rank and school these analysts, you know this a lot of factors that we use in a morals as you can imagine accuracy, of course is the is the topless one but we should have just stopped their accuracy is important, of course in judging the quality of the forecaster, but there's a lot of other factors which important example timing. So if forecast service gives a full cast early on should get some credit compared to a forecaster which gets gives gives you the same forecast for same accuracy

later on. So the person who actually had the same molecular earlier should be favored. Direction is very important. So sometimes you know full cast of me not get the forecast spot on but they still deserve something will go up and and the actual quantity of the metric Ashley went out. So that makes it makes it is impactful because you know, these forecasters are basically being used by by side for billing training Saturday. So as long as they pointed their defender in the right direction, they kind of

did the job even if they did not get the exact number right and that if it's a very informative today are who they want to call a call out the the most quoted years old before the other all the forecast as we go out on a limb and say that I want to be different. I want to you know be bullish on this this metric for this company and if they end up being Lights band did they get credit for that? Because they were brave enough to stand away from the crowd and give the forecast

and in terms of aggregation or building a consensus we care about these principles of uniformity and transparency to be able to come so that we can aggregate the scores across different. So across non volatile volatile. Across different measures, so not just sales, but you know, how about we come up with the ranking for analyst across sales and dividends and revenue and all those scoring should be Apples to Apples and should be able to Advocate as well as countries and sectors and things like that. So

we should be able to add any slice and dice View. I'll see you sign out of space and and have comparable comparable the scoring method. So scoring method should not have any bias or customer customisation or specific type of indicators and finally is important to our method should be you know, hopefully Expendables simple and transparent enough that we can if we can educate our clients so they can understand exactly what's going on behind the scenes and also we can use the Stars Burn scores to build this Bloomberg premium consensus

or smart consensus number as well. Bore you with this live because it's really just emphasizing about the benefits of scoring using normalization a navigation rule be around that already direction is is also known as I said, it was important factor. So for example, we want to make sure you note for FX start race. We have mentioned this Matrix of outcomes that are possible wear forecast Direction could be up download Flash and the actual could be up Donna flags and

and based on exactly where you lie in The Matrix, you could get one of the three direction direction score 0.5 or 121 is exactly exactly right Orange 5 is that you know, either you said flat or the market was flat when you get a partial score and if you got to get it completely wrong, then you get a zero score. Finally timing time is very important. So, you know, we would basically be no use the skid Squidward law of scaling to say that the any errors in the in the forecast should we should be scared

by the √ time remaining to the forecast. So this is really a way to normalize the address that you see earlier on in the in the in the production process versus the the forecast at coming very close to the Declaration time or the actual release time for some indicators, which a non-traceable. We need to use a slight variation of getting because there's a certain amount of uncertainty that is not resolved for non-trivial indicators for free that that the that the actual Bill bill bill

bill be equal to the spot rate because they are observing the market for economic indicators. We don't know the actual values that are not observed in the market until they are released in the in a comic release. We use a different method there is really no no no, no, no speakers volume is usually are known unknowns because we know how much volatility there is for the tradable indicators, but for non printable indicators, we have an unknown unknown, and

if it has to be handled using the right surgical methods, which I described here, I won't go into detail into some time. And finally we have cleaned up our forecast of their time Decay a timing as well as Direction BSM Street. Go back to doing what you call high school math. Which is like essentially grading on a curve. We basically put all the analysts and their forecast the Northern Lights forecast after all the corrections on a curve as you will as a high-school teacher will grader students and 44

cast which is exactly at the at the red line, you know, all the green olive green points in back under the curve are actually worse than the Varsity Red because they are farther away from the black Lolly which the actual value so anybody anybody who is in the white region between the between to read bass is better than this target analyst and anybody who's in the grey in the green region is worse than the Target hours. So I calculate area of the car which is green and we call that the the score

for the Dallas and then once we have the score for the end of this week and then do a vision test. I'll Dept of Peace School rated School weighted average to build a premium consensus and we've shown in our analysis that he no consensus beats the other two Benchmark on Baseline methods so would be gone Mister forward and mr. Consensus. Mr. Consensus is just a virtual forecast that we should just do the median of all the forecast. Mr. Forward is looking at the

Marcus Theater of the actress portrayed which are being currently been traded as acid me to Flat forecast and we can see that forecasts are valuable. But especially more valuable than their combined in more intelligent ways like we do for mr. Premium hair. Okay with that. And here are so any guesses on what is the future of future forecasting? So I guess the answer is we should go to the consensus. So I hope that's one takeaway here that we have said that you know, we should really go with the wisdom of the crowd. We should really harness all the

data and and the value in the data rather than going with the single espadrille singer forecaster. Of course, you know that I put it already but recently earlier prediction is remains a very difficult task, especially this about the future but we have to still be very very robust in a matters and you know Santa Cruz river that we use to build this matter, you know, this is this is is still unsolved problem or a very tough problems to really show that deep. Casting has some pretty to power and it's not not just

is it a combination of what forecast is already know so we've been able to do some back testing and treating morning to show that the forecast was really a causal driver of something that we observe in the markets as the future values of those companies metrics that I'll talk.

Cackle comments for the website

Buy this talk

Access to the talk “Future of Forecasting the Future in... By Arun Verma, Head of Quant Research Solutions, Bloomberg LP”
Available
In cart
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “Global Artificial Intelligence Virtual Conference”
Available
In cart
Free
Free
Free
Free
Free
Free
Ticket

Similar talks

Sanji Fernando
VP Technologies at Optum
+ 4 speakers
Matthew DiDonato
Senior Staff Product Manager - Artificial Intelligence at GE Healthcare
+ 4 speakers
Nataraj Dasgupta
VP Technologies at Rxdatascience Inc.
+ 4 speakers
Slava Akmaev
Chief Technology Officer at Scipher Medicine
+ 4 speakers
Shahidul Mannan
Head Of Data Engineering & Innovation at Mass General Brigham (Partners Healthcare)
+ 4 speakers
Available
In cart
Free
Free
Free
Free
Free
Free
Chris Fregly
AI and Machine Learning at Amazon Web Services (AWS)
Available
In cart
Free
Free
Free
Free
Free
Free

Buy this video

Video

Access to the talk “Future of Forecasting the Future in... By Arun Verma, Head of Quant Research Solutions, Bloomberg LP”
Available
In cart
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
566 conferences
22974 speakers
8597 hours of content