About the talk
Models are only as effective as the design of experiments that evaluate, compare, test and validate them. Between real-world constraints, limitations of your computing environment, and everything in between, there are many decisions that need to be made to design an experiment to ask the right questions. These decisions and how they translate to more insightful experiments is the subject of this panel discussion. Michael McCourt, Head of Engineering & Research at SigOpt, will facilitate a discussion that cuts across a variety of enterprise and university research applications to distill a set of foundational principles that can be universally applied to experiment design. This discussion will touch on how Paul Leu from the University of Pittsburgh applies multiobjective Bayesian optimization to run more efficient and insightful simulations for glass design. It will cover insights from Vishwanath Hegadekatte of Novelis on how enterprise constraints inform the way he structures experiments related to recycling processes. And it will include design decisions tha Marat Latypov makes to boost the efficiency of his research amidst resource constraints. Attendees will leave this panel with practical lessons they can apply in their own experimental design processes to drive more efficient, effective and scalable modeling.
Marat Latypov is an assistant professor in the Department of Materials Science and Engineering and a faculty member of the Applied Mathematics Graduate Interdisciplinary Program at the University of Arizona. Marat received his Ph.D. from Pohang University of Science and Technology (POSTECH) in South Korea followed by a postdoc at Georgia Tech/CNRS lab in France and the University of California in Santa Barbara. He joined the University of Arizona in 2021 after two years in the aluminum industry. Marat has a wide range of interests in the field of materials science including materials informatics, modeling and simulation, artificial intelligence, materials design and process optimization.View the profile
Mike leads research and engineering at SigOpt, an Intel Company. We are responsible for developing, deploying, and maintaining the SigOpt platform both by creating tools to power intelligent experimentation and making sure these tools robustly meet the usage demands of our users. He has been with SigOpt for the past 6 years, almost since its founding. Prior to joining SigOpt, he spent time in the math and computer science division at Argonne and was a visiting assistant professor at CU-Denver where he co-wrote a text on kernel-based approximation. Mike holds degrees in Applied Mathematics from Cornell and IIT.View the profile
Prof. Paul Leu is an Associate Professor and the BP America Faculty Fellow in the Industrial Engineering Department at the University of Pittsburgh. His research group the Laboratory for Advanced Materials at Pittsburgh (LAMP) focuses on functional materials which have included functionalities such as antireflection, light trapping, and haze control in plasmonics, transparent electrodes, and solar cells. Recently, he has been integrating simulation and experimental methodologies with machine learning for materials discovery. He has been recipient of the Oak Ridge Associated University Powe Junior Faculty Enhancement Award, UPS Minority Advancement Award, and the NSF CAREER Award. His research has been showcased in Scientific American Frontiers, Pittsburgh NPR, and Pittsburgh Magazine.View the profile
Vishwanath Hegadekatte works for Novelis Inc. as R&D Manager and Principal Scientist at its Kennesaw R&D Center located in Georgia, USA. Apart from people Vishwanath Hegadekatte works for Novelis Inc. as Global R&D Manager at the its R&D Center located in Kennesaw, GA. At Novelis, Vishwanath leads the global AI and Advanced Modeling group. Apart from people and project portfolio management responsibilities, his current scientific areas of interests include machine learning, advanced modeling for sheet metal products and processes. Before joining Novelis, Vishwanath worked as a post-doctoral research associate at Brown University, USA. He holds a doctorate in engineering sciences from Karlsruhe Institute of Technology, Germany. Outside of work, Vishwanath enjoys experimenting with espresso, hiking and landscape photography.View the profile
Very excited to be here. Today. I am Michael McCourt head of engineering here at cigarettes, and I have the great pleasure to share a panel discussion with three really outstanding users of cigarettes today. I'd like to introduce yourself in particular fish. Would you mind going first, please? Thank you. My name is vishwanath mobile and I work out of the Orinda Center Supply, aluminum sheets to the garbage can in the space and also be adding. Yes. No. It's actually, thank you. Thank you very much fish, right? Maybe we can go to you next.
Thanks, Michael. My name is Rob. I'm an assistant, professor of Materials Science and applied math, the University of Arizona, in Tucson, Arizona, and I just started here, working on time and materials informatics. And I used to work with this one out at novellas. Excellent, excellent. Great to hear it. What's, what's a department? You say you're in again at University of Arizona, Science and Engineering, Materials Science and Engineering. Excellent. Thank you. Paul. Would you mind introducing yourself, please? Everyone. I am
about to call the University of Pittsburgh associate, professor and America fellow. I've been here at University of Pittsburgh since 2010 and need the laboratory of laboratory, for Advanced Materials at Pittsburgh or lamp restore, my Soco direct National Science Foundation Center. With the Case, Western Reserve University Center is called the center for materials data science reliability and degradation. Excellent. Excellent. So everyone can see a star-studded panel really great to have you all on board. And what I'd like to
do in this discussion is chat about how cigarettes can be used in this in this design space thinking about in this case soon. So these materials problems for everyone here, but then also just more broadly than on ML side of things. Obviously, we're here in a lot of people talk about ml but they got can be used in a lot of others setting. So I in particular fisheye. I don't we just heard your talk, where you were explaining how you're using segottes to design
materials in particular material that then you run these a finite element simulations on if you can explain a little bit about how we still got plays a role in this process. Yes, sure. My group he was the one who introduced us to Cigar in the first place. So you've been sending me and 80% of it was put in place, right before coming back to your question. We are coming down as Moroccan. Watch for this. What we did was when we started using cigar. The idea was to use
Bayesian optimization in place. Of the regular Six Sigma methodology is the factorial design. Now, the company is a time-consuming and expensive and has the biggest impact of using a narrative design of experiment methodology. I am busy enough fumigation would be the the Riviera. Undercover agents are easy to convince people in my team to use Bayesian optimization. It's not, it's not very difficult to do that. So the work is still in progress, but we have to
use this technique. So index in the examples that I showed today in the software, that will be used to give us the next best position for running a financial Domestic Relations. So what it means is that the final assimilation means to be programmed in such a way that it is on the Fly and then it it cannot run the next best simulation. So that way what are designs for targeted performance? And one good thing that comes out for free which is the underlying surrogate model. So now the sun is trying to make him
explain to learn the lay of the land. In 42 find the global Mini. it does a pretty good job of finding the lay of the land that can be used to replace the number of experiments that we don't know the difference between between Tampa and In laboratory experiments with how to conduct experiments in batches the Derby, run in the lab or if you're if you're getting some time in one of our plans. Now, but in a minute at a time, Abandoned in a particular case of the base station can run a little more efficiently or a little more accurately. If you will. Then I
in this sequential setting. I'm wondering if Paul. Can you comment at all? How closely does wood fish? Describe match, your own experiences working with a sick up. Yeah, so our expense has been very similar instead of this Six Sigma design process. We basically were looking at it as a better form of optimization. So doing machine learning by Eugen, optimization. And so we did some benchmarking operation with the mic and Harvey for a cigar box and looked at a
Terrain, engine optimization with genetic algorithms and our work was a work. If you had a chance to look at the presentation was focused on glass. So we focused on trying to improve the transparency of the glass, basically improve the Answer your question, properties. We looked at the glass across different wavelengths. So, you know, for a lot of Auto electronic applications, things like this plays Windows, you want to maximize the Pharisee in the visible wavelengths of. We also looked at, for example, the solar module glass, where he wants to maximize
transparency over all the solar wavelengths, that include things like stew and bread, and So yeah, we did both some of this combination of computer simulations as well as a experiment and very similar arguments were also done in batches. We did them in batches of 55 experiments based on suggestions from the by using model simulations are also done sequentially. Oh, yeah, our has been very similar in that. We really wanted to reduce the number of a computer simulation that we did
as well as the number of surgical experiments. So yeah, we were able to glass with the enhanced transparency and Clarity at repellency of different liquids. Using this process. Paul in, in previous versions of this, would you have been primarily doing a classic design of experiments, a strategy to study our options. Here is that we might have occurred. Yeah, I mean, I think what a lot of special, especially with experiments. What a lot of people seem to do is, there's really a very large design parameters, 78 variables, possibly
even more. And what a lot of people do is, go just sticks. All of the variables except for one and then just systematically very that one variable. So, you're you end up really searching, a very limited part of the old sign proof of the bishan. I will really be able to explore and exploit the whole more thoroughly. All right. I'd like to hear from you. Now. In particular, you've seen things now from both sides. You've been on the industry's side, you're now on the academic side, but you're seeing houses. The field
is above it may be potentially hopefully evolving away from just demanding, the classic full sets or you'll design for everything. Just something potentially More Sample. Efficient. I'd like to know what your thoughts are on this transition. As you've seen it in Industry, as you're seeing it now in academics. And what, what really needs to be done to help push people who are doing design more in this, a sample efficient Direction. Write a very good point and I think one of the key factors here is the cultural shift or which
was also mentioned in one of the talks. Are we? We, we already brought up these engineering statistics approaches like Six Sigma that people get strained and they learned it during their engineering training. So they're familiar and more comfortable doing this. And when will you bring this new machine learning all day? Too driven approaches the results on The Resistance because they blackballed and they didn't see successful case studies which are relevant to their work. And I think they're there is a lot of excitation and resistance from
the people who can actually take advantage and speaking of their calculations machine learning model. Skinning machine learning models is great, but When I came across this Bay is in a position approaches and she go in particular we found that the best way of would be in experimental work or in the plant trials because each of the duration is really expensive compared to either physics by stimulation, even if it's time-consuming or tuning machine, learning and training RAM and we felt that we could see a lot of payoff in
reducing the number of observations. When we learned. Our team has a Sheen to paint after the same time. I think there was much less comfort in adopting this approach has And so we we've done all these computacional benchmarking studies. And yeah, we see a lot of speed up in the computation of field to do the same in the experimental world is much more difficult because it means we would need to run in parallel to experimental campaigns. And most often times. We cannot afford to do that. And
so, yeah, that's that's one of the constraints is that we're Limited in experimental. A benchmarking to go to other Advanced. Data-driven approach is for accelerating optimization and experimental discovery. That's one thing. And another thing is that in our field data, especially in their research field experimental, research field. The date is extremely limited so that we cannot leverage a lot of historical data. And on top of that our observations and budgets also extremely tight. When I see all this case, that is
for cleaning machine, learning models. Yes, this is great. We can do, we can do the recommended number of observations, but then In the experimental side, we have extremely tight budgets and when a black book software for an experimental is the engineer. Give some suggestions. I think people would like to see some more information. The first W went behind the model, that makes the model believe that this is the next best of the relation to do. Or one thing that I was frequently Asked is and we get the estimate from the model that we
will observe if we actually follow the suggestion, so that isn't this is one feature that I was frequently Asked whether it's possible to give the estimate of prediction of belief of the mother. What will happen if we follow because it would give some comfort to the experimental Baxley. Follow the suggestion. That's a, that's a fantastic point. And I'd like to I'd like to follow up on. Exactly that topic there because you were talking about what is it going to take? What's it going to take to get more comfort in the community
more willingness to embrace these very sample, efficient methods and I can recognize that. Yeah. If you go away from the full factorial design, you you lose some sense of certainty about the space because you haven't fully for the space. You've been exploring the face in a very Pious way towards the optimum the benefit, his, the gain in efficiency. The drawback is the concern about missing out on things or or perhaps the model being flawed. So I think that that's absolutely something. The tools likes they got in our
research community over all this sample. Efficient exploration Community can be providing to practitioners to make them feel more comfortable. This, do you have any thoughts on other things that they're the community to be thinking about to help make our tooling more accessible and Advocate more effectively for the use of sample of the expiration date was doing before. Before he left new, lsv was actually created for our return to spend a lot of money about 15-20 years ago. Painting that people with Six Sigma methodology
work. Confident that the tool work and they don't need to learn anything new and so they go forward with what they know already done something new. So you need to let us know about about Bayesian optimization as a more within our internal training programs. So that people are exposed wanting, what you guys can do, is probably have some kind of a, an equivalent to 6 Sigma Black Belt, for a certification that so that the company as, as as as an excellent painting.
Olympic Valley weather. Especially among the experiment list to use the tool. But I left, there were many people who wear concerning the road emails to me asking about who is going to support us with this. And then we actually are you how are the support? You don't worry about. It is working about 6-7 months for us to get there. Let me let me ask home. Now. Do you have upside, right? Did you ever follow up on them? Yeah, 5 follow up on my previous comment as well. So you can be absence of opportunity and resources
to run a different UPS. Innovation strategies in parallel. One of the questions to the machine learning and bathe in its Malaysian communities in 25 success, if we cannot run the Benchmark and well-positioned status and yeah, and I think another challenge that they didn't mention before inserting experimental work, is that in modeling? We have a nice set of parameters and we we can Define them. Exactly. And then we can measure the response or metric very clearly, and carefully
I've ever in the experiment. I mean, even sitting out there to Malaysian task is sometimes challenging because we have a lot of metrics, or should we include them in, in their Innovation should wear and then one by one? Open be metric Specialties be metric. Constraint and leave. A a m a a m. We now can affect the result so they cannot be directly control. If you change one or the other parameters in directly swayed as well as industry in academic experimental research. As there, are many challenges that you don't
expect coming from a machine-learning modeling autistic by Starley. Very good point. Very, very good point. I wouldn't forget that as well. Let me just ask very briefly, both Murat and Paul. Do you feel that? There is any element of publishing which is somehow problematic in the mix. You're in the potential. Mix for shifting away from classical design of experiments. You feel that editors for reviewers really expect to see the full factorial of fractional, factorial design of experiments? Or do
you think that by and large you can do as you choose in? As long as the eventual design that gets used, is a winner that's enough. Murottal ask you first. Yeah, I don't think there is a restriction in this case or some expectation of using a certain set of experimentation prodigals and I would say it's the opposite ways an optimization is now very fashionable in mechanics of materials and materials. I'm so just just revisiting simple rhyme with their feeding problems with the maze in the simulation approach and leads to a
paper. So yeah, I think to to make this my district happened which made the very good point that we should Target the training and education center in the company of can be a small course Mojo but we are being in Academia. We can Target it's even earlier in the process. We're training a new generation of Engineers and I think we can discuss with the machine learning. Any chance she got that mental, how what would be the best way to interpret these data driven approaches into engineering curriculum? So that when they come to novela some other manufacturing company, they would be,
the seniors would be much more open to this new approaches and would bring it to the table. Call. Do you have a, do you have a thought on that first off? How is it that you or your students? You're both in your department in your your own graduate student. How is it that they can come to learn about Advanced tools? Like this. Is it at all? Actually, in the classroom or is it, is it most common just get their advisor exposes them to it? Probably a combination of both sick right now.
There's through a little bit of a separation between Material Science and data science communities. So I take a lot of your materials papers. They just tend to focus on fundamental characterization demonstration of some new material with your property, or your functionality. And then a lot of the science Community I think. Focused on perhaps more algorithms or some of the fundamental under 16. So, I think that's perhaps one of them in terms of. Getting our usual optimization out to the
materials communities. So I think, you know, having additional classes and I think there is a big thrust than that and universities additional Majors additional minor certificate things like that, where students are trained more in that and then certainly at the PHD level if you have a faculty that are Interested in both of these areas. Then their students will certainly get deeper into this. Yeah, I think it's a very good. I think it does. Seem is a very new area. Thank and I think more more people getting into us. What can you use fokken
earlier about some work. You've done specifically with the goal of having materials scientist data scientist working together. Speaking together trying to make new results happen. If you could elaborate a bit. So yeah. Do the work that we did with the, with the new types of glass. I was thinking in particular the work you've been doing with Case Western to build up. Bill Consortium, was it, you know, we we have a new National Science Foundation Center,
IU CRC industry, University Cooperative, Research Center, in the center. As a mentioned, in the intro, is the MDS or Ally for short materials data science for reliability and degradation. Do a lot of this. experimental data is Brady are severely limited and Specifically with regard to reliability data or how much she owes properties change over time, how different functionalities change over time where our Center is focused on how these properties are functionalities. Change under various stressors of, the stressor could be exposure to ultraviolet.
Light could be changes, in temperature, temperature cycling, exposure to scratching or some soft fog. So I think this is an important. Ariat expecially with regard to bridging research and development. That happens in industry, and I think this by Eugen optimization could be an area. This could be an area where bizon optimization also makes a big impact because I just be a cost of doing an experiment where you need an extended Bossier, to, to some sort of stressor, even under accelerated testing these
very little and take a long time to run music. 3.3 point, and yes, so many of these situations have this need for sample, efficiency. I'm glad to hear that. The nff has agreed to support this project. He said look forward to seeing some big stuff coming out of that the future. I want to go back now to a point that that mourad had talked about which is this question of a we have a bunch of of metrics are restoring them all. Are we trying to optimize some
of them? Do we only consider some of them important up to a certain point? And then it's like it's all good which might be more of a constraint style situation and I think that actually in some ways it's kind of close to something that Paul talks about earlier, which was his question of. Okay, we might have a hundred or a maybe, I freeze all, but one of them and study one of them at a time or or maybe I just turned on for something like that. I think that at the end of the day, no matter how you're studying your problem, whether it's with the full picture of the sign or without sensation.
What is it that I'm going to measure what is it that I'm going to record and over. What space will I be interested in inconsiderate set of possible to sign this? I'd like to ask you how is it that these decisions are made at novellas when experimentation is taking place. How do you decide what the acceptable or design space of interest is and what the metrics of Interest are. I know that you would given us in your talk and example where you studied I think for parameters to improve the the pressure of the the shape of it and was able to be used up.
But I know what you said. Maybe there are hundreds even how did you make a decision like that? Yeah, I mean most of it is based on prior knowledge of the subject of operation and also, so that's not a surprise and we have a full weekend and we can make up a n dentist in town. So so that's That's how about a recent example was that one of our Engineers was using cigar? Add to be using minute. So then I asked him the question that I'm interested in which I understand very well
is what I want to see and that I can see it with me Nita. So, so what what I think is is needed. Is that here? That is an example where where we probably do not fully understand what I am. We need to include in our study. And it is those interactions between the various parameters, that, that is probably guiding the engineer to apply his experiment. In my mind somewhere. We need to have a We need to find a way where people can easily be trained from from what they know about the Six Sigma methodology to
the Newark Acme, s currently available. If it goes with the with the idea that okay, forget everything that you know, something new, that's not going to fly. Yeah, maybe it's a good example of a low-hanging opportunity, how to make a that transition. And in the one of the earliest the speaker mention the cultural shift in also very industrious sizing. And their answer was how to how to make this couch out of shape is the language that the engineers understand Amanda Eve Eve
software like Sigel, I can provide visualizations that that people are used to based on their engineering background. It could be also some comforting Zone and transition exam your, I can see the stuff that I want to see, even if it's using the underlying engine optimization algorithm is different. Maybe it's an opportunity to go about it. I like that. I like that. I think that is I even set up two nights, good opportunity. All do you have any thoughts? Then about how these decisions are made around? What parameters you consider
in your design process? And also what metrics your optimizing for versus just watching versus constraining? Based on how do you make those decisions? Based on our experience. It's probably best to just record as much data as possible. Even if you don't think you're going to use it still better to record it cuz you might later come back to it because Bishop said a lot of those decisions, I think are based on your physical intuition, just your experience and working with the design problem with the material.
But yeah, I think a lot of times your physical intuition can be wrong and why it's best to Keep the problem as open as as you can and allow the guide the process. Yeah, I think even pissed a lot of people just simply just using their physical intuition, I think can make Y'all come up with Bree new designs, our new products and such, but it's still, it's still, I think fairly limited. And I working together with the Bison optimization we can We've shown that we can come up with so much better designs. Much better properties and functionalities.
And I think speaking to that point and pushing and maybe even one step further vishu, spoke about the idea that good values but it's not it's not getting me all the information that I want to know how I actually want other information than just the optimum. And I'll let you guys in on a secret. Many customers have the same feeling and it really is born of in some cases, the sample efficiency, the fact that if you're going to be sample efficient, you're going to get less. They do you get my stating
loan less about what's going on and it can also be born of the fact that I got this sampling. It's just her. And I can feel unsatisfying and it can feel like you didn't learn as much as you would have during a a full picture of the whole heartedly agree with that. I'm wondering. Is there? Do you have any thoughts on what some sort of Middle Ground might look like, or what it what it might look like to want to do optimization to still try and be sample. Efficient. What what other
data can be provided or or how can the energy in the gas? Maybe be a little bit split up to also learn some stuff about things. You have any thoughts on what that might look like? Or or what the, what the desired outcome would be there. So, I know from the problems that we worked on together. We worked on these multi-objective optimization problems. So there's not just a single Optimum, but there's really a parental Frontier of Optum that I could be a line, or some Curves in higher spaces. And we've done some
post-hoc analysis on what are the solution that's Paul on this curve on the surface that Storm's the preto frontier and we've been able to get some sort of physical inside in 20, you know, these types of designs seem to be best for her example of a reflection. And even though, you know, it seemed like there was earned an angle of some of the structures that were looking at that seem to be favored and and being on the stuff ready. Cradle Frontier. Now, we weren't exactly sure. Why does angle was necessarily the best but at least it
didn't seem like there was some sort of relationship. There. There's some sort of underlying physics that makes these sort of solutions Optimum. And I think perhaps that's where you know, the the underlying scientific knowledge to better try to understand. Maybe this is why these sorts of solutions tend to be optimal. The point is a very good point. Do you have any thoughts on that, what sort of additional analysis could be done or, or maybe in particular, instead of spending 100% of the energy just on optimization. Could it be
like a 70/30 split optimization and something else? I'm wondering you. Yeah, I think I'm the context of extremely limited observations budgets, the people often ask. So we we set up the musician problem and she got pick up generated suggestion and the subject matter expert can be surprised about the suggestion. And second following the surprise. There is always a question. Why safe? If you stick out in other ways in approaches, could provide any indication? What
drove to go out of this particular suggestion would be very good. I think, Scott, in the, can a session mentioned if we could provide information on the particular acquisition function, that drove the suggestion that could help, but also, yeah, I mean in Sea Girt, we don't know if, if this particular suggestion is driven by their exploration with the variation or exploitation. I'm getting to the local big. So if anyone has Grey's Anatomy station experts, you know better what
indicators could be used to so inform the experimenter better. I think it would be very very So there's there's, there's an emotional element there to some degree. There's this sense of you don't want to be wasting the few experiments that you're actually going to be doing this. Would you agree with that? Much better. Very good, sir. So I'm studying design of a beverage champion. And what is the, the little bit more than learning space, which it has asked us to it and outside of the parameter in which it has not asked
us to wait. So, what exactly is it flooding in? So, my feeling is that, that somewhere there should be some clever way of showing this pictorial as the optimization is, is moving forward in this family. But something more than that to show. That's that something that actually makes sense that the user that when there is a reason behind wise to go up or any other busy and basically asking you to do that experiment experiments on, I'm going to be slightly random because it needs to learn the lay of the land. Did we have seen that? If you have
any prior information about the problem, then then submit those results to the division of something. So that's kind of what I think and currently with a g y somehow, the surrogate model is not being used as much as it should be used in the center, with more than is in the background. You don't really know what it is black and white. A lot of people that I know what would be the the the the desert. So so I think you have that isn't it is just a question of
the gym and also somehow during the Victorian showing what is going on. I know you like a point. That maybe you could show sections or particular parameter and just use for work permit or 20 that respond service currently learn. Do you think and pulling a we've done this at one point? I do think that if this information were available at some of these experts, what do you want to make their own decisions supersedes. They got to take the time to wait to try this instead that might start happening, which could be cool by that
has been designing, a new ally has to do certain things. And some of those things may not be good for me, then go and change the dababy producer of a normal State, and you may get lucky and get a better result. I'd like to hope so. Okay. I think we do need to wrap it up right now. We are about to terminate this session. Thank you all for participating. This been outstanding. Thank you. All in the audience. We are going to do a q-and-a session starting right now. At a
Buy this talk
Buy this video
Our other topics
With ConferenceCast.tv, you get access to our library of the world's best conference talks.