100: Cloud-based genomics using Terra/AnVIL
Sehyun Oh (City University of New York)
Levi Waldron (City University of New York)
9:00 AM - 9:55 AM EDT on Thursday, 30 July
The rapidly growing size of genomic datasets introduces challenges of data transfer, storage, access, sharing, and computing. In this workshop, we introduce cloud-based genomics platform Terra as a potential solution. Terra hosts large-scale genomic and genomic-related data sets and provide secure remote access to them and to individually uploaded data resources. Terra also provides on-demand computational capacity through Google Cloud Platform, and interactive analysis interfaces such as Jupyter notebook and RStudio. Users can run best practice tools and pipelines already implemented or upload their own data or analysis methods to workspaces. Workshop participants will learn how to use Terra through the use-case examples, instructor-led live demonstration, and the Bioconductor AnVIL package.
Moderator: Lori Shepherd
Seeking opportunities in translational research, clinical informatics, where my +10 years of experience in cancer research can be leveraged to accelerate clinical application of scientific knowledge.Перейти в профиль
Professor Waldron received a BSc in physics from the University of British Columbia, an MSc in physics from the University of Waterloo, and a PhD in wood science from the University of Toronto. He developed his interest in biostatistics and public health during post-docs at the Ontario Cancer Institute, Dana-Farber Cancer Institute, and the Harvard School of Public Health, before joining the faculty at CUNY.Перейти в профиль
Welcome everyone and thanks for attending. So, you know, it's workshop on a cloud computing on the Sarah and both platforms. So you joined my lap, just a couple of years ago with a background in molecular. Biology is a wet lab biologist but at aiming to become a fireman from efficient and as you will see today it's unbelievable. The amount of progress that she's made enough time and she's in the enviable position of having a very strong biology background and also knowing how to communicate with biologists and
develop things in a way that is understandable to a broad audience. I just thought I would very, very briefly here. Give you an example of what I am talking about Wednesday in first joined the lab, she created this Trello board to to outline, for course of study and brothers who wants to come from a biology back. Brown and learn violin for Maddox, so, With that, I would like to hand off the Baton to say, you know, and I look very much forward to your presentation.
I probably have a lot of things still can learn from those things but I will probably start this year. I'm representing approval backwards on their n. H e g r. I a n before that and the content of this Workshop is contributed from the BIOS and then Brody's shoes. And then there are 48 from our team of piracy and Bill team and then brought him. So they will help you if you have any I think you can access their self print a semester and I put the link in the possible
chat room. So this Workshop is aiming to introduce the salt block for me as a potential solution to the challenges. If it's really boring size of a gentleman, such as sharing and Computing at cetera issues, and how the key and Bill and make it even easier. so, The first actually to, to play with the platform to platform, you need to register and have a top count. So, the registration needs a free and all you need is pretty much a Google account or a cetacean Institute, email account,
linked to our building building happening during the workshop. But if you don't have a right to create your child count, following gifts, Section. And then, And then leave your email linked to your account in the chat room, then place can add you to the billing account. So I will start the introduction. First when we went to the plan, B, Ridge to the light, dimmer section your account, should be ready to go. And the one other thing I want to do because we are fairly large group out to start menu One X from the same building on the next couple minutes.
So let's say 9:06 we will start light that time. So you I put how to instruction here so you can go to go to the workshop and then aren't there. Live demo demo workspace section to do you say that while we were So we will work space. We are using the better we are using for live demo is here, but once you get there and then you will go on this workspace. Probably you don't understand what they are. But once you open it, and then click the three. Menu in the upper
right corner, and you don't even drop down menu and then click the phone and then you will have a pop-out window and you will get you should give her unique name and then billing for Doctor Is FiOS in 2020, were Workshop Jupiter, let me show you. How is your wife? So good place and there's a three. And then you click a clone. And then you do get workspace name. and that it directs you to the workspace, which is your own And then what you need to do it from your phone, works a school notebook pack. And then manifest and then click the
playground mode McDonald's. And he said, I do either take a few minutes. This is what we will do. Just 102 minutes. Yes, there is no question. Who everything is a smooth or nothing is working? It looks like that. Someone was a little lost on how to access the work space. Could you just go over finding the workspace one more time? Art. I don't know. I'm not sure. So I can everyone access this material because I put that link on the beach at the presentation. Materials are the top hose in the chat so if you go there,
What's the workspace? Mixing a live demo. And then there is the links you go to. If you don't have access, Do you need a towel, and then click this one? Will leave you to the If you were in the list that were just getting added to, after we say, you were added, please refresh your browser. That will also help. Okay. so, images of how many people. What people wore in Prairieville? At 9. So I will. And then give you the couple example of how people actually can use this form and then we will go back to the park at
the Justice docket. My friend. Actually, there are many different cloud-based, genomics, platforms, available already and a host of different data or different analysis. Tool stores will be the factor major factor for you. To decide what platform will be work best for you and here is a very short list of them and Sarah and I put the link to this different platforms. What is Sarah mattera? You can have a secure remote access to public genomic data. And you can bring back after schools and pipelines over the implemented
to Tara directly and easily, apply them to your date of interest. And you can also do in track team analysis, using your familiar, and your face is which Monday you can and then get Amanda complication of capacity. Change a very easy and flexible later. Just like the and my wants at this the more direct answer to like what kind of analysis can you do and Tara, because usually that's going to be picked up. When you look for any track team analysis using the computer, So that's what Tara is like.
So I will go through more in detail before going through watch available in Terra at beta workflows and Note 4 leather introduction. On workspace, workspace is the main building block of Terra are different resources. If you're, if you're a veterinary office workbench, and depending on your back, or you can use the workspace for hosting data or building production pipeline or collecting analysis, template or oral there in one place. And this is a screen capture all the work space Heather. So they are
a 5c menus near. So dashboard is rare documentation and Brooks this information list and you can organize and access data in the fall under that attack. And purple Mitre box so that fast processing can be done on their workflows and work will taste its infer SATs off with based on Tara's day tomorrow. So we're blowing data at parties, very cool. And actually wire conductors and data and notebook connection and job, history page is where you can monitor
and then troubleshoot your work. Mission. So after you submit workflow and jockeys to receive the order the time and clothes and then go back and look information will be there. But this is the basic building block of Terra and so your products will reside leaving the workspace Union. And then, let me show you what kind of resources are available. How to explore tab resources, you can go to the hamburger menu on the top left and then you click the library and then he'll give you the drop-down menu containing data showcase and workflow.
No, I did not. I will give the horse go through first and then be able to play with the platform than later. So first sign that I've dated a showcase work. Let's go through today. First Post open and controlled access data and select data sets, have a built-in functionality or exploring the data. Keurig available data II era through various Awards, support ccbg, CMG, attacks, emerged, and growing still, growing list of all their data, set, and Steve supports hot mess. And there is a
nursing house study, the human populace and cold, I just want to make, I want to make a note that the register in for Terra itself, does not automatically Grant access to our available data, very simple. So, pretty much you link your account and I figure out what kind of data that you have access to let me go through the same to show you, but this is my workspace light. My. Auntie there and under my name, I mean profile as group. Building a salon and work spaces list of my work space. So we are looking at the library parsley, five-wood, beta. And these are the data
set. Right now, and let's check to see if I will show that they like school work with me. Simple overview of Ed 351 all stuff. That they said they did the same as a while and then female. And let's say that we still population like $61 and I can say thank you for this day. And once I have the court, I can start to be put into dissing work space so I can expect support is sought. The elected made us stayed up to the one of the workspace. I own Or I can start to beat the new work space Witcher gave the
same problem than you yourselves when you called earlier. So I can give a workspace name billing for Zack and then the work, this new work space will contain that the 61 participants data. So it makes it really easy to What is this data, next? I'm going to show you your users can reproduce Wagers and there any established mathologist. So you an interesting column contains a covid-19 related, work spaces and has which of the work or give you your a chance to understand their peers experimental
design. And the last day, for example, workspaces, here in the workspace and developed and then then make sure pick me so I can hear and burger menu and go to the library and then will showcase. So these are you can see there are the long list of work space, where you can find the things you need to guess year and that's just me and you so information and then because it's more of the attractive. So actually data and works with this can be the used Taste of like slacking, the
organist from Oregon resources at the same time, Jupyter notebook, that you can go through. Do the speed up the last resource? Tara is offering is a cold and Brooksville section. Section contains two fools and pasta made of familiar with Running Brook. Pool can use this as a repository to find workflow campanas de Pedro. Rey Orchestra gather using read, all the workflow description language Osprey. And look at the right sectionals under the find additional work for there is a box store and then browse map of the deposit and then the doctor and bill
for the next game for the lotto backwards on back door. So if it has a lot of well-documented workflows and pulls up there, this is, The end of my overview. So I just went through. What Tera is what is its basic building units, showcase and workflow. Next next movie theater. So I will give your two years cases of Juanita. Tara for paper, we should my own personal exemption used platform to teaching. 4 top or paper. I actually Linksys at your work space.
Here is my workspace. I feel that. So I built for my paper and I briefly mentioned during the workshop, on Tuesday, CNG Workshop, workspace is providing a flea reproduce and then I have a workspace Animals that are the major benefit of having tile works. Best with research paper is made in his story and put your impact Pipeline and bouncing analyses are available. In one place, we can prove there before this ability, I don't lie and the sharing cooled and providing additional information
not including the paper if you are available. And I want to. I put the major feature of this work space at the data and workflow and notebook level. And I will briefly show back and forth to share with public. And they it has the available through Terra and my own data files which is bad this year or so. And so let me go to the Future Part 3. So dashboard as I introduce O'Leary's, has information on the information about the paper and review and then how to prepare your
friends while and then what kind of notebook are available and then they beat us there. One time environment is sore, like to do with information on your dashboard and I just pulled your data. So are you up for now? And then Tara doesn't have these are the different data data format and these are the or different type of theater and then I told you the sum of the reference data is readily available. So if you click here, How far is the basic unit used to be 37 human between here and then Disney? It's the most popular of why do we
use reference data set? And these are over by Tara, so you don't actually need to pay for no reason. So if you see, they are actually stood in the walk. Back normal on GTA V. Additional work and down their workspace day. I could put different my own dataset. So for example, 8019 blacklisted, one of the info file for this works. It's work. And then I put that play made in the Terra Google call bucket of my own storage place. And then I think that to here,
The bastards data aspect of this work space and the word flu aspects of this case is ascribed to hear. Well, use in this paper and pink boxes are on each segment of the pipeline are built into each different workflows and based on their modularity. And that you put the number represent. The order is work for should run to hear. There are three words that start with one on Pandora on at the same time because they don't have any dependents. Is it? My funeral near
CMB work space? If I go work flow and then I have all this for food fare and how it looks like let's check new texts. And this is. Are Paige. And then if I go script, Give the word the different in fraud. How this for this is an order in food and how they are linked to tasks, are linked to each other and then extra test part. You have the raw Sprint so you can have or the detailed information available and then make very intuitive input provide information. So you can even just unload the Json file with information and then there is you
can link his wife told you. I will not be don't have enough time to cover the data model or workspace back to your file will you don't need toothpaste on different orders or like friend fill the creative each other so that you can put the You can't put off me, you can link Alfred Back to the Future original day tomorrow. So everything is moving together so I guess you will see the job history but it's there for now and the last part is a notebook. So, in this works for this work space, I have five jupyter notebook, using our and for our forward data, preprocessing,
and long are mommy's $4,000 and then end tables direct connection between data and notebook. And then we will run this is the only use case on Terra for paper and the Bible cover, Taryn the quest. so, I'll just Mention this fairly briefly to say that I've used Tara for teaching one PhD course, in an applied statistics and the approach that we use this more or less. What you're saying in this in this Workshop. Hear this first little bit here about how to share, basically tells you how to do, what we've done for this Workshop,
the creation of a group that allows you to, to share a workspace and share compute with the entire group at once. And also to, to take back I'm privileges such as billing or write access to a workshop, or to work space. After deadline sew in. The contacts. This is a fairly small class, but I think the methods with scale just fine to larger classes was also a class on biostatistics and not an environment for Maddox or specifically cloud computing. So my goal was really just to use it as,
as, as a platform to allow people to do a date analysis by our conductor without having to install software and without having to deal with a lot of different to different environments on everyone's computer. And I would say that it worked well for this purpose. Maybe I'll tell you that, you know, there it has some pluses and minus minuses on on the plus side. If it actually works very well for beginners, I think. And I used to be in jupyter, Notebook
feature, which I have found for beginners is actually has a much less of a learning curve than than our studio. You can have people who have never programmed in our before, just start using a jupyter notebook. And I'm not very long to, to figure out how to use it and and start using it. Of course it is not nearly as full feature does our studio is for four more advanced user has ended for developers but for doing an analysis and providing a a literate coding document, that is a document, containing writing and Code is
runnable. Increasing its output. It works well. The. So that's a plus the ease of creating groups and sharing workspaces is is also a plus maybe if you go down a little bit to billing, also, the ability to share billing is a plus as long as you have some budget for doing this, I found the that cost quite reasonable because because the the platform allows each student to select, which runtime environment stay they wants and most of it. And it says how much the cost is per hour. It also
puts them to sleep when they're not in use, so that all worked out fairly well on the negative side. It is difficult to predict the costs in advance, and if you had students using it a lot and selecting large instances, they could. It could run up a bill. I didn't have any problems like that and I didn't monitor it reasonably regularly. This I mentioned here, this Onyx billing account which I used also makes keeping track of billing a fair bit easier. And it also makes it more compatible with the billing
procedures I think it most universities in with Granny, if you can prepay, you can specify the amount that you want to pre-pay. And and then, just see how how your account is used up, which is a little bit easier to manage than the the regular post a system that you will use with Google Cloud platform videos on the negative side, the file transfer access his you seen already, it's a little bit different, it's a lot easier. Now, with the with the Anvil package that you've seen mentions and you can Look
into that a bit more, but it makes putting taking files from your local computer into the workspace, interior environments. And between those places much easier than it was before before that existed. I think that's all I was going to say about Tara in the classroom on, but I'd be willing to answer some more questions about about how that went in general. I'm now very keen on using Cloud platforms for teaching, courses, because it takes so much of the issues that you have with with students having difficulties
with their laptops losing data difficulty, installing packages. Sometimes it just takes away all of that and gives everyone the same environments, makes it easy to share data with them. We do have a bunch of questions rolling in. Did you want to take some time to answer some questions now? Or did you want to wait till this weekend? We can actually go through the crack and just before I go, if you want to learn more about, we even covered much of an envelope asking yourself. If and then I saw a couple questions and any more further
information I put them in here and then if you want to learn more about Tara's, Tara's how to start Tara, I put them into the Terrace user guide, and actually are now ready to jump to the like that Mopar picture of a pretty much available through the notebook, be started. And then, once it started, you can actually open the other Other notebook run time over the skin and just briefly the ejecting workspaces, I think. And then I need more about and their packages functionality. In the end we can play around and other than that I think
we can go to the park so I think we should. I use the word for collaboration aspect. I think the term is, these roads are the cloning a workspace, make you, the owner and allows you to experiment to be cold and data in your own copy of the works first base without running. The risk of a fact. Uber space at the original work space, but you can do anything you want, whatever you make will not affect the original copy, or we work together in the same state, if your share will be from here and then give
then you and then we did already and then sure do look like this. If you said you, everyone will be looking at looking at the same work space so that so then you can see Alders change it. And one thing I probably maybe you guys notice but I asked you to do use the playground mode but you can actually because in theory you cool in the workspace Eastbrook space so you can just edit and he's not going to stress others work. But I tried to just mentioned it. I asked you to play
Creek playground mode to any on the playground mode. Any changes, you're making your notebook, will not stay, and then will not affect. And then actually playground mode, allow many people accessing say notebook at the same time. So that's the difference and then you'll be useful concept. If you want to use Towers Facebook, laborate liberator. What? That's all nasty 9:1, all day and then I think you're going to take some questions I can I can pass them on to you. First one is a perfect question for you with respect
and you make your Anvil workflow robust against fire conductor packaged and packaged updates and data updates and how does this compare to? For example, having a companion GitHub repository for a publication that includes a Docker image of the workflow. I think Rob was the hero. If I take kids in the fridge, I will understand that I was like, whether I can update my work. Stayed and all their date update. I will say, it might be not that easy, cuz I need to manually maintain the thing, but other side is an order
and then I have an extended. It is fairly new work space. So I have them for too much work. On that people get that the states around it, but the for me, funny videos, people can rip 4 years. So, everything is like Frozen in That Snack. Shack in the snapshot at the time of publication to actually it increase the refrig ability a lot. So I think that Park is kind of A game of how you can make a workspace that always uses that same bioconductor release. I
could guarantee that's going to work and And the words with information is very easy, I think. I think that's one of the feature. One of the Elizabeth you can correct me. If I think that's one of the features they were considering that the places. I mean different work. Can have a version because I told us that is slowly moving but there is a clearly marked, they are clearly marked but yeah so works lulabelle I can do better than you and I think it sure is something there, there is a question that
actually if if Martin is here might be nice to ask about the Via current state and future of Candle package for local youths. And I know that there is a section in the vignette on local use of of the Anvil package on, but it's pretty short section. Miss Keisha, do you know more about Martin land? So, I can tell you that right now, most of the functionality built in the envelope package is for use on a art studio or jupyter notebook, which is running on the Android platform.
But there are very limited features which can be used outside of it right now and maybe it'll be expanded in the future to be used. outside of the Android, but we need to figure out a way to Better connect the Anvil with the outside world, more complicated than it should be, is just moving a file from the local computer into into the Note 4. And the end fact, I know that you can do that now to the work space with the NFL package. In my course, I just make put all made all day to either publicly or
publically available or actually, no, I just made them publicly available data to make it easy to This is Martin, I'll just mention that I've developed a notebook for an anvil Workshop, a couple of weeks ago. And at some point I realized that I talked to developing The Notebook on my local computer even though it was not meant to be running the anvil cloud and so is almost interchangeable with her eyes. I have access to all of that. And all functionality, the pocket associated, with my work space, the the
data tables moving files up and down. It was all a transparently accessible may be the only thing that son that route. There's some of that stuff features that are still kind of a work in progress, but I'm Princeton set. It's possible from your local computer node, your local computer to start an AR Studio instance. It's actually in the end. And then and then use the art studio in instance, through the browser like we're using in the in the workshops at the conference at the art studio, in Latvia in the 9:00, I think those are really pretty exciting
opportunities in no problems. It was pretty neat. And then, whenever I change is just like the utilities for an overdose in bed that into the coldest part of pain, but make it really are friendly. So I really, really like it. Next question was, can multiple people work on the same work space at the same time? And how is that? Managed and weld it? Because I am in my classes. I generally had everyone work in the same work space By by making a copy of the file that we're going to be working on and just end up in the same workspace. And yes, that works just fine.
As soon as one one person is editing, editing. A file is locked for everyone else, so two people cannot edit the same phylum, the same as everyone else can still access the playground mode. Next question, was do for work clothes and Tara, does it have to use Whittle? Can they use cwl snake? Make next lowest. Setting. OK. Google please correct me Ryan. Now the beetle is the real running and I think they're all right now it's a little more but I mean I want to use this power as like easy to store send Sania to think about the infrastructure building and maintaining
I want to do, I want to use. Don't have much motivation to use chroma in the local but not in Terra, at least for now. Okay, there was a question for me, the ballpark figure the cost for my class. And would it be a linearly with the number of students have big computer requirements for me. So I asked them to use an instance that cost $0.10 per hour so I would say and total for my class it ended up costing a couple dollars per student. It was really not very much and yes, it would seem nearly with the
next question, regarding the public data sets is a mainly human data or are the data for other organisms and maybe Elizabeth could answer that one best. And I it is there. I think most of the data sets are primarily human however a couple of them like the end code in the human cell Atlas also have Mouse data. Which parts of this art are in a question. I kind of have in my head. It's a really complicated to put in the words is the platform it is and then end here is the word x o x o
n g r. I so they are interested in. So it is from nhlbi has different interest in the different data set so they are supporting different date. I said but they are sharing the platform so it's kind of behind the behind the scene, you can access to everything but I have little hard time to push the word Tara and then for the animal side of it we do have there's an anvil branded theraplatform but it's still Tara and then you can any of Docker images. So your notebook run time, you can import Anvil, specific Docker images, that Angela has created. You can access
tools from tear that are Angeles Pacific and you can access data sets from tear that are related to cancel. What is Warsaw? The thing is, like something like it's like at the landing page for is like the end of Tara and then likes that the stated they are stages Tara are different. But behind the scene, everything is increasing profitability between the difference because you can actually access resources across the project. Is it correct resources from from the Anvil? Brand? Doterra that you could access in
all the additional resources? I should say that? Anyone else could access Frontera? That's all I can buy a conductor package and Bill is like a still at a dangerous, but can be used in my own mind. This is the main and he'll components are from. I mean, the new accessibility of the form or bioconductor users and and Galaxy users is really Anvil and the project and any Nest, not all of it, but from from our perspective that's the that's the big part of it.
The next question, this is an easy one. Can we start private in-house state of the other people can't see? Answer is yes. And the nice thing about it is that the sharing is very, you can control it easily. I mean, if he falters that no one else can see it. But then, it's all. So he, he say to share it with your lap group and other people who should have access to that particular access. Okay, next question. Something you have experience with seyoon suppose that I have earache almonds access can. I directly get access. Granted to all the available data. I will say yes I actually have
only one dataset access to let me go to my account. Go to the file and actually there is no not yet and I know you're right. And then once I do that, Figure out what kind of school. For example, the key text, a notarized and Target. I'm not authorizing pcj. I'm afraid to act like will be in and then it's like expire if every month or something every every month figuring out that I have access to, Can I sing to once? You have access through era, Commons, you just have access through care of like it just takes care of that. For you has any shoes, so they
can sign off. A lot of the questions have been answered her or answering in the chat and we can stick around for a couple extra minutes. But if anyone wanted to make it 10, I'm just letting you know that we are, at least three more question, sir. I didn't know how that was done. If it got into the chat, it's pretty safe. That's us. I think we can finish the question. Virgin control, I covered the Onyx billing account. I think, Onyx billing account, is great
configuration options with you recommend as well. I will mention. It was also a question and that was answered in the chat about if it was HIPAA compliant. Elizabeth in third. And she put a link in there to about the a comment on the hippest action and not, it's not, it's a fisma moderate, so that one was also answered in the chat. If anyone was interested in that, got the check parts. Can you say? But check history. I will see what kind of questions were
there. I guess and it will be available. Also, I think in a public forum section, I think every chat is visible there an extended. Of time and I'm glad that people could follow enough to ask questions and very happy and I hope you enjoy and my guys
Купить этот доклад
Купить это видео
ConferenceCast.tv — архив видеозаписей докладов и конференций.
С этим сервисом вы можете найти интересные лекции специально для вас!