Talk with the BioC Core Team
Lori Shepherd (Roswell Park Comprehensive Cancer Center / Bioconductor) Senior Programmer
Hervé Pagès (Fred Hutchinson Cancer Research Center) Bioconductor core developer
12:00 PM - 12:55 PM EDT on Friday, 31 July
Join the BioC Core Team for an open discussion on build system, single package builder tracker for new submissions, bioconductor infrastructure. Bring your questions!
I originally started my education in the hard sciences (biology, chemistry) but decided I needed more computational skills to be competitive in the working industry. I switched degrees and got a Bachelor's in Bioinformatics with minors in computer science and computer theory. I had an internship at the Southern California Earthquake Center for applied computer science. I completed my undergraduate degree and immediately started working for the Biostatisics department at University of Buffalo and Roswell Park Comprehensive Cancer Center where I have worked for the last 12 years. During my time, I went back part time for my Master's in Bioinformatics, a joint degree from the university and cancer institute. I now work for the subdivision of the Biostatisics department, Bioconductor. I program in R and help manage project infrastructure using python and ruby. I also assist and present with numerous short courses, workshops, and conferences.Перейти в профиль
I'm a mathematician and computer scientist, working full time as a Bioconductor core developer at the Fred Hutchinson Cancer Research Center in Seattle. Bioconductor is used by many researchers at the Hutch and by tens of thousands of bioinformaticians around the world, enabling them to perform a wide range of analyses and helping them obtain reproducible results. Since I joined the project in 2005, I’ve seen it grow from < 150 packages to almost 2000 packages, and embrace many new research fields. My role as a core developer is to support this growth by designing/consolidating core data structures, and to help maintain the quality and cohesion of the entire ecosystem via the daily builds. It’s really exciting, motivating and gratifying to see that the software is used by so many researchers at the Hutch and around the world.Перейти в профиль
So welcome to the the chat with the 14, I guess, just for good measure. I guess. Let's have the 14 go around and introduce to say who's on the calls. So I'm Laurie Shepherd. I'm in Buffalo, New York with Martin and they do a bunch of various different parts of the project, mostly the single package filter for new submissions and the hubs but also have had a hand in the Amazon machine instances and help Erica with the build system as well as a bunch of other random
projects. Scaling bioconductor and its Solutions. On to the cloud. I work on the doctor images, the animal projects and infrastructure for bioconductor. The gate ecosystem. And yeah, I also, right? You are packages which help me go in this direction. Part of the core team for about 2 years. Now, I'm based in Buffalo New York with Martin's group. And I, my first project was the BIOS reset package, which is typical representation of Gene sets using some of the deep fryer functionality. But I've kind of branched into being the shadow of a bunch of things that
Bassnectar works with such as the single package Builder. Experiment of annotation Hub. I was responsible for getting The annotation packages together for the past couple releases and various things with the website and things like that. Sorry, for the background noise. You can hear me fine. I'm part of the coaching. I work for multi no agreement and not in Goodfellow & Co. I just hatch. A bean is a project for 14 years, 15 years, maybe now and doing various things descendants in the beach system for the Nike.
Beats the streets to generate a package. Download statistics in or taking care of values. Cool packages like by strings, oranges that were Victors in Mequon juice and more recently delayed or A & H, e a v o r e. And I have been in the Bible doctor group for about a thing close to four years as a 3 last time that I'm thinking about it. I think it's for I work on mainly almost as a experiment, but I also do a bunch of other packages like your attitude CJ data, see about portal data, and I work on Bayou, City Manager,
as well, and other packages. So, a lot of things were common building books on the BBS. I don't know if we're supposed that wanted to expand on that any at all. That was from Aiden. Is there a plan to have a formal support the building during this these books in the Billing System? I know, I know what's the weather in Natick to get this Oscar booked up and running. Posting it by Robyn using GitHub Pages. GitHub pages, is a bit of a pain, actually, Let me go. So, yeah, yeah. We we try we try to pay Aaron. We try to sneak in your, your
books, the reading of your books with the wood floor and we can try to come up with something better for the books and yeah, I'm open to welcome. I thought you were maybe going to go for. some other solution we working, we please on some of the solution using Are you willing to use my laptop? Okay. My experience is, it has not be too hot, you modify a little bit, the assistant to make it work with a different kind of be like the long test. Be worthless Beast. H Street has
its own requirement, but the base code is the same and Dallas shootings that needs to be mowed. If I may be so for the books we could try to come up with something that works. Well. Thank you. Do steel BBs question. Can we get something like, the wind builder for the VBS? These Windows errors are frequently. Killing me from the soundtrack. To the question is sum-sum on demand feeds right now. The GPS is for the Daily Breeze, right? Yeah. There's there's also the similar question here is same
as when Builder question above, but with build on demand for all platforms. So I'll say the recent, I don't know how many of you to submit new packages, but we did do an update for the single package Builder where we started adding packages to get that bioconductor, auraglow Tatian on submit to also help people get used to pushing to get. Buy a conductor and help people understand remotes up front And because of that, we made a lot of changes in the back end, to be able
to get the repositories from to get that buy a conductor. Org location and the plus side of that. Is, we set it up where hopefully, eventually, we can open it up at least for the devel. Branch is to build onto man's like the single package filter does with the valley Persian bump. It's still a work in progress and we still have some like safety nets that we have to put in place as far as building and cleaning up reports because obviously that extends a lot of load for the single package filter
that we want to make sure that it's more stable and that there's some cleanup methods and place cuz we have to store all these build rapport somewhere then. But there is sort of a work-in-progress to try to get that available to people. We don't really have a good I'm up for it. I'm, I'm hopeful that we can move towards that within the next release or two. So, hopefully it's worth coming. And I guess I'll take this one too. Since that I'm I'm already talkin in unmuted Hub. Connectivity is pretty sporadic from China. May be due to problems. Connecting to S3
through the firewall. Are there any thoughts of having a Chinese mirror? That's a great point. I don't think we've thought of that, I don't think because we've we've developed it here and I don't think we've really thought about the issues with with us three to other parts of the world. But that is an excellent points. And I certainly could look into that and work with a w s to try to figure out a solution because we definitely want it accessible to you. She's everywhere. Otherwise it defeats the purpose. So I will definitely follow up on
that after the conference because it is a good idea to do this, one's addressed specifically for you, what are the current best practices for paralyzation with hdf5 array, for example, reading things in a BP PPL, apply Loop, he was busted on the on the slack. I think it's a good question. That was asked is Audrina pizza shop, and I am sitting on the dock and the answer is that. Yeah, you can use detail apply. That means you didn't do your own homemade cooking for reading these important but But you're going to still use
a block apply which is a looping function that is provided in the letter. A that is specifically designed to make it easy to to work on a blocks off of nari, right? Of the deal a direct object and any can working in a paralyzed person so you can Define the position back in, you can set up or is it back in and do the block procedure in in parallel? Yeah, it's something to keep in mind is that they are all Reggie. Many operations that undo block persisting behind the scene. When you do something like
when you call the Matrix summarization function, like Romans or a robot when you do a shin and they are all the things in my slack answer, I give a few other examples of that those operations already do some kind of block apply behind the scene and if you have set for ization back and they will operate in parallel One thing to keep in mind about positions that and I think it made made the point about that. He's that never give you as much peppermint boost as as you might call, it
might give you a little Improvement but used to be mine. We was delayed real checks nexu5 already know where the bottle neck is. Sore is very noisy here Okay. The bottleneck is the aisle. So when you paralyzed things, when you're paralyzed, something like reading the blocks of of of object to work on Silver blocks at a time, you are so You will be in a situation where reading the blocks is actually reading far is blocked at the same time. He's going to compete for 4 that I might does. The reading of the individual
blocks might actually slowed down in the other or performance might be a little better in the end. Now, that much Chief, if you are, if you are limited by the, by the I do have the actions for package Builder. Do you want me to answer that? So it's complicated, right? So bioconductor can't actually leverage. Get her ability to get her boxes instead. Actually, because we have these tar balls, which are built. And these are the powerball's which we need to use as package dependencies for the package, which is being built again every
night. So it's not directly possible at least right now to use GitHub actions for package building. Does that make sense or actually? There is a CI auction which is given by. I think our studio actions are like biosi actions or some repository like that. And you can say you was this action in, like, they use this package, but it gives you a A build a package Builder on GitHub actions, which is very limited. It's not, you cannot mimic the bill system directly.
That's the best I can do. And I will point out just from shadowing. Are they like the last year or so on the build system the intricacies and add cases and details that have gone into building the build system. I feel like the general public and even the core team doesn't fully appreciate until you actually try to look in the code and help manage it. That they're there is a lot built into it that it well, it seems straightforward. It really is. There was just some discussion about this in the public
data resources workshop at Shawn and Levi held and at least. To begin with. We were, you were thinking about supporting somebody conductor, GitHub actions for checking your package, and things of that nature. But replicating the bill to some of the tasks. I don't think that's going to come anytime soon and it's on the table. So we probably would have to rely on your hug too heavily and that might be an issue with the design of the website to be cooler with D3 and everything, something like
that, slick. But with bios key stickers, I will say, no, it's a frequent requests to redo the website and that it is. It does certainly need a refresh. Again, I think there's a lot to it, that goes into it and the main limitation is that there's no real web design expertise currently on the 14th. And I I know from sitting on the community Advisory board that there's been some discussion tool. Into funding to try to find someone or to think about the redesign of the website in coordination with the
and their pages and pages of it. sorry, it's not this redesigning it but it's also Organizing all the helpful content in it, it'll take some time whenever it starts happening even then I anticipated like taking a year or so at least with a full-time developer just doing the website. We understand that it's kind of difficult right now to find things that are there. So we do recognize it. One thing I've been trying to push has to redo the landing pages at least with package down, but we we have a system that is really complicated for
building and could use some help in doing that. If that is a priority, an issue when building a package where to look for help on the question could be very general such as errors, when building a PDF file I guess, as of the Diocese of el mailing list, its purpose is to ask for help When developing packages in about package implementation. If you're already in the submission process, we do encourage you to write on those GitHub issues that you open when you submit a package with your reviewer. At along with that. When we
do package reviews, we do we talked about it in the other meeting how every week, we have a longer team meeting where we review, you open issues on the support side and on the mailing list and we also review package reviews and how many of us have outstanding package reviews do. So that we do, try to keep each other on track and we'll go through and see if anyone has posted questions that need help pushing it through the review process. But while developing the support site and the mailing lists are great options.
Try not to cross list though. Please put it either in the support site or in the mailing list so that there is one consistent thread and we don't get fractured conversations. I don't know if anyone else has anything else to say about that. Run in English or in Japanese group that helps with building packages, yet another resources to organize a meet up. If it's more about General our package building, maybe in a meet-up, would be something that would help you out of the
package Japanese, but they do do Google meat that has the closed captioning option. So I think it does you do have an option to have it translated into English, it just might be a little sewing broken but I never use their fans leaving so I don't know how well it comes across. But it made you use Google meet for that purpose. This support site beta has been in beta for a long time. Is it going to be released soon? Yes. So I'm in, I'm in charge of that and working very closely with that along
with Kayla. We are waiting for the BIOS. Styles to roll out there's cuz they've been basically are full on beta test of the the testing beta site. They also took a month-long Hiatus this past month and plan to roll it out when they get back. So I anticipate it with no other, hopefully, no other delays. Hopefully by the end of August early, September at the latest, but I think we're shooting for for that right now. So hopefully not too much longer Marcel. Can you say a bit more? What the specific challenges are with integrating package? Down
Orchid for. I think it's yeah. Page down or packaged down. So so the our landing pages, take BBs products and Lori. Norris knows more about how the intricacies of how this works, but essentially it takes BBS products and Jen. The landing page. So the way we generate those pages using packaged down would have to integrate with that question. I guess it's still there. While you guys all think, I know when that we get a lot on the the help page in the mailing
list is I put an update to get but I don't see my updates on the landing page or I don't see a new builds Rapport, and I guess I'll just briefly mentioned timings and how that's all interconnected right now. So firstly, if you push an update to the get that bile connector.org. It has to include a valid version bump and if it didn't include a valid version bumps and it's not going to propagate an adult summer to the landing page. So make sure that when you push that you do to a valid Persian bump, And then we do do nightly builds and so
the daily Builder until we get the on-demand up and running is, is once a day and are they can correct me on the times. I think we start get Pauline all of the repositories to the Builder around 4:45 Eastern Standard Time. And so if you've made a commitment anytime before that for the previous day that it would be included in that build rapport and if it's after 4:45 Eastern Standard Time p.m. I guess I got used to. So what 1645 And if it's after that time frame then it would go on the following
day's build rapport and the build rapport normally takes of a day. It gets posted the following day roughly right now, around 1 p.m. Eastern Standard Time, if and there's a date at the top to indicate if it's the new days built, so it does take that time. So that right there could be anywhere between a 12 and almost a 24-hour waiting period for that. And then the landing pages will get updated once the full build rapport is generated. So then I could take another hour or so after that to see those changes
reflected on the actual landing pages for the package. So just kind of comment there cuz I know that's a frequent question that we get on the mailing list. anyone else on the team, on a jackal, some frequently asked, questions or address anything, considering organizing our or recognizing the need that the learning content on nor Underbar connector website needs some topics to include and easy organization so that you can find things easily. I guess I'll Trust my notes that I know sometimes can be hard. Eric, can you maybe
briefly touch upon like what sort of environment variables are? I know one of the questions we get is I am running commands build and check locally, but I'm unable to reproduce reproduce the errors that I see that are recognized by a CD, check to buy a little bit. The behavior of, I'll send a check to make it a little bit more Street in in some areas. So I don't have it in my head right now. But if you look at the report for your package at the top of the page
of that report, there is a link to the document. At least. Those variables is actually a fire that you can do it be used on your own machine. Environ Fido, use the same environment variables locally, you should be able to reproduce the around, you see, on the reports of course, you need to make sure that You are using the right decision of our and that your packages are up-to-date and do that. Chances are high that you did a tremendous, what you see on the report
and down here and develop, we have the build rapport and we have our beliefs and our design build reports and we can go to the report here and then these have the report for specific packages and what are they was talking about. If we click on page, you'll see this link for, please make sure to look the following settings. And so that has the information here about what were using and a little bit of how to use it. So that was what he was referring to how much feedback
is collected from the community before, building certain solutions, for example, the website Spend on a to-do list for two to three years. Now, to get browsing tool as well. I mean, again we do appreciate all the community feedback and we do encourage the community to give us feedback. And it's not that we're not listening, it's more of as it gets either six of us for a very large project with a large amount of tasks so we want to get them done and it has been on the to-do list but again it's just more so limited expertise and limited resources.
It feel as the biggest thing cuz we do talk about them before. At the start of every released at the end of every release, the core team does sit down too kind of map out where we are with the project and where we want to go and set some goals for the next release. And those are always on there and it's just, it's, it's hard without the skill set to do it. Do you think this is a good moment to get into possible hires and stuff? Which we had on the Google Doc as well?
Share screen photo? Let me see if I can pull up. The link. Because we are always hiring. So if you know people or you know, people that would be interested, especially ones that could help us out with that. There is a job posting right now. For Roswell Park called a scientific software developer right now. We are looking And we encourage you to apply. Maybe I'll stick this fixed. I think there's a job section. Is there a job Channel and Community slack? If there's not a job channel in the community slack, I'll
stick it in the general Channel link to this job posting. But we, we would encourage anyone that knows of anyone to apply certain truck. There's also jobs tab in the platform. Perfect also good. Maybe I'll make sure it's there. And if that doesn't seem completely applicable, still apply or Noah said to reach out to, to Martin. If Mercedes that the qualifications aren't quite matching up, but you still think that you would be large, can it's a strong candidate? Then we encourage you to reach out to us. So can we run
routinely run? Asin, you peace and excetera on the VBS maybe once a week because that would be a question for you. Everything I say and you see Stan excetera on the PBS maybe once a week. Maybe, if whoever asks us wanted to expand on that, I don't know if I'm so sorry I meant the address sanitizer from clang Behavior. Sanitizer on our packages that are that had to fix was a rather embarrassing out of bounds access, right? And it just seems like to be the nice information for those of us who
The information provided for us. How much time should I go to the, how much load in the big machines, right? And I don't know if you would be able to do this on all platforms on only knows where it's always easier to run more things. Yeah, we also have them at least two, maybe even near future to, to run a check on on packages. So it's a sign and you be sad if I didn't actually know about them before until I looked at Kranz Kranz checks. Bunch of checks with me but I don't really care about like you can bet it was a different Matrix body but
it is something that it's easy for you to run locally on your machine. I haven't tried running those two. I usually try running an errand and the phone with valgrind is that it just takes so long to increase by tenfold and I don't have any but when I look at some of their reports is that all you know something stupid the goalies was suggested to run. Green on the big machines for the Daily Breeze and that's something that we we said we we're not going to do. Because the cost of rain. Grand on an old package was just
a crazy. That's not something we're going to be what you do. But I can look at those and see if you can get it to integrate that to the riches that Mia could be a good thing, you know, in a few days. Can you talk more about how projects are chosen by the core team? How does byassee decide what to work on day-in-day-out someone else want to field this one. All right, so I think I talked about it a little bit. Maybe it was yesterday's lunch to how we kind of structure.
Like we have our daily meetings, where we check in about how, our progress is going on our projects. But we set ourselves up with these two weeks rent, where we sit down at the beginning of the week and we say, this is what we'd like to accomplish. He's got a sticky notes on, a Trello board, kind of to lay out. And then at the end of the two weeks, hopefully those cards have kind of moved the dumb column and we can give like a live demo of what we've been working on and I mean in a perfect world. All those cards would be done in 2 weeks but sometimes some of them carry over if it's a
really intensive development or if you're developing a package or something like that, sometimes two weeks isn't realistic. We at least want to see some progression and some some of the tasks done so that kind of Keeps Us constantly working and focus on breast far as how we decide what projects we work on. At the end of the releases, we sit down and we have like a 6-month check in to be do the released every six months. And we say, this is what we wanted to get done the last release. Did we accomplish that? What
we want to get done, this upcoming release and I think a lot of that, if you've been keeping track of the other boards and such threw out by a doctor, I think some of that kind of comes from the technical Advisory Board because they kind of decide what do we need to get done. Technically, what areas do we need to improve on or could be further work on? And I think that kind of trickles down to what can the core team accomplished. So that kind of is Martin's kind of the guy that relays what happened at these technical advice. Board meetings and what may be
the community and other users would like to see done in by a conductor. And then we kind of tackle those based on I think certain people have their areas of expertise than their nicias. So we kind of just tackle those ends and realize are, are they going to be accomplished in six months District carry over from the last release and again, in a perfect world? We could get those tests done in six months time. But sometimes more if not more important, but other things come up, like, what happens if something happens on the field system, will battle call Lori
and are they away from their other projects for a little bit because that's something that needs to be addressed right away or if something happens to the website or the support site and you know like people are potentially happening in or something like that. Like obviously, there's things that pop up that we need to address right away. Which kind of takes us away from our projects but there's things that we have to do on a daily basis. Obviously monitor, the bill reports monitor the website to support site package reviews. We are really striving for
like, 2 weeks Windows, where we, we give back reviews, but we also expect that from also, our review. He's not something that we have to constantly keep in the back of my mind to is that we need to be doing these package of you hopping on various conference calls whether we're working on the Anvil project or for hopping on the developers Forum, or for developing our own packages. So we wear many hats, which can be exciting because you're involved in a lot of things but not speak very stressful when it comes time. For these two weeks, all reports are six months reports or release time
because you know, you you started it really wanting to accomplish a lot of things, but there's just so many things that might pop up that pull you away. I think we all really strive to just that we try to stay on on Focus as much as possible and I think our Trello boards and our two weeks Prince help that but sometimes it's inevitable on some of these things. Like like everyone keeps messaging mentioning the website. Trust me we all wish. You know, we could get that done and it's been on our board for a while. But you know, things happened. So we appreciate the
patience and the willingness to work with us. And I think if you have projects in mind, I think you can bring them up to either of us on the core team or to any of the boards. And I think they kind of find their way to the proper people, but that's kind of a broad overview of our day in and day out activities, and some of our responsibilities, and how they all work together. Where to we do do these longer meetings once a week where we do do a clean of the support site, the devel mailing list and any open, get Hub issues that were reported and we do keep track of them.
We don't always get to them again in a, in a timely manner in a perfect world when we always need more people. So keep that job posting in mind if you know of anyone or just more time in the day, I think it was. Another issue is just, you know, if there were twenty-five hours in the day we could get more John. But we also sometimes have to sometimes, but we are aware of any open issues that are reported. And obviously, if something is is a larger bug, will will try to get that to get to that right away. If it's a, if it's a feature request, it might be pushed back a little
bit. But we do keep track of everything that's opens and do review. Them seems to have one more question. Robust is the testing procedures, in packages developed by the bioconductor core team. Do you use code coverage as a measure while writing test cases? What are some of the best practices used by the core team for our package development? So we are no difference. We still go through the submission process for any of the packages this week. We submit we cannot review our own
path even though when they first get submitted, sometimes that happens where it gets assigned to us. But then Martin goes, Andrea signs it to somebody else. And if we act, as if we're a typical developer and we have to go through the review process, our package has to pass all the bills in the checks, I mean, one thing that might be difficult as it is all supposed to be documented there. But sometimes, you know, we might just yell across each other. Be like in the trash, can you check my package really quick or something like that? So we try to keep it as documented as possible on the
issue tracker because, you know, we we are trying to be as, as a typical developer. So you can see all of the commenting back and forth. Like, when I submitted my package Marsalis, my reviewer and he gave me a lot of positive criticism. Things that we could change things that you can't. And that's that's a typical reprocessed there. Should be back and forth between you and your view as far as like, the testing process is the same as we go through the same issue, tracker through the single package Felder code coverage. We, we have unit test
I don't think we stressed code coverage as much. I think I'm just as with any other package, we want to see unit test and we want to see as many functions being tested as possible. I think I mentioned in in my presentation, Yesterday by a seat that has like hundreds of pests that run very quickly. But when I checked code coverage after the fact, I think I was only up to like 78% and I've been developing since so. I know my test could improve. And that's that's an area that I'd like to to improve. But I don't think we
really hone in on a specific percentage of coverage and then best practices for the core team uses. For our package development, I went through on our website under the developers. Chad, we have a package. The mission at a package guideline to documents the submission. Kind of blocks you through the typical process of a submission as far as what are some extra How does the review process go? What are the different packages wee-wee review and accept? And then kind of after-the-fact, some some points of there and then the package guidelines, really, really lays out what
do we look for in a general package structure? What do you look for in the description file? The name space style, the our files and yes the man cave unit, test the r code. So we do look at a lot of different things and I mentioned in my workshop when I sit down to do a package of you, I kind of go through the package guidelines and see that. It's not a strict thing, it won't make or break your package but that's kind of a good Baseline. As far as what I look for when I'm repeating a package, that would say, take a look at that documentation. And that's exactly what we go through with our
packages as well, and they will point out to revise, you check, if you think that introducing code coverage or some linkage + 4N, He sort of future request to love. We're not opposed to pull request. It makes it easier for us to review something rather than just requesting us to do it for you. So we would really like, for request, I think in general like code coverage. It's essentially, it's a challenge against how much of the cord you written, is being
touched by your test cases and sometimes, as developers rewrite, the code in one way and I'll test cases are biased because they're both in the cold already. So, I think there is a way to, like, improve test cases as more people use the package, but but again that's a deeply philosophical thought process. We have one more question. I think, my I want the suspect Aaron is asking this. Who is the best developer on the Bayou? Secor I can say for sure it's either Martin or Irvin but I don't want to jump to any conclusions. I think
everyone's pretty good to see, everyone has their strengths and weaknesses but watching are very Barton code and what they come up with is sometimes pretty spectacular. Spectacular. Even from our point of view, I know I sat down with Martin even when it comes to a couple different languages, see Python and watching him code and and are available will come up with Solutions. And then are they comes up with this really eloquent? You know, one, two, three lines Solution. That's Completely efficient. And I just feel like a bad
programmer and then, you know, finally asked for help from Martin or someone on the team and it's just like, wow. Like honestly that was what was dumping me so. But again, I think it's, like Lori said, we all have strengths and weaknesses and their Specialties in our niches, I think we're learning as a group, how to use each, other's strengths and weaknesses of our abilities and collaborative. I can spell with all of the integrative work and attached with
GitHub in there, a resident, get Hub in or get in doctor, X person. And we all know. It's important to find out about the, the depth of the thought process for which Martin, and I were put into when they lighting even a single line of code right leg. That is an immense amount of thought, which goes into writing, each line of cold and it is it's not something we. At least I do. I just write code like I'm writing a piece of paper. May be bred like when they lighted it's it's got like a couple of paragraphs of God which you can spell out very deep
so we can sign up to make sure you guys can get to the one but thank you all for joining us. I know that was some great discussion and please feel free to reach out with anything else on the slack channels are on the mailing lists and we look forward to hearing from you. Thanks everyone.
Купить этот доклад
Купить это видео
ConferenceCast.tv — архив видеозаписей докладов и конференций.
С этим сервисом вы можете найти интересные лекции специально для вас!