What Can Gleep Do?

This is part of a transcript from an Applied Deep Learning Lecture at Columbia University. Students may have questions about some of the topics but don't have time to review the entire video lecture or scan slides again when looking for this info. So instead, they could ask Gleep! First, we give Gleep our text in the box below.

All right, so let's get started. So, here on the screens, what we'll talk about tonight, and the reason I wanted to start with effective use of unlabeled data is it's one of the most important trends in machine learning today. So as you all know, basically, the situation where we are, is that supervised learning works super well. The challenge, of course, is that getting labeled data is very expensive. And there's not a lot of it, and we always need more of it. And one of the best ways to get labeled data is to do it for free. And instead of labeling it manually, you can use something called self-supervised learning, where you can use a large amount of unlabeled data, and then you can infer labels for it. And, of course, by doing that, it means that you can potentially train models on huge datasets without incurring the costs of labeling them yourself. And this is the idea behind things in NLP like Bert, which we'll talk about tonight, which is a really big deal. And it's starting to be the idea or the driving idea behind models and computer vision, like SimCLR or Sinclair, I assume it's Simclair, but I actually don't know, I think it stands for simple, clear.

Anyway, here's what the midterm is gonna look like. So we'll do it as a courseworks quiz, it's going to be open notes, you can also use web search, you can use whatever you want, you just have to solve it yourself and, can't talk with other students about it, of course, you're gonna have three hours, it's two weeks from now, and I'm gonna, I'm gonna make it I'm gonna set it up. So it's available for you in a window. So we'll go available on December become available on December 3 during class, and you can take it any time in that three day three day period. And also anytime you want. I know folks are in different time zones, so you can take it whenever you want. But you can only take it once. That's how coursework works. And for folks with Disability Services, some students in the class, you know, if this applies to you, please send me an email. And instead of taking it on courseworks, which is timed, I'll send you like a Word doc or something that you can fill out and mail back as a PDF in the appropriate amount of time.

And so here are some of the trends that we've seen in this class, or so far, and in deep learning in general. And we've talked a lot about the last or a little we've talked a lot about the last one. So with medical imaging, and the key takeaways there is that in 2020 image classification is relatively straightforward. Especially when you have high quality labeled data. And all the challenges are no longer really in developing classifiers. But all the challenges are about how can we effectively apply them so they're on the data science side. And we've talked a lot about things like evaluation. And they're on the machine learning side, when we talked about a little bit about how you can interpret and explain what a machine learning model is doing. And then the second to the last trend, we've also talked a little bit about, and this is, you know, instead of training enormous models, training small models, I know I’m going in reverse order here, because it's late at night. Um, the reason I'm going in reverse order is the main trend that we haven't talked about is the first one. And we know, if you look at the second bullet for models, you know, larger models generally outperform smaller ones. And we've been able to train larger and deeper models, you know, by virtue of compute, and GPUs, and, you know, various tricks like batch norm, and different weight initialization strategies, residual connections, and all these tricks on the layer side that let you train increasingly large models, which capture better representations of the data. But the thing we're gonna talk about tonight is the first one. And this is how we can do supervised learning without having to manually label all these data sets ourselves. Because that’s, that's terrible. So the first one is the next trend in deep learning. We'll look at Bert briefly from 2018. And we'll look at simple clear SimCLR from 2020.

And so there's been a lot of talk, you know, on tech Twitter, from folks like, you know, Ian Goodfellow, and Yann Lecun and all that on semi-supervised learning. And I just wanted to start because these, these terms are used a little bit loosely. But I just wanted to start by taking a look at roughly what we mean by self-supervised learning and what we mean by semi supervised learning. And does anyone happen to have seen this term before? So what does it mean to do self-supervised learning? Yeah, one form of self-supervised learning is like a feedback loop, where you could use the output of your own model in training the next iteration of that model. So basically, another way of saying that is, you can infer the label for the data. And so in self-supervised learning, learning one way to look at it, what we're trying to do, um, I should really change this slide. It's not converted unsupervised learning problem. But it's really converting an unlabeled data set into a supervised learning problem. And so instead of paying for labels for the data, we have some sort of technique where we can guess the label for the data. And an example of this is something like Word2vec, where the way that you train a language model, I'm just going to draw on the slides. I know this is terrible. But one example of this would be, you know, I went to the bank, if you have a sentence, I went to the bank, you can train, you can delete a word. And you can train a model with a, you know, a softmax classifier to predict the missing word. And because you know, the sentence, of course, you can get for free from the web, you can download Wikipedia and use all the sentences from there. And you know, the label because you deleted the word, so the label is done. And so this is an example of how we take an unlabeled data set Wikipedia, and we convert it into a supervised learning problem. And now we have, effectively an infinite amount of data to train a classifier, and Word2vec. We don't actually care about the ability to predict the missing word. But usually what we care about is the embedding, or the representation learned by the model. So the whole point of training Word2vec is, you know, a word embedding model is to use the embeddings. And so we often throw away the classifier that does this. This is just this a procedure we use to train an embedding layer, which we keep, and the same theme you'll see with, with Bert and SimCLR. And another way, well, yeah, so anyway, um, so that's an example of how you would create a labelled data set to train a model for word embeddings. Another example of this would be if you wanted to learn sentence level information. So maybe you want your model to have… So remember word embeddings. They're not contextual. So they apply to a single word at a time, regardless of the surrounding words. And one thing you might want to do is, let's say you wanted to train a, you know, a deep NLP model that has some knowledge of sentences. Another thing you can do by downloading Wikipedia, you can create a different type of labeled data set on sentence level. I went to the bank and I took out a loan. So here we're looking at two sentences at once. And what you can do is you could train a model to predict, does the second sentence, follow with the first sentence. And in this way, you can train a layer that knows something about words in sentences and how they relate to each other, as opposed to just words in isolation. And this is one of the tricks that's going to be used by Bert. And we'll start on Bert tonight and finish it next class.

Um, okay, and there's some, there's some slides walking through what I already what I just sort of talked about. But let's see if there's anything I can add here. Yeah, just in terms of terminology. So you might, you might see the word mask, here and there. And the idea is that the mask is showing you which word is going to be missing that the network will need to predict. And then there's so basically, and I'm using this loosely, but by self-supervised learning. You know, this is sometimes a term from robotics, but in, at least in the NLP sense. And in the computer vision sense, I'm thinking about mostly, it's a trick where you start with a large unlabeled data set. And you turn that data set into a supervised learning problem by doing something like this. And this is enormously impactful, smart ways to do this are a big deal there. They're just as important if not more so than fancy new layers. And then just because this term is often confused with semi-supervised learning, I just wanted to mention what this is to as anyone's seen semi-supervised.

So what is semi-supervised learning. In a semi-supervised learning setting, you typically have a very large amount of unlabeled data, and a small number of labeled points. And the idea is to use both the labeled and the unlabeled data together to train a better model than you would on its own with either one. And the classic, at least for me, I know there's other examples too. But the one that makes the most sense to me is actually this label propagation example from scikitlearn.

You can read through this if you want. But let me explain what this diagram looks like. So here on the left, this is a before and after diagram. On the left is before it on the right is after. And the idea here is on the left, you're given a bunch of unlabeled data. And that's all of the orange points. They're all unlabeled. And here, I know, it's very hard to see. But you're given only two labeled examples, this teal point, and this dark blue point. So you have to label two labeled data points and a whole bunch a couple 100 of unlabeled points. And the idea is how can you label the rest of the points with just these two. The way that's done label propagation is you find the nearest neighbors of each point. So here's the teal point, and you say that its nearest neighbor, neighbors are these two orange points, and you propagate the label out. And then you do the same thing for the next point propagate the label out, you propagate out, you propagate out. And after you finish propagating, you'll see the label the inner circle, teal, and the outer circle, dark blue. And the idea here is that the data is on a manifold. But anyway, it's the point here is that you're exploiting the structure of the data to train a classifier with very small amount of data, and the unlabeled data that helps you do that effectively.

So that's what it means to do semi-supervised learning. And then here's some things that we've talked about briefly, but there's lots of good reasons. And of course, we've all known this, you know, in machine learning for, you know, since the very beginning, but for a very, very long time. The emphasis has been on, you know, what can we do with unlabeled data? Because, and how can we use this to train better models. And there's been examples like Word2vec. But in the last couple years, it's been, it's really been a smash hit. And that's with things like Bert and, and we're seeing the very start of that in computer vision. And both of these models are going to be used in in a way that you've already seen before.

So um, basically, when we have a convolutional neural network, the thing that we care about is getting a good representation of the data. So, you know, we've talked about this a ton, like we want to go from edges to textures, and so on and so forth in the different layers. And we want a good representation of the data. And then once you have that, you're basically training a fancy feature extractor. You usually add a dense layer on top and the dense layers, what classes buys the data. So it's fancy features that you get for free. And that's what deep learning is doing, followed by multi-class logistic regression to actually classify the data. That's what we're doing with computer vision. with things like Word2vec, it's a similar idea. But traditionally, what we're doing is we're getting word embeddings. And then the idea is that those word embeddings, are trained on a very large data set in the same way that, you know, good representations for an image are trained on large data set, and then you add a classifier on top to customize it for the task that you care about. And that's transfer learning.

Now that Gleep has our text, we can ask a series of questions. First let's ask:
"How do you predict missing words?"

Answers:

What Can Gleep Do?

Now that Gleep has our text, we can ask a series of questions. First let's ask: "How do you predict missing words?"

Thanks Gleep! We can ask as many questions as we like, and select different text by pasting or uploading a new file, try it for yourself!

Now that Gleep has our text, we can ask a series of questions. First let's ask:
"How do you predict missing words?"