In this video, you'll learn about data augmentation, its methods and its uses, and how GANs can play a part. First you'll learn about what data augmentation is, what it's used for, and then how to implement it in the context of GANs. Data augmentation is typically used to supplement data when real data is either too expensive to acquire more of or too rare, when you don't have enough of it. Let's say you only have some few data samples, these green ones, and then maybe you can supplement it with these seven orange additional data points here. This is just an icon, so let's keep going. GANs are well suited for this task because you can actually generate fake data to supplement that real data and then use that for, let's say, a downstream tasks such as a classifier or a detector, or a segmentation model, or any type of discriminative model actually. Actually one common way of doing data augmentation is not using GANs necessarily. It's actually just taking an input image. Let's say you have this one image in your real dataset. Your task right now is to detect whether there is a dog and the image using a classifier. You're turning your classifier with this image, but maybe your classifier doesn't have enough data. What you can do with this image is, you can do all sorts of augmentations on this image and then feed all of these images in to the classifier as real data as well. That could be horizontal flip that you see here. It could be some kind of rotation. It could be zoom and crop or it could be applying some kind of filter to it like that. Now how a GAN could help with data augmentation, is that it can generate a ton of different images. Here you can see that it's generating all sorts of different dogs, and its diversity far exceeds what is seen here, at least in the breed of dog, and even in the lighting schemes and everything in the background, even in the orientation in some ways as well. Of course it's not as rotated as that dogs, so perhaps these methods could be overlaid with each other. You can in fact use all of these methods, traditional data augmentation techniques such as crop, flip, rotate, and using again to augment your data as well. Definitely you can mix all of these data augmentation techniques. A fun fact is that there is a cool body of literature that explores how to best combine images from all different types of data augmentation techniques. That body of literature, tries to find some kind of policy in determining which image you need to give at what point to your classifier. If you're interested in that, I would start exploring something called RandAugment. What can this be used for? Specifically the GAN generated data, how can this be useful in data augmentation? Well, one interesting use case, is that in social sciences, people often use gaze detection to determine where someone is looking and focusing in their studies. It's really hard to collect additional samples on which direction an eye might be looking. Typically what these researchers do is they generate synthetic examples that are not necessarily generated by a GAN. These are synthetic, but this is a fake eye generated with a graphics computer program. Unfortunately, this actually doesn't help the downstream classifier. There's something odd about the synthetic example that is not useful. The realism that the GAN generated data gives is much better and improves the downstream classifiers significantly. That's actually quite an interesting task where now we can use GAN generated data in addition to real data to then help with classifying which way an eye is looking. This is for a social science application. I'd like to emphasize here that this data is hard to collect. That's why we need to maybe generate or augment this dataset a little bit more. That's hard to collect because it's hard to call in new participants and ask them if it's okay to take a video of their eyes, and to get thousands of participants, that would be very costly. Another interesting use case is in speech emotion recognition. What that means is when you're speaking, detecting what emotion you might have in your speech is an interesting task where there isn't that much data out there that's labeled on whether someone sound angry or happy. This could be important for, let's say, your Alexa and Google Home or something that is trying to detect what your emotion is, to try to help you, hopefully in a very positive way. There are perhaps some real samples of angry sounding voices and happy sounding voices, and these look really interesting. They lack a visual. These are spectrograms, which is a visual representation of the spectrum of frequencies that a person might give out when they do speak. Based on some literature, these spectrograms have notable signatures that are associated with being angry or happy or other emotions. While it might be difficult to gather more data, you could perhaps instead use a GAN to generate some fake spectrograms for you as well. The data you're working with might also be very hard to obtain, such as brain scans or mammograms of tumors due to patient privacy. You can use a GAN to actually generate synthetic liver lesions. These are liver lesions generated by DCGAN, which you saw in course 1, week 2. In other cases, it might be unethical to acquire more data samples. Like if you're trying to get cancer samples, you don't want to cause cancer in people. You don't want to get data that you're not supposed to be accessing from another institution, for example, so generating fake data could be one option there. In summary, GANs are useful in generating fake data when real data is either too hard to come by or expensive to acquire. GAN generated data has very cool immediate use cases, including the data augmentation use case, you see here.
