"99 little bugs in the code, 99 little bugs. Take one down, patch it around, 117 little bugs in the code."
Previously we've discussed how AI models rely on data to be trained through machine learning. In this post we will further discuss how the data is used to train and test these models.
In my previous post regarding Machine Learning, we discussed the difference between Machine Learning and Artificial Intelligence and briefly touched on how machine learning works. In this post I would like to go deeper into the data a model uses and more specifically how it's used. We've previously mentioned how it is important to train a model with as much data as we can in order to prevent bias or overfitting. However you may be surprised to know that you don't want to use all of you data in order to train your model. Data is separated into three main categories, each of which have their own function in the creation of an AI model.
Training Set
This first category of data contains the data which will be directly used to train the model. This set of data contains the most amount of data and it is was is used in the assigning of labels to data. This set of data is during optimization when making a guess, measuring loss, and using calculus to move down the curve of a Gradient Descent. If these words confused you thats okay as I will go into further detail in my next blog regarding Optimization, Loss, and Gradient Descent. For now lets just stick to Data.
Validation Data
This second category of data pertains to the data that the computer model does not use during the training process, and is instead used to test the accuracy of the current model with data it has not seen yet.
Test Data
Test data refers to the data which similar to validation data, was held back during the training and validation process. This data is used to test a finished model and acts as a real world test to the model.
Now that we have a better understanding on how data is distributed during the creation of a model, we will next discuss how the Optimizations Method is used to increase the accuracy of and will touch on the concepts of Loss, Gradient Descent, and Learning Rate, and how they are involved in the machine learning process.
0 Comments