While training machine learning models in Keras, Mean Squared Error (MSE) is often used as the loss function for regression problems. Unlike classification, regression deals with continuous values, which makes it inapplicable to standard accuracy. Since MSE focuses on minimizing errors rather than predicting the discrete labels, accuracy functions are not defined by Keras for regression tasks. Instead, metrics like Mean Absolute Error (MAE), R^2 Score, and Mean Absolute Percentage Error (MAPE) are used. However, a custom accuracy function can be created which will help to measure predictions within an error range that is acceptable. This provides an accuracy-like interpretation for regression models.
In this blog, we are going to talk about Mean Squared Error, defining an accuracy function for regression, and how accuracy is defined by Keras while using MSE as the loss function. So let’s get started!
Table of Contents
What is Mean Squared Error?
Mean Squared Error (MSE) is commonly used for regression tasks. It is used to measure the distance between the predicted values and the real values.
The formula for MSE (Mean Squared Error) is given below:
Where,
- y true denotes the actual values.
- y pred denotes the predicted values.
- n denotes the total number of samples.
If the MSE of a model is low, it means that the predictions of the model are closer to the true values.
Does Accuracy Work with MSE?
In classification tasks, accuracy is straightforward. It is used to measure the number of predictions that match the actual labels. Accuracy can be defined as:
Accuracy = Correct Predictions / Total PredictionsThe concept of “correct” or “incorrect” predictions does not exist in regression problems, where the goal is to predict continuous values. Instead of using binary correctness, predictions are evaluated based on their distance from the actual values. Thus, accuracy is not a suitable metric for regression tasks.
MSE (Mean Squared Error) is used to measure the average of the squared difference between the predicted and the actual values. Therefore, it does not focus on counting correct predictions, rather it helps in minimizing the overall error. Due to this difference, accuracy is not defined by Keras as a default metric when the loss function is MSE. Instead, other evaluation metrics like Mean Absolute Error (MAE), R^2 Score, and Mean Absolute Percentage Error (MAPE) are used which helps to assess the performance of the model in regression problems.
Defining an Accuracy Function for Regression
Keras does not provide a built-in accuracy metric for MSE. Therefore, you can define your own custom accuracy function which can measure accuracy while using Mean Squared Error (MSE) as the loss function. Unlike classification, where accuracy is straightforward, regression accuracy is used to measure how close the predicted values are to the actual values.
1. Using Mean Absolute Error (MAPE)
You can measure accuracy in regression by using Mean Absolute Percentage Error (MAPE). It is used to calculate the average percentage error between the actual and the predicted values. The formula for MAPE is given below:
MAPE expresses the error as a percentage. This makes it a useful metric for evaluating the alignment of the model’s predictions with the actual values. The lower the MAPE, the better the performance of the model.
2. Defining a Custom Accuracy Function in Keras
Another approach would be to define a custom accuracy function that considers a prediction “accurate” even if the prediction falls within a specified tolerance range of the actual value. For example, you can declare the prediction correct if the forecasted data falls within a 10% margin of truth.
This approach helps in translating the regression performance of the model into an accuracy-like metric. This makes it easy to interpret the effectiveness of the model in scenarios where it is not meaningful to have absolute correctness of the model. By implementing these functions in Keras, you can track the improvements of the model for regression tasks in a better way.
Implementation of Custom Accuracy in Keras
Now we will train a simple Keras Regression Model and then define a custom accuracy function. Below are the steps given for the implementation of the same.
Step 1: Import the dependencies
Example:
Explanation:
The above only imports the necessary libraries like NumPy, TensorFlow/Keras, and Scikit-Learn for the building and training of the neural network. This code will not generate an output because it only contains the import statements. It does not define or execute any functions, models, or data processing steps.
Step 2: Generating a Sample Data
Here, we will create a simple dataset where y = 3x + 5.
Example:
Explanation:
The above code is used to generate synthetic data. It uses a linear equation with noise and then splits the data into training and testing sets using the 80:20 ratio. This code doesn’t generate an output because it only splits the data and does not include any print or visualization statements for displaying the generated values.
Step 3: Defining a Custom Accuracy Function
Now we will define an accuracy function for checking the number of predictions within 10% of the true value.
Example:
Explanation:
In the above code, the function custom_accuracy is used to calculate the percentage of predictions within 10% of the true values. This is done by calculating the relative error and finding the average of the correct predictions. This code does not generate an output because it only defines a function and no execution is done with any input data.
Step 4: Building the Regression Model
Now we will create a simple neural network using Mean Squared Error and our custom loss accuracy function.
Example:
Explanation:
The above code is used to define a neural network having two hidden layers. It then compiles it with the Adam Optimizer along with Mean Squared Error (MSE) loss. It then trains the model on the dataset while tracking a custom accuracy metric. The above code does not generate an output because the training process of the model ( model.fit ) runs silently ( verbose = 0 ). This means that no logs or progress updates are printed.
Step 5: Model Evaluation
Here, we will be evaluating the performance of the model.
Example:
Output:
Explanation:
The above output shows that the trained model has an MSE loss of 4.3449 on the test set, along with a custom accuracy metric of 0.6641. This means that 66.41% of predictions were within the defined accuracy threshold.
Alternative Metrics for Regression Accuracy
For alternative ways of measuring accuracy in regression, you can consider the following:
1. R^2 Score (Coefficient of Determination)
It is used to measure how well the predicted values match the actual values. The higher the difference the better it is.
Example:
Output:
Explanation:
The above is used to calculate the R^2 (coefficient of determination) score. This evaluates the performance of the model by comparing the actual and the predicted values. It then prints the results which are rounded to four decimal places.
2. Mean Absolute Error (MAE)
It measures the average absolute difference between the actual and predicted values.
Example:
Output:
Explanation:
The above code calculates the Mean Absolute Error (MAE) between the actual and the predicted values. This measures the average prediction error of the model. It then prints the result which is rounded to 4 decimal places.
Conclusion
While using Mean Squared Error (MSE) as the loss function in Keras, standard accuracy metrics are not applicable. This is because regression problems do not have a concept of “correct” or “incorrect” predictions. Instead, you can define custom accuracy metrics like Mean Absolute Percentage Error (MAPE) or an accuracy function based on tolerance, which will evaluate the model meaningfully. In addition to that, metrics like R^2 score and Mean Absolute Error (MAE) helps in providing deeper insights into the prediction performance of the model. By choosing the right evaluation of methods, you can better assess and improve the accuracy of the regression model in Keras.
FAQs
1. Why doesn’t Keras provide an accuracy metric for regression tasks?
Keras is useful for defining accuracy for classification problems, where predictions are either correct or incorrect. For regression, predictions are continuous values, which makes traditional accuracy inapplicable.
2. How can I measure accuracy in regression tasks?
Instead of measuring accuracy, metrics like Mean Absolute Error (MAE), Mean Squared Error (MSE), and R^2 score are used for the evaluation of the model performance. A custom accuracy function can also be defined based on a tolerance threshold.
3. What is Mean Absolute Percentage Error (MAPE), and how is it useful?
MAPE is used to measure the percentage between actual and predicted values. This helps to evaluate regression models by showing their relative errors. It can be used while dealing with datasets with varying scales.
4. How can I implement a custom accuracy function in Keras for MSE loss?
For defining a custom accuracy function, you should calculate the percentage error for each prediction and consider those that are present within a threshold. Then, you should calculate the mean accuracy.
5. Which metric is best for regression models in Keras?
The meat metric depends on the use case. Two of the most commonly used metrics are MSE and MAE, while R^2 Score measures how well the model explains variance. MAPE is useful for business applications when percentage errors matter.