How Cyber department use Confusion Matrix

We used to think there was just right and wrong. But when it comes to Machine Learning, there are false positives and false negatives. Do you know the difference?

Topic Covered

1. Introduction to the confusion matrix
2. Ture Positive, True Negative, False Positive, and False Negative
3. Benefits of the confusion matrix
4. Recall, Precision, Accuracy, and F-measure in the confusion matrix
5. Important terms in the confusion matrix
6. How TN,TP,FP n FN can be used in Cyber attacks detection
7. Conclusion

What is Confusion Matrix?

A confusion matrix is a table that is often used to describe the performance of a classification model(or “classifier”) on a set of test data for which the true values are known. The confusion matrix itself is relatively simple to understand but the related terminology can be confusing.

In simple words, a confusion matrix is nothing but a table with 2 dimensions. “Actual” and “Predicted” and furthermore, both the dimensions have “True Positives(TP)”, “True Negatives(TN)”, “ False Positives(FP)” and “False Negatives(FN)”.This is how a confusion matrix looks like. Just think of the Positives and Negatives as the right and wrong values for now.

Let us go over the meaning of initials -

• TP-> True Positive
  The values which are actually true and were predicted to be true by the model.
• TN-> True Negative
  The values which are actually false and were predicted to be false by the model.
• FP-> False Positive
  The values which are actually false and were predicted to be true by the model.
• FN-> False Negative
  The values which are actually true and were predicted to be false by the model.

Now, let’s understand the classification concept in terms of True vs False and Positive vs Negative with some examples.

Case 1: A simple story of Boy and a wolf,

For having fun, a boy shouted out “Wolf”, even though there is no wolf, villagers ran out to save themselves but soon got angry when they realized the boy was playing a joke.

One day, the boy saw a wolf in reality and called out “Wolf is coming”, but villagers denied to be fooled again and stayed at home. And then, the hungry wolf demolished the village, destroyed their crops. After that, the entire village suffered many problems.

Making definitions:

• “Wolf” is a positive class
• “No wolf” is a negative class

Now, a wolf-prediction can be designed using 2x2 confusion matrix that could reflect all four possible conditions;

Benefits of Confusion Matrix

1. It gives information about errors made by the classifier and the types of errors that are being made.
2. It reflects how a classification model is disorganized and confused while making predictions.
3. This feature assists in prevailing over the limitations of deploying classification accuracy alone.
4. It is practised in conditions where the classification problem is profoundly imbalanced and one class predominates over other classes.
5. The confusion matrix is hugely suitable for calculating Recall, Precision, Specificity, Accuracy and AUC-ROC Curve.

Error/Misclassification Rate
The number of predictions that the classification model predicted wrongly.
Error Rate = (FP+FN)/(TP+TN+FP+FN)

Accuracy
The number of predictions that the classification model got right
Accuracy = (TP +TN)/(TP+TN+FP+FN)

Precision
When the model predicts the positives, how often is it right?
Precision = (TP)/(TP+FP)

Recall
Out of the actual positive labels, how many did the model correctly identify?
Recall = (TP)/(TP+FN)

Example:

Consider we have a dataset that looks like this -

x=[“cat”, “ ant”, “cat”, “cat”, “ant”, “ant”]

y=[“ant”, “ant”, “cat”,“cat”, “ant”, “cat”]

Here x is the actual data that we have, and y consists of the predicted values by the model.

Explanation

Cat was predicted correctly only two times and so was ant.

The number of times cat was predicted as an ant was 1 and vice versa.

This is how we build a confusion matrix, some other examples would be -

• A patient has cancer or not
• Did a student pass/fail in his class?
• A cyber attack has occurred or not?
• Someone is pregnant or not?

Have a look at this image, what do think of it??

Type I and Type II Errors

Type I Error — A type I error is also called as False Positive(FP).

Type II Error — A type II error is also called as False Negative(FN).

Type II errors are considered to be more crucial than type I errors.

Consider you have built a model that detects if a patient has cancer even though he/she doesn’t.

However, if the type II error is high, then it means that the patient having cancer is predicted to be fine. This may take away that patient’s life as the treatment will not be taken thinking that everything is fine.

Note:
- This confusion matrix is an example of a binary classification problem.
- If there are 3 categories, we just add 1 more row and column and so on for more categories.

What is Cyber Attack?

A cyber attack is an attack on the servers or computer in the public or private internet where the attacker seeks to expose, damage, alter, disable or try stealing the current data or changing the system configuration, and that is done unauthorized. The act of doing this cyberattack is called cybercrime.

Some of the examples of cyber attacks are:

• Stealing corporate attack and hacking servers
• Exposing someone privacy and harassing
• Stealing bank details and card details
• Fishing Sites and Scam
• IoT device hacking
• Flooding the servers with unnecessary traffic etc

Let’s now see how confusion matrix works in cyber security…

There are many attacks happening every now and then… we as a human can’t work 24*7 to monitor these attacks. We use Machine Learning, train the model and make it predict of the attacks happening.

Now the 4 Scenarios come into picture…

1. True Positive
2. True Negative
3. False Positive
4. False Negative

True Positive Case in Cyber:

Some intruders attack and the model also predicts about the attack, since the model predicted we become alert and prevent the intruder from attacking.

True Negative Case in Cyber:

Intruder will not attack and the machine also predict the same that their is no danger, everything is safe and fine. The cyber department also chills in this case.

False Positive / Type I Error :

This type of error can prove to be very dangerous. Our system predicted no attack but in real attack takes place, in that case no notification would have reached the security team and nothing can be done to prevent it. The False Positive cases above fall in this category and thus one of the aim of model is to minimize this value.

False Negative/Type II Error :

This type of error are not very dangerous as our system is protected in reality but model predicted an attack. the team would get notified and check for any malicious activity. This doesn’t cause any harm. They can be termed as False Alarm.

Conclusion

A confusion matrix is a remarkable approach for evaluating a classification model. It provides accurate insight into how correctly the model has classified the classes depending upon the data fed or how the classes are misclassified.

Thank you! keep learning! keep growing! keep sharing!

Krithika Sharma
If you enjoyed this, follow me on Medium for more
Let’s connect on LinkedIn