Transfer learning - small database

Question

I am trying to use transfer learning in medical (ultrasound pictures). The problem is - I have very limited picture database = 400 (360+40). I am using resnet50 (I don't think this is important but maybe I'm wrong). Resnet as feature extractor + SVM is not great but normalized confusion matrix is somewhat about:

1.0 0 0.4 0.6

Now, I wanted to fine-tune resnet. And the problem is that CM at the beginning looks like:

0.8 0.2 0.6 0.4

is something like this:

1.0 0 0.8 0.2

Below you can see training + test loss/accuracy.

Now I thought it is overfitting (due to too large rate capacity / database) but someone pointed that network might not be learning. What is the case?

Answer 1

Seeing your graphs it does not show that the model is not learning, as the training accuracy is high. If the model wasn't learning anything even this would have been low. Also, you are right that the model is overfitting because it is performing very well on the training set and poorly on the test set.

The overfitting scenario is also confirmed by the fact that you have a very small dataset. You might want to fine-tune all the layers of the resnet as shown here. Also, try early stopping and dropout to prevent the model from overfitting.