It can be very hard to struggle with worries about this sacrament. For many centuries, priests have known that it can be quite common for people to experience worries about their confession. (I myself have had worries about confessions I have made.) For some, such worries are an occasional thing. For others they are a more regular struggle. I’m sorry that this is causing you some agony on what seems to be a regular basis.
The advice you got from that priest is spot on. In fact, given the struggles you’re facing, it seems to me that the Holy Spirit prompted him to make that remark for your benefit.
As you were likely taught when you were preparing to become a member of the Church, the Sacrament of Reconciliation has three parts.
The “three parts of the penitent” are: contrition, confession, and satisfaction. The "part of the priest " is absolution.
What I would encourage you to remember is that the most important part of the penitent is contrition.
We are not obliged to confess sins we cannot remember… because that would be impossible.
But if we have contrition (even imperfect contrition) we have the intention to confess all our sins.
If you intentionally hide a mortal sin, that shows that you don’t have contrition. Intentionally hiding a mortal sin is not compatible with contrition.
But forgetfulness and contrition can co-exist. (I would note that your post shows suggests that you would confess sins you have forgotten if you could. This could actually be taken as evidence that you have contrition.)
You should trust that if you intended to confess any mortal sins you may have forgotten then you had the proper disposition to be absolved. And if you were absolved, then you are forgiven.
(By the way, there is no obligation to confess venial sins, even though it is a very good thing to do.)
So my advice is to not obsess about the matter, but instead to thank the Lord for his mercy that endures forever, to recall your good intention, and to commend whatever you have forgotten to the Sacred Heart of Jesus who forgives even what we forget.