So, of course, I can quote some Sutta and say this is similar to your situation and so-and-so happened with Gautama Buddha, da-di-da-di-da etc., and so you must do this or that. But I want to get practical here.
I think it is important to understand this. What you are essentially trying to do is looking for a "philosophy" that is a hammer-on-a-nail for every little situation we are faced with.
In this light, you are seeing this "concept" of Buddhism in your head, which has useful texts regarding certain aspects of life. Also it is a fact that times have gone on. The world is constantly changing. Reliance on a certain philosophy and buzzwords (like "let go") and the general people's perception on Buddhism per se, can cause conflicts like these.
That's why my answer here is a question back to you, because we need to think this through carefully. Perhaps it is your "concept" of Buddhism as this philosophy that asks you to "let go" is the problem. Why look at Buddhism for somebody insulting you?
Too many words are going around like this today. "Let go", "let it be", "mindfulness" are all these so-called Instagram/Twitter hashtags these days. Perhaps we are making a mistake in holding on to some concocted philosophy which you don't know is right or wrong.
Situations in your life and Buddhism as your "concept" are two separate things, don't mix them up. :) Try to change your idea of Buddhism. It is not trying to tell you what to do when someone insults your wife. It is trying to tell you about samsara and liberation essentially.
Finally, coming to your situation of insults and everything, don't look at Buddhism/Hinduism/Christianity/whatever for answers. Will the answer ever be there in the texts as to what is the right thing to do? Has it ever been that way? If your answer is yes, then probably I cannot help you.