In MN 21 (Kakacupama sutta), the Buddha says:
"Monks, there are these five aspects of speech by which others may address you: timely or untimely, true or false, affectionate or harsh, beneficial or unbeneficial, with a mind of good-will or with inner hate. Others may address you in a timely way or an untimely way. They may address you with what is true or what is false. They may address you in an affectionate way or a harsh way. They may address you in a beneficial way or an unbeneficial way. They may address you with a mind of good-will or with inner hate.
In any event, you should train yourselves: 'Our minds will be unaffected and we will say no evil words. We will remain sympathetic to that person's welfare, with a mind of good will, and with no inner hate. We will keep pervading him with an awareness imbued with good will and, beginning with him, we will keep pervading the all-encompassing world with an awareness imbued with good will — abundant, expansive, immeasurable, free from hostility, free from ill will.' That's how you should train yourselves."
And at the end:
"Monks, even if bandits were to carve you up savagely, limb by limb, with a two-handled saw, he among you who let his heart get angered even at that would not be doing my bidding. Even then you should train yourselves: 'Our minds will be unaffected and we will say no evil words. We will remain sympathetic, with a mind of good will, and with no inner hate. We will keep pervading these people with an awareness imbued with good will and, beginning with them, we will keep pervading the all-encompassing world with an awareness imbued with good will — abundant, expansive, immeasurable, free from hostility, free from ill will.' That's how you should train yourselves.
An addendum to this would be when the authors of rude language and idle talk or gossip are buddhist themselves committed to the practice. In that case, with the above in mind, one may, at his/her discretion, chose to bring their attention to this, so as to help them steer back to their practice. Of course, one certainly may chose to intervene in the case of non-buddhists, but since they may not be committed to not perform these kinds of speech, raising the matter can make things worse -- that's when taking responsibility for being timely and beneficial applies specially to ourselves.
In the case of monks engaging in these acts, there may be something specific in the Vinaya, the collection of monastic rules, about how to handle these situations.