In Kakacupama Sutta, the Buddha says:
"Monks, there are these five aspects of speech by which others may address you: timely or untimely, true or false, affectionate or harsh, beneficial or unbeneficial, with a mind of good-will or with inner hate. Others may address you in a timely way or an untimely way. They may address you with what is true or what is false. They may address you in an affectionate way or a harsh way. They may address you in a beneficial way or an unbeneficial way. They may address you with a mind of good-will or with inner hate.
In any event, you should train yourselves: 'Our minds will be unaffected and we will say no evil words. We will remain sympathetic to that person's welfare, with a mind of good will, and with no inner hate. We will keep pervading him with an awareness imbued with good will and, beginning with him, we will keep pervading the all-encompassing world with an awareness imbued with good will — abundant, expansive, immeasurable, free from hostility, free from ill will.' That's how you should train yourselves."
[...]
"Monks, even if bandits were to carve you up savagely, limb by limb, with a two-handled saw, he among you who let his heart get angered even at that would not be doing my bidding. Even then you should train yourselves: 'Our minds will be unaffected and we will say no evil words [...]' That's how you should train yourselves.
-- Kakacupama Sutta, MN 21
An addendum to this would be when the authors of rude language and idle talk or gossip are buddhist themselves committed to the practice. In that case, with the above in mind, one may, at his/her discretion, chose to bring their attention to this, so as to help them steer back to their practice. Of course, one certainly may chose to intervene in the case of non-buddhists, but since they may not be committed to not perform these kinds of speech, raising the matter can make things worse -- that's when taking responsibility for being timely and beneficial applies specially to ourselves, specifically:
Abandoning false speech, he abstains from false speech; he speaks truth, adheres to truth, is trustworthy and reliable, one who is no deceiver of the world. Abandoning malicious speech, he abstains from malicious speech; he does not repeat elsewhere what he has heard here in order to divide [those people] from those; thus, he is one who reunites those who are divided, a promoter of friendships, who enjoys concord, rejoices on concord, delights on concord, a speaker of words that promote concord. Abandoning harsh speech, he abstains from harsh speech; he speaks such words as are gentle, pleasing to the ear, and lovable, as go to the heart, are courteous, desired by many, and agreeable by many. Abandoning gossip, he abstains from gossip; he speaks at the right time, speaks what is fact, speaks on what is good, speaks on the Dhamma, and the Discipline; at the right time he speaks such words as are worth recording, reasonable, moderate, and beneficial.
-- Culahatthipadopama Sutta, MN 27 (Nanamoli/Bodhi)
In the case of monks engaging in these acts, there may be something specific in the Vinaya, the collection of monastic rules, about how to handle these situations.
