I'm not sure what difference it makes whether the panhandler is aggressive.
If they weren't agressive, if they were just begging instead, what then?
Here's a line from the novel Kim, when he (the child protagonist) meets a Tibetan Lama who has come into India on a pilgrimage:
'And whom didst thou worship within?' said Kim affably, squatting in
the shade beside the lama.
'I worshipped none, child. I bowed before the Excellent Law.'
Kim accepted this new God without emotion. He knew already a few
'And what dost thou do?'
'I beg. I remember now it is long since I have eaten or drunk. What is
the custom of charity in this town? In silence, as we do of Tibet, or
'Those who beg in silence starve in silence,' said Kim, quoting a
native proverb. The lama tried to rise, but sank back again, sighing
for his disciple, dead in far-away Kulu. Kim watched head to one side,
considering and interested.
'Give me the bowl. I know the people of this city - all who are
charitable. Give, and I will bring it back filled.'
I realize this is just a novel but I found it a memorable phrase: "Those who beg in silence starve in silence".
There might be some good reason to withhold charity. When my parents visited Mali they were advised to give any gifts to the village leaders and not to give to children, because tourists giving to children would encourage children to skip school.
If I remember times when I've been charitable I basically don't regret doing that. That might be a consequence of having done a good thing; the Kimattha Sutta (AN 11.1) begins with,
Skillful virtues have freedom from remorse as their purpose, Ananda, and freedom from remorse as their reward.
I'm more troubled by the fact that I could've helped him, but I didn't simply because I didn't feel I should judge him differently from other salesmen and I really didn't need the small items he was selling.
... so if that troubles you, maybe try doing something different next time?
There's this video (actually an advertisement) from Thailand: Heartwarming Thai Commercial. It starts with,
What does he get in return, for doing this every day?
Here is a long article about "Dana: The Practice of Giving". Maybe the most relevant section of it from the point of view of this question is,
THE MOTIVATION FOR GIVING
The suttas record various motives for exercising generosity. The Anguttara Nikaya (A.iv,236) enumerates the following eight motives:
- Asajja danam deti: one gives with annoyance, or as a way of offending the recipient, or with the idea of insulting him.
- Bhaya danam deti: fear also can motivate a person to make an offering.
- Adasi me ti danam deti: one gives in return for a favor done to oneself in the past.
- Dassati me ti danam deti one also may give with the hope of getting a similar favor for oneself in the future.
- Sadhu danan ti danam deti: one gives because giving is considered good.
- Aham pacami, ime ne pacanti, na arahami pacanto apacantanam adatun ti danam deti: "I cook, they do not cook. It is not proper for me who cooks not to give to those who do not cook." Some give urged by such altruistic motives.
- Imam me danam dadato kalyano kittisaddo abbhuggacchati ti danam deti: some give alms to gain a good reputation.
- Cittalankara-cittaparikkarattham danam deti: still others give alms to adorn and beautify the mind.
Favoritism (chanda), ill will (dosa) and delusion (moha) are also listed as motives for giving. Sometimes alms are given for the sake of maintaining a long-standing family tradition. Desire to be reborn in heaven after death is another dominant motive. Giving pleases some and they give with the idea of winning a happy frame of mind (A.iv, 236).
But it is maintained in the suttas (A.iv,62) that alms should be given without any expectations (na sapekho danam deti). Nor should alms be given with attachment to the recipient. If one gives with the idea of accumulating things for later use, that is an inferior act of giving. If one gives with the hope of enjoying the result thereof after death, that is also an inferior act of giving. The only valid motive for giving should be the motive of adorning the mind, to rid the mind of the ugliness of greed and selfishness.
To be honest there are Buddhist scripture which would be interpreted as recommending your being generous towards the Sangha more than towards agressive pan-handlers.
It may be surprising how much money (dollars per hour) some beggars can make, conversely it might also be surprising how poor some people can be. It's hard to assess others' needs.
Look at this story for example: Where is the story of the non-returner who “sold” pots on the side of the road?
I admire the sutta on Stinginess (SN 1.32).