How does Buddhism define “self?” I read in some book that Buddha would avoid such questions, but I wonder why.
The Buddha never avoided it. All that occurred here is there was a teaching (SN 44.10) where the Buddha refused to answer certain wrongly put questions by a confused individual.
The Buddha taught:
The underlying tendency to form "self-views" lies within every new born baby (MN 64)
"Self" is an ignorant assumption that is a mental fabrication or deluded thought (SN 22.81)
The arising of this "self" thought or view is the arising of suffering (SN 12.15; SN 5.10)
This "self" idea is a "disease" - Ud 3.10
This "self" idea is a cancer, dart & affliction - MN 140
This world is burning. Afflicted by contact, it calls disease a 'self.'
Bhikkhu, ‘I am’ is a conceiving; ‘I am this’ is a conceiving; ‘I shall be’ is a conceiving; ‘I shall not be’ is a conceiving; ‘I shall be possessed of form’ is a conceiving; ‘I shall be formless’ is a conceiving; ‘I shall be percipient’ is a conceiving; ‘I shall be non-percipient’ is a conceiving; ‘I shall be neither-percipient-nor-non-percipient’ is a conceiving. Conceiving is a disease, conceiving is a tumour, conceiving is a dart. By overcoming all conceivings, bhikkhu, one is called a sage at peace.
The Buddha never avoided the topic of the self. It is in fact an important topic in Buddhism.
The self is the core of one's being. Usually, one assumes one or more of six view-positions, from MN 22:
"Monks, there are these six view-positions. Which six? There is the case where an uninstructed, run-of-the-mill person — who has no regard for noble ones, is not well-versed or disciplined in their Dhamma; who has no regard for men of integrity, is not well-versed or disciplined in their Dhamma — assumes about form: 'This is me, this is my self, this is what I am.'
"He assumes about feeling: 'This is me, this is my self, this is what I am.'
"He assumes about perception: 'This is me, this is my self, this is what I am.'
"He assumes about fabrications: 'This is me, this is my self, this is what I am.'
"He assumes about what seen, heard, sensed, cognized, attained, sought after, pondered by the intellect: 'This is me, this is my self, this is what I am.'
"He assumes about the view-position — 'This cosmos is the self. After death this I will be constant, permanent, eternal, not subject to change. I will stay just like that for an eternity': 'This is me, this is my self, this is what I am.'
Such an assumption is itself is a mental fabrication according SN 22.81. The tendency to assume this is an underlying tendency that even babies have, as stated in MN 64. (both mentioned by Dhammadhatu)
For e.g. if somebody touches your hand and you may reply "don't touch me" or "don't touch my hand". Here, clearly you assume that your form is your self, or your hand belongs to your self. This can be extended to a lot of things, like your car, your spouse, your soul or you as the soul (which is a mentally constructed concept), your consciousness or you as consciousness etc. Even when you say "I exist after death" or "I don't exist after death", you still have a self-view.
The unenlightened also suffer from objectification-classification or reification (prapanca from Snp 4.14). Everything is objectified and classified relative to self. For e.g. this is my car, but that is not my car. Or if I see some unidentified object, I may wonder if that object poses any threat to me, or if it can profit me in any way. This reification is related to craving, as explained in this answer.
Suffering is closely related to the assumption of the self. For e.g. you will suffer when your possessions are damaged or your body gets disease or you get a disease (assuming form as the self) or your family member dies or you age. You will also suffer if something you like is denied to you.
From the Buddhist perspective, the assumption of the self arises from the inter-working of five aggregates (form, feeling, perception, mental fabrications, consciousness) just as music arises when different parts of a musical instrument function together. But if you break the musical instrument to pieces, you will not find music. Similarly, if you break the five aggregates into pieces, you won't find a self.
From SN 35.246:
“Suppose, bhikkhus, there was a king or a royal minister who had never before heard the sound of a lute. He might hear the sound of a lute and say: ‘Good man, what is making this sound—so tantalizing, so lovely, so intoxicating, so entrancing, so enthralling?’ They would say to him: ‘Sire, it is a lute that is making this sound—so tantalizing, so lovely, so intoxicating, so entrancing, so enthralling.’ He would reply: ‘Go, man, bring me that lute.’
“They would bring him the lute and tell him: ‘Sire, this is that lute, the sound of which was so tantalizing, so lovely, so intoxicating, so entrancing, so enthralling.’ The king would say: ‘I’ve had enough with this lute, man. Bring me just that sound.’ The men would reply: ‘This lute, sire, consists of numerous components, of a great many components, and it gives off a sound when it is played upon with its numerous components; that is, in dependence on the parchment sounding board, the belly, the arm, the head, the strings, the plectrum, and the appropriate effort of the musician. So it is, sire, that this lute consisting of numerous components, of a great many components, gives off a sound when it is played upon with its numerous components.’
“The king would split the lute into ten or a hundred pieces, then he would reduce these to splinters. Having reduced them to splinters, he would burn them in a fire and reduce them to ashes, and he would winnow the ashes in a strong wind or let them be carried away by the swift current of a river. Then he would say: ‘A poor thing, indeed sir, is this so-called lute, as well as anything else called a lute. How the multitude are utterly heedless about it, utterly taken in by it!’
“So too, bhikkhus, a bhikkhu investigates form to the extent that there is a range for form, he investigates feeling to the extent that there is a range for feeling, he investigates perception to the extent that there is a range for perception, he investigates volitional formations to the extent that there is a range for volitional formations, he investigates consciousness to the extent that there is a range for consciousness. As he investigates form to the extent that there is a range for form … consciousness to the extent that there is a range for consciousness, whatever notions of ‘I’ or ‘mine’ or ‘I am’ had occurred to him before no longer occur to him.”