Let me answer this question with one of Milarepas songs:
A yogi, I roam the mountains.
Like a great Mandala, my body is full of bliss.
Cleansed of desires and pride, I feel well and happy.
With longing for diversions extinguished, I feel joy in solitude.
Since I have renounced all things, I am happy in a desolate land.
Since I have cut all ties with kin, getting and saving are not
worries.
Happy and joyous do I live ... without plans or schemes.
I want neither fame nor glory. Wherever I stay, whatever I wear or
eat, I fell truly content
Also the third Karmapa used to practise in different places from mountain to mountain without a fixed places.Most of the yogis choose a solitary retreat and they rarely return to monasteries.
My native land is all lands, In no particular direction. My monastery
is the solitary mountains, In no particular place. My family is all
the beings of the six realms.
~ Shabkar
We seek freedom from attachment when you realize that all that appears and exists to be your mind, there is no path of enlightenment apart from that.
As long as there exists a place in Samsara to give us freedom,we aren't truly free!