The danger is when we start to procrastinate and complain about our situation. When a person thinks his volition is not guided by virtue but by an external circumstance that person is in servitude, and for a person who is in servitude there is no liberation.
All virtue lies in detachment and if a person acts virtuously then he is free from attachment. However, if a person acts for fear of losing an external thing, for example, a partner or child his/her's volition is attached to an external thing and if start he admits to his mind that these things are above his/ her freedom this is servitude and that is the only danger for man.
We have to maintain at all time a free volition, free to follow virtue, and if you ever feel subjection and unable to act with goodwill and if you start to complain about your situation then you have to remind yourself that the door is open, if you don’t you will start to live a life of a worm not of man.
Quoting from other schools....
Don’t be more cowardly than children, but just as they say, when the
game is no longer fun for them, ‘I won’t play any more,’ you too, when
things seem that way to you, say, ‘I won’t play any more,’ and leave,
but if you remain, don’t complain
Aversion is simply altering your right intention directed by virtue due to fear or ignorance while still maintain your freedom of choice in your thinking whereas a danger is admitting a mindset of servitude subjecting your volition to be controlled by someone else, thinking that you are not free when you are truly free.