The Buddha taught (at SN 10.12) that family members should try to develop truthfulness, mental training, patience & sacrifice/generosity towards eachother.
Therefore, if your father wishes for you to visit him, you should try to train yourself to have patience and generosity towards him & his family.
You should also try to be honest & truthful and speak your concerns to your father & ask your father to ask his sister to stop being unfriendly towards you.
If your aunt is unpleasant to you, you should try to be truthful & patient towards her and tell her you wish to visit your father; you wish to do what is best for your father; so to please stop being unfriendly towards you.
If a family member is making you feel bad, you should try to truthfully communicate your feelings to them.
While your situation is difficult & challenging, you should always remember it is not you that is making bad karma but them.
As a child, you should never blame yourself of doing bad karma when it is your parents & your aunt & uncle that have done bad karma.
Your parents divorcing was bad karma done by them. Your aunt & uncle getting angry at you is bad karma being done by them.
You seem to care about your father, which is good karma done by you.
Try to see who is doing the bad karma & try to develop the goal of being better than your parents, aunt & uncle.
The Buddha taught some parents have children that are morally better than the parents. You should try to develop yourself to be morally better.