It is completely normal, it does happen to me as well.
What occurs to me is that it is actually my pulse and as such swaying back and forth is in the exact rhythm of it, as it follows the rhythm of the blood flow in the body (presumably heart or artery). It happens by itself even in state of higher concentration and calm in Samatha.
I just notice it and let it happen, I cannot do anything about it really. Neither should you, so I advise the same.
Note though that if it's indeed your blood flow (as with me), it might be due to a health condition background. Be cautious and examine it, for your body shouldn't be pacing at such calmness.
My solution is that I just notice it and let it happen, I cannot do anything about it really. Not immediately. Neither should you, so I advise the same.