Rather than getting bogged down in ever-changing perceptions, which can be quite nauseating, for me, this was a matter of keeping things rather simple: knowing when concentration was present, knowing when it wasn't, and knowing its qualities when it was present.
Jumping prematurely to the punchline, as this practice develops, it points the mind away from the disturbances of forms and towards the emptiness, the signless and the aimless which is where knowing, as we conventionally understand knowing, no longer happens. In short, the typical human consciousness stops dead, and yet there's still heat coming from the body, breathing continues, and motion.
Initially, however, one firstly develops a familiarity with the disturbances - sometimes called mindfullness - which can take an entire lifetime for some. This is what you're trying to describe in your question: how to reconcile with the practice of mindfulness. These disturbances are listed in various formats two of them being the 5 hindrances and the 4th and 5th fetters. If you have broken the first three fetters, these disturbances stand out like a sore thumb and are thus much easier to deal with.
To answer your question more directly: the attention is always wavering; it needs a perceptual achor to define itself as 'attention' and since all perceptions are changing, so too the attention changes. As such, the attention has various qualities, textures and scopes.
Just to throw a curve ball: there is a part of your awareness (not your six sense awareness) that doesn't have any perceptual qualities whatsoever. It doesn't move, it doesn't not move. The Buddha described it best as: "one is neither here, there nor in-between the two."
