"so what I am left with is the striking fact that this amount of delusion is possible in someone who calls himself a serious Vipassana meditator, which worries me with regard to my own practice. If someone who has been practicing for over 10 years thinks they are aware and are everything but, what am I going to do?"
While this is disheartening indeed, we probably should not measure ourselves against anyone. Consider the following sutta:
[...] From there the Blessed One addressed the bhikkhus, "Bhikkhus, I will teach the method of the forest. Listen and attend carefully, I will teach.
Here, bhikkhus, the bhikkhu abides in a certain forest stretch. When abiding there, unestablished mindfulness does not get established, unconcentrated mind does not concentrate, not destroyed desires do not get destroyed, and the not attained noble end of the yoke is not attained; as for the four requisites of life for the one gone forth, robes, morsel food, dwellings, and requities when ill, are collected with difficulty. That bhikkhu should reflect:
'I abide in this forest stretch, to me abiding in this forest, unestablished mindfulness does not get established, unconcentrated mind does not get concentrated, not destroyed desires do not get destroyed, and the not attained noble end of the yoke is not attained; as for the four requisites of life for the one gone forth, robes, morsel food dwellings, and requisites when ill, are collected with difficulty.'
Bhikkhus, he should not abide in that stretch of forest, he should leave it by night or by day.
-- MN 17 Vanapattha Sutta - Jungle Thickets
Thus, to evaluate progress, we can reflect on the things like:
- does concentration comes easier then before or harder than before?
- does unwholesome thoughts come more often or less often?
- etc. (can go so far as all the seven enlightenment factors, five hindrances, etc)
The immediate lesson of the sutta above, on the other hand, is that the environment (work, duties, people, food, etc) we find ourselves in may enable or obstruct some aspects of our practice. While someone fully committed to the practice may just take the words above to heart and leave whenever the environment is unsuited, those of us who did not make such commitment might need to find a trade-off.
Another perspective to bring is that the Vinaya orient the sangha members to come forward whenever they transgress. Thus, we should expect people (even the ones we consider to be advanced) to make mistakes.
"But these impulses are strong and the mind is clever. What can I do to prevent this behavior? Can there be a mistake in the meditation practice itself that causes such obscurations to persist?"
Here's a few suttas describing how the Buddha handled these issues in his own mind:
[...] As I abided thus, diligent, ardent and resolute, a thought of [sensual desire, ill will, cruelty] arose in me. I understood thus: 'This thought of [sensual desire, ill will, cruelty] arose in me. This leads to my own affliction, to others affliction, and to the affliction of both; it obstructs wisdom, causes difficulties, and leads away from Nibbāna'. When I considered: 'This leads to my own affliction', it subsided in me; When I considered: 'This leads to others affliction', it subsided in me; When I considered: 'This leads to the affliction of both', it subsided in me; When I considered: 'This obstructs wisdom, causes difficulties, and leads away from Nibbāna', it subsided in me. Whenever a thought of [sensual desire, ill will, cruelty] arose in me, I abandoned it, removed it, did away with it.
Bhikkhus, whatever a bhikkhu frequently thinks and ponders upon, that will become the inclination of his mind. [...]
But with excessive thinking and pondering I might tire my body, and when the body is tired, the mind becomes strained, and when the mind is strained, it is far from concentration. So I steadied my mind internally, quieted it, brought it to singleness, and concentrated it. Why is that? So my mind should not be strained.
-- Dvedhāvitakka Sutta [Bodhi trans.], MN 19
and:
[...] If, while he is examining the danger in those thoughts, there still arise in him evil unwholesome thoughts connected with desire, with hate, with delusion, then he should try to forget those thoughts and should not give attention to them. When he tries to forget those thoughts and does not give attention to them, then any evil unwholesome thoughts connected with desire, with hate, and with delusion are abandoned in him and subside. With the abandoning of them his mind becomes steadied internally, quieted, brought to singleness, and concentrated. [...]
If, while he is trying to forget those thoughts and is not giving attention to them, there still arise in him evil unwholesome thoughts connected with desire, with hate and with delusion, then he should give attention to stilling the thought-formation(*) of those thoughts. [...]
If while he is giving attention to stilling the thought-formation of those thoughts, there still arise in him evil unwholesome thoughts connected with desire, with hate, and with delusion, then, with his teeth clenched and his tongue pressed against the roof of his mouth, he should beat down, constrain and crush mind with mind(**) (***).
-- Vitakkasanthāna Sutta, [Bodhi Trans.] MN 20
(*) "relaxing of thought-fabrication", per Thanissaro Bhikku
(**) "crush his mind with his awarenes", per Thanissaro Bhikku
(***) "crush unwholesome states of mind with wholesome states of mind", per Majjhima Nikāya Atthakathā