I'm going to try my best and not give too much away, but unless I use a specific example, I'm not sure how else to proceed. Let's use:
Mumonkan - Case 4: The Western Barbarian With No Beard
The Case:
Wakuan said, "Why has the Western Barbarian no beard?"
Mumon's Comment:
Study should be real study, enlightenment should be real
enlightenment. You should once meet this barbarian directly to be
really intimate with him. But saying you are really intimate with him
already divides you into two.
Mumon's Verse:
Don't discuss your dream
Before a fool.
Barbarian with no beard
Obscures the clarity.
First, some preliminaries. So what is a koan? Koans are usually a secondhand account of an exchange between a master and a student or between two enlightened masters. Read as literature, these exchanges were either teaching moments or a test of another's quality of experience and/or current state of mind. Practically speaking, koans are teaching tools. Every koan points to something. Sometimes it's an aspect of the enlightened mind (like in Gutei Raises a Finger or The Buddha Holds Up a Flower), other times it can be some psychological hang up like in The Western Barbarian Has No Beard or in The Golden Carp Falls Out of the Net. There are also what are known as break through koans that are designed to mess your head up completely to really advance your practice. These would include things like Mu or Nansen Cuts the Cat in Two. (I think Hakuin breaks koans into five classes, but this is the general gist.) Like a legal precedent (koan literally means "legal case"), koans are records of past insights that can help a student form a basis of enlightened understanding (or supramundane knowledge as the Theravada would have it).
Each koan is broken into two basic parts. First, you have the case (the quoted text above). To go with it, there is what is known as the "turning phrase" (hua tou in Chinese) that express the essence of the koan. The hua tou is given to you by your teacher when you begin to work with a particular case. In meditation, you simple say that phrase on your out breath. In the case above, on every exhalation you'd just say "no beard".
Koans are an insight practice. In order to work with them effectively, you first have to develop some mastery over working with mushin or emptiness. In Theravada, this would roughly equate to an extremely deep state of access concentration. We Zen folk believe, much like you Elder folks, that unless the mind is quelled, there is no way for insight to occur. It would be an exercise in futility to ruminate intellectually over a koan although this remains a problem that many students run into. To avoid that and guide the student toward an experience of emptiness (e.g. deep access concentration), we use the mu koan to tame the mind (you can read more about that here).
Once mu or emptiness is established, the student can then start working with other koans. Working with a koan can be hell or heaven. If you've practiced jhana meditation at all, you know how oppressive deep concentration can sometimes feel. In the case of our example, saying "no beard", "no beard" to yourself, sincerely, on every out breath, with no intellectualizing over what "no beard" means, gives rise to a very deep experience of emptiness. The Mu koan equates this with "swallow[ing] a red-hot iron ball, which you cannot spit out even if you try." This is the Great Doubt. It's important to distinguish this kind of Doubt from the small doubt you'd have of, say, the Falcons winning the Superbowl. The Great Doubt is visceral and is the product of a full suspension of intellectual examination and the concentration of the mind in a state of emptiness. And it can suck. (That being said, sometimes piti can arise and become so menacingly strong that it can sweep you right into the 1st jhana. But still, you say "no beard", "no beard". Sometimes you'll be able to relax into the 2nd jhana, other times you go right back to the suck. But always - "no beard", "no beard".)
So that's the meditation side of things. Now we come out of meditation with our minds made calm and apply it to our Barbarian. First, some context. The Western Barbarian is, of course, Bodhidharma. He has a beard. Everybody knows that. To say he doesn't is like denying the 800 lbs. gorilla in the room who is throwing feces at you while ripping up your first edition copy of The Fellowship of the Ring. This simian is an asshole and you really need to acknowledge him. As a student works with "no beard" things are going to come up (we are yogacara peeps so we'd say that they arise from the alaya-vijnana or store house consciousness). For this koan, what arises will probably be something about your psychological makeup or habitual patterning that you'd rather sweep under the rug. This koan says no, no sir. Tell me what's there. You leverage emptiness to pry something out of your psyche.
I would actually say it's unwise to try to use koans to quantify progress. Some people can answer hundreds of koans and not reach enlightenment. Other can get there with just a handful. It's kind of like a game of psychological Jenga. There are pieces that can be taken quite easily at no risk to our mundane selves. You'll answer those very rapidly. Other pieces - danger! These can take longer to dislodge, but when you pull them out the entire structure can come toppling over. Your teacher is basically like the foreman of a wrecking crew. It falls on his or her experience to select those koans that will cause your little Jenga tower to come down in the least amount of moves. Koans can also be used to prod or gauge a student's mind. If you are having trouble with a particular koan, it can help point to those areas where your teacher might want to attack. In post-enlightenment practice, they also help demonstrate where you are still blocked.
Koans really aren't all that mystical. They are actually very practical in their purpose and execution. I hope that gave you at least some inkling into how they "work".
