Speculation is the rumination on any topic that can only be truly apprehended through direct insight. Unfortunately, about the only way you'll ever figure out if you're guilty of that is by asking your teacher. I know this sounds like a flip answer, but it's the only safe way to ensure that you aren't venturing into intellectualism and speculation. Let me give you an example. There's a koan that goes like this:
Case 29 from the Mumonkan: The Sixth Patriarch's "Your Mind Moves"
The wind was flapping a temple flag, and two monks started an argument. One said the flag flapped, the other said the wind flapped; they argued back and forth but could not reach a conclusion. The Sixth Patriarch said, "It is not the wind that moves, it is not the flag that moves; it is your mind that moves." The two monks were awe-struck.
Mumon's Comment
It is not the wind that moves; it is not the flag that moves; it is not the mind that moves. How do you see the patriarch? If you come to understand this matter deeply, you will see that the two monks got gold when buying iron. The patriarch could not withhold his compassion and courted disgrace.
Mumon's Verse
Wind, flag, mind, moving,
All equally to blame.
Only knowing how to open his mouth,
Unaware of his fault in talking.
When I was working with this koan, I was convinced that Master Mumon was talking about something along the lines of how the wind is karma, we're flags, and that our mind moves as a result of that. If only we could suspend karma and escape that, we'd be liberated. My teacher told me I was full of shit. Next I told him that everything exists in the mind; that experience isn't real and that only our minds are producing it. He proceeded to throw the meditation bell at me and asked if it was all in my head, why did I flinch when he threw the bell!
I eventually answered the koan (and just an FYI - it has ABSOLUTELY nothing to do with anything I wrote above), but without my teacher there to reject my ridiculous ideas, I'd still be working on it today. So to answer your question - you never know if you're speculating or not. Only your teacher can tell you when you've gone off the rails.