I think this is a very important question, but I can't see any way that a beginner who is unfamiliar with the Dharma can figure it out without essentially moving at least a little off the beginner's position. There's really no way to "bootstrap" yourself into trusting teachers without doing some upfront work yourself. So the main advice I'd give on that front is, go study the Dharma!
But perhaps there are some general rules of thumb that could also be useful. For example, to begin with perhaps a newbie should stick with the more established traditions: Theravada, Zen, mainline Tibetan, etc. That would necessarily exclude things like Soka Gakkai International (for example), but it's not at all arguing that SGI etc are bad. Rather it's simply to say that until someone finds their feet and understands what all the styles of Buddhism have in common, it's probably best to stick with more established styles, and then as one comes to understand things decide if moving to a newer or less well known approach is right for the person concerned.
Another general principle is to watch out for the "cult style" as the OP mentioned. Some good pointers in that direction are to be found around the net -- here for example. So if they're demanding unreasonable amounts of money, or seem to be highly dependent on the personality of the teacher, it's worth putting those on the list of "reasons for possible concern".
A third point is perhaps more controversial, but I'll offer it anyway being based on my own experience. I confess that to some extent I am a newbie in this situation. I have studied quite a bit, and have established a basic meditation practice, but I still haven't yet decided on a teacher, tradition, and so on. I consider that to be a very important part of my practice right now. And one of my considerations is to look at the life of the tradition's founder and those of its major figures. For that reason, I find myself a bit hesitant to engage with, for example, Shambhala. I still find myself trading off the huge contributions they have made, in terms of publication and teaching, and also some of their ideas, with the lifestyles and actions of the founder Chogyam Trungpa and his successor Ösel Tendzin. This is a very personal thing, and as I say I'm not rejecting Shambhala outright, but I suggest that observing the correspondence between the actions and teachings of major figures in the organization is a useful test.