According to most sources, a soul stays in samsara until it concentrates enough karma to get yet another rebirth.
However, I failed to find any sources explaining if this process is eternal. In other words, what happens if a soul has committed so many sins that it's unable to accumulate enough karma? Or, in other words, is there a "dead end" when no more rebirth is possible?
There is no sense of accumulating sins, per se; your karma (actions) change who you are; there is also no sense that it can ever be too much to recover from (eventually). Eventually, your karma will catch up with you, sending you to heaven or hell, and making it more difficult to accumulate further good or bad karma, until you usually wind up right back where you started, somewhere in the middle. Something like the law of diminishing returns; the better or worse your rebirth, the harder it is to make further corresponding karma.
At any rate, given the infinitude of samsara, there is no such thing as too late; if there is time to accumulate good or evil, there is also time for its effects to wear off.
See the Assu Sutta, SN XV.03:
From an inconstruable beginning comes transmigration. A beginning point is not evident, though beings hindered by ignorance and fettered by craving are transmigrating & wandering on.
First of all, saṃsāra is the cycle of birth and rebirth. Within this cycle there is an unceasing flow of consciousness - not to be identified with a person, though - that passes from existence to existence. The only way out of saṃsāra is nirvāṇa - this is your "dead end". To reach this state all accumulation of new karman has to stop and all accumulated karman has to be extinguished. Unless nirvāṇa is attained at some point, the samsaric existence is without end. Without beginning, it is anyway.
Now, at the end of one existence, the flow of consciousness - vijñāna-srotas - is thought to enter the womb of the mother to form a new being, if rebirth takes place in the human sphere. other than the human sphere, there are five other spheres of rebirth. They are galled gati - more or less "gait", but also "way" or "station", sometimes, in Mahāyāna also loka - "world". These are the spheres of: gods, demons, animals, hell-beings and hunger-ghosts. Check here for more: http://en.wikipedia.org/wiki/Desire_realm
I'm not hundred percent sure, but this seems to be rather a mahayanic conception. Note also, that salvation - escape from the cycle of births - is only possible in the human sphere, since for example gods are just to blissful, to feel a need to stop this process.
To get back to your question, in popular Buddhism there may well be a hell or different hells, where people are also thought to be punished according to their deeds. To confirm this, there is the famous Wewurukannala temple in Sri Lanka, called the hell temple, which is full of rather gruesome pictures of these hells, see, for example, here: http://lanka-houses.com/sightseeing/wewurukannala-hell-temple.html
Buddhist doctrine would rather explain these hells as painful states of mind.
Yes, there are approximate equivalent realms:
http://en.wikipedia.org/wiki/Buddhist_cosmology
Although, one thing to keep in mind is that the Buddha taught his followers not to speculate on cosmology, soul, or eternalism, since such speculation increases our suffering:
http://en.wikipedia.org/wiki/The_unanswered_questions http://www.accesstoinsight.org/tipitaka/sn/sn35/sn35.023.than.html
My best guess is that the stuff about other realms was added after the Buddha's death, as there was a lot of contact with Hindu/Vedic practices in the centuries that followed, moreso in the Mahayana and Vajrayana traditions.
There is also a sense in Buddhism that heaven and hell and psychological
The Zen monk Hakuin was asked by a Samurai warrior where heaven and hell were. For his answer Hakuin started to insult the Samurai. The Samurai was infuriated and drew his sword. Hakuin then insulted the sword which made matters worse. The Samurai's rage was now at its peak and he moved as if to behead Hakuin. At this point Hakuin said
Here open the gates of hell
On hearing this and seeing Hakuin's utter composure the Samurai sheathed his sword and bowed. Hakuin then said
Here open the gates of heaven
For Hakuin and the Samurai heaven and hell are us and our responses.
Adding to @qweilun answer, on the subject of non-cosmology.
Although, one thing to keep in mind is that the Buddha taught his followers not to speculate on cosmology, soul, or eternalism...
Samsara, death and rebirth can be seen by a metaphorical perspective.
Within this cycle there is an unceasing flow of consciousness - not to be identified with a person, though - that passes from existence to existence.
You can look at you past and see how many different people you already were. That child of some decades ago don't exist anymore. That adolescent also now exists only as a memory. In a metaphorical sense we die, rebirth and cycle countless times through a single life.
So, back to the main question, heaven and hell also can be seen in a metaphorical sense. For example, when you achieve something important you can feel yourself in heaven; now when you think everything is going wrong with your life, it would feel like hell. Someone can be in the heaven in the morning, then feel in hell in the night.
And also karma can be seen as an accumulation of thoughts, certainties or feelings that makes you change, and propel the wheel of life.
As samsara implies, the cycle of phenomena keeps turning, it doesn't stop with a dead end.
Buddhism teaches about 31 planes of existence
But it doesn't teach about a soul staying somewhere or going from one place to another. No stay at any place is forever. But you can get stuck in hells for a long time if you commit 1 of 5 weighty bad Karmas like Matricide, Patricide etc. On the good side, you can attain Arupa Jana and get stuck in heavens for a long time. But the closest you get to a deadend is being stuck in Sansara(cycle of birth and death) because of your clinging to certain serious false views.
Firstly in Buddhism there is no concept of soul. Life is governed by the 3 marks of existence. Hence even the result of Karma is impermanent. Hence there cannot be any state which is eternal either positive or negative.
There is a state where no rebirth is possible. You have to transcend conditioned existence to get to this state.
Rebirth does not happen due to having accumulated karma. When you die you naturally get reborn. They happens when you run out of karma sustaining to this birth or the body cannot be retained any more due to decay or damage or end of the natural lifespan of the particular type of being. Your next existence is either due to a strong karma caused by something you did recently, habitual karma or some other mental state that may arise at the moment of death. If your mind is free from imbalance and attachment at the time of death then you'll reach this "deadend" which is Nirvana.
In addition the are multiple planes of existence which are have varying degrees of pain and pleasure. These also are analogous to heavens and hells.
