This is a profound question. Somebody may want to delve into spirituality because he is a "loser" and isn't prospering in life. This is not a new concept. In fact, the Ratthapala Sutta addresses this very point. You should go to the link and study the sutta in full.
In Majjhima Nikaya 82, we are introduced to the ascetic Ratthapala, who took up the life of the holy man even though he was from a privileged background, and wealthy, athletic, successful, in the prime of his youth. We are led to believe that, in the secular life, he had countless opportunities at his disposal.
In the Sutta, the king asks the ascetic Ratthapala, in effect, Why are you joining the spiritual life with the other ascetics, when they are not rich or strong and successful, and you are?
King Koravya sat down on a seat made ready and said: "Master
Raṭṭhapāla, there are four kinds of loss. Because they have
undergone these four kinds of loss, some people here shave off their
hair and beard, put on the yellow robe, and go forth[.] What are
the four? They are loss through ageing, loss through sickness, loss of
wealth, and loss of relatives.
And what is loss through ageing? Here, Master Raṭṭhapāla, someone is
old, aged, burdened with years, advanced in life, come to the last
stage. He considers thus: ‘I am old, aged, burdened with years,
advanced in life, come to the last stage. It is no longer easy for
me to acquire unacquired wealth or to augment wealth already
acquired....
And what is loss through sickness? Here, Master Raṭṭhapāla, someone is
afflicted, suffering, and gravely ill. He considers thus: ‘I am
afflicted, suffering, and gravely ill. It is no longer easy for me
to acquire unacquired wealth.... This is called loss through
sickness. But Master Raṭṭhapāla now is free from illness....
And what is loss of wealth? Here, Master Raṭṭhapāla, someone is rich,
of great wealth, of great possessions. Gradually his wealth dwindles
away. He considers thus: Formerly I was rich, of great wealth....
But Master Raṭṭhapāla is the son of the leading clan in this same
Thullakoṭṭhita. Master Raṭṭhapāla has not undergone any loss of
wealth.
And what is loss of relatives? Here, Master Raṭṭhapāla, someone has
many friends and companions, kinsmen and relatives. Gradually those
relatives of his dwindle away. He considers thus: ‘Formerly I had
many friends and companions, kinsmen and relatives....'
Answering the king, Ratthapala more or less responds that worldly success and prosperity is overrated, and not worth chasing. It doesn't bring peace.
“Great king, there are four summaries of the Dhamma that have been
taught by the Blessed One who knows and sees, accomplished and fully
enlightened. Knowing and seeing and hearing them, I went forth from
the home life into homelessness. What are the four?
“‘Life in any world is unstable, it is swept away’: this is the
first summary of the Dhamma taught by the Blessed One who knows and
sees, accomplished and fully enlightened. Knowing and seeing and
hearing this, I went forth from the home life into homelessness.
“‘Life in any world has no shelter and no protector’: this is the
second summary of the Dhamma taught by the Blessed One who knows and
sees…
“‘Life in any world has nothing of its own; one has to leave all and
pass on’: this is the third summary of the Dhamma taught by the
Blessed One who knows and sees…
“‘Life in any world is incomplete, insatiate, the slave of
craving’: this is the fourth summary of the Dhamma taught by the
Blessed One who knows and sees…"
Personally, I see nothing wrong with somebody taking up the spiritual life simply because he is a "loser" in the "real world". Don't feel self-critical; don't feel lonely. There are a lot of "losers" in the "real world", almost all of whom cope with severely dysfunctional forms of escape (alcohol, drugs and whatnot). It is better to have "escapism" with something that is actually an escape, and more importantly, develop the ability to help others to truly escape, losers and successful people alike.
Remember that losers and unsuccessful people, suffering people, full of confusion and despondent from lack -- are the very reason that the Tathagata appeared into the world.
Don't let anybody, especially non-Buddhist, criticize you for following Buddhism for the wrong reason; they don't know any better. Grow yourself. Decide for yourself whether you "need" more material prosperity than you currently have, but don't be stressed or distraught about it. Practice skepticism towards ideas that aren't life-sustaining, and that weren't developed with your interest at heart. Choose good role models, people who actually have empathy and inner peace of mind -- not just the the temporary trappings of what passes for status or success. Learn what is beneficial for this life, and the hereafter, and follow that.