From the Ayacana Sutta (SN 6.1): the Buddha thought that it would be too hard to teach the dhamma and considered not doing so.
Then, while he (Buddha) was alone and in seclusion, this line of thinking arose
in his awareness: "This Dhamma that I have attained is deep, hard to
see, hard to realize, peaceful, refined, beyond the scope of
conjecture, subtle, to-be-experienced by the wise. .....
And if I were to teach
the Dhamma and if others would not understand me, that would be
tiresome for me, troublesome for me."
Reading on, Brahma Sahampati requested the Buddha to teach the dhamma, saying that there would be those (with little dust in their eyes) who would understand it.
Then the Blessed One, having understood Brahma's invitation, out of
compassion for beings, surveyed the world with the eye of an Awakened
One. As he did so, he saw beings with little dust in their eyes and
those with much, those with keen faculties and those with dull, those
with good attributes and those with bad, those easy to teach and those
hard, some of them seeing disgrace and danger in the other world. Just
as in a pond of blue or red or white lotuses, some lotuses — born and
growing in the water — might flourish while immersed in the water,
without rising up from the water; some might stand at an even level
with the water; while some might rise up from the water and stand
without being smeared by the water — so too, surveying the world with
the eye of an Awakened One, the Blessed One saw beings with little
dust in their eyes and those with much, those with keen faculties and
those with dull, those with good attributes and those with bad, those
easy to teach and those hard, some of them seeing disgrace and danger
in the other world.
So, from the Buddha's perspective, not many would be qualified to truly understand the dhamma when he first taught it. This would explain why it does not become wildly popular in his time.
Like some other religions, Buddhism got adopted by the masses due to being endorsed by the rulers who have accepted it.