Actually, there's a lot of factors regarding this.
The Monks started to liberate the buddha's teaching.
The Buddhism started to decline after the 4th buddhist council of India. Unlike King Asoka, King Kanishka of Kushan Empire, not keeping the precepts and still going on wars with other empires. He chose the most Sangha in the community back then the Sarvastivadin-Mahayana.
The last was because of the invasion of Turkic-muslim-Mamluk Dynasty which ended the Most Enormous Buddhist University, Nalanda.
As what Buddha said in DN16 Mahaparinibbana Sutta, there are 7 things x 7, which prevent decline in the community.
Now at that time venerable Ānanda was stood behind the Gracious One
fanning the Gracious One. Then the Gracious One addressed venerable
Ānanda, saying:
1) “Have you heard, Ānanda: ‘The Vajjians assemble regularly and
assemble frequently?’” “I have heard this, reverend Sir: ‘The Vajjians
assemble regularly and assemble frequently.’” “For as long, Ānanda, as
the Vajjians will assemble regularly and assemble frequently surely
growth, Ānanda, is to be expected for the Vajjians not decline.
2) Have you heard, Ānanda: ‘The Vajjians assemble unanimously, rise
unanimously, and carry out their Vajjian duties unanimously?’” “I have
heard this, reverend Sir, that the Vajjians assemble unanimously, rise
unanimously, and carry out their Vajjian duties unanimously.” “For as
long, Ānanda, as the Vajjians will assemble unanimously, rise
unanimously, and carry out their Vajjian duties unanimously, surely
growth, Ānanda, is to be expected for the Vajjians not decline.
3) Have you heard, Ānanda: ‘The Vajjians do not establish new laws
that were not established, or cut off old laws that were established,
and carry on with such laws as were accepted in the Ancient Vajjian
Constitution?’” “I have heard this, reverend Sir: ‘The Vajjians do not
establish new laws that were not established, or cut off old laws that
were established, and they carry on with such laws as were accepted in
the Ancient Vajjian Constitution.’” “For as long, Ānanda, as the
Vajjians do not establish new laws that were not established, or cut
off old laws that were established, and they carry on with such laws
as were accepted in the Ancient Vajjian Constitution surely growth,
Ānanda, is to be expected for the Vajjians not decline.
4) Have you heard, Ānanda: ‘The Vajjians honour the elders of the
Vajjians, respect, revere, worship and think them worth listening
to?’” “I have heard this, reverend Sir: ‘The Vajjians honour the
elders of the Vajjians, respect, revere, worship and think them worth
listening to.’” “For as long, Ānanda, as the Vajjians will honour the
elders of the Vajjians, respect, revere, worship and think them worth
listening to, surely growth, Ānanda, is to be expected for the
Vajjians not decline.
5) Have you heard, Ānanda: ‘The Vajjians do not coerce and force their
women and girls to dwell with them against their will?’” “I have heard
this, reverend Sir: ‘The Vajjians do not coerce and force their women
and girls to dwell with them against their will.’” “For as long,
Ānanda, as the Vajjians will not coerce and force their women and
girls to dwell with them against their will, surely growth, Ānanda, is
to be expected for the Vajjians not decline.
6) Have you heard, Ānanda: ‘The Vajjians honour the Vajjian shrines
amongst the Vajjians, both within and without the city, respect,
revere, and worship them, and do not allow the righteous sacrifices
that were formerly given, formerly made, to be neglected?’” “I have
heard this, reverend Sir: ‘The Vajjians honour the Vajjian shrines
amongst the Vajjians, both within and without the city, respect,
revere, and worship them, and do not allow the righteous sacrifices
that were formerly given, formerly made, to be neglected.” “For as
long, Ānanda, as the Vajjians will honour the Vajjian shrines amongst
the Vajjians, both within and without the city, respect, revere, and
worship them, and do not allow the righteous sacrifices that were
formerly given, formerly made, to be neglected surely growth, Ānanda,
is to be expected for the Vajjians not decline.
7) Have you heard, Ānanda: ‘The Vajjians have made good arrangements
in regard to the lawful protection, safety, and guarding of the Worthy
Ones, so that Worthy Ones in the future can enter the realm, and
having entered the Worthy Ones can live comfortably in the realm?” “I
have heard this, reverend Sir: ‘The Vajjians have made good
arrangements in regard to the lawful protection, safety, and guarding
of the Worthy Ones, and that the Worthy Ones in the future can enter
the realm, and having entered the Worthy Ones can live comfortably in
the realm.” “For as long, Ānanda, as the Vajjians will make good
arrangements in regard to the lawful protection, safety, and guarding
of the Worthy Ones, and that the Worthy Ones in the future can enter
the realm, and having entered, the Worthy Ones can live comfortably in
the realm, surely growth, Ānanda, is to be expected for the Vajjians
not decline.”
Then the Gracious One addressed the Magadhan chief minister the
brahmin Vassakāra, saying: “At one time, brahmin, I was living near
Vesālī near to the Sārandada Shrine and there I taught the Vajjians
these seven things which prevent decline. For as long, brahmin, as the
Vajjians maintain these seven things which prevent decline, and the
Vajjians agree with these seven things which prevent decline, surely
growth, brahmin, is to be expected for the Vajjians not decline.”
After this was said, the Magadhan chief minister the brahmin Vassakāra
addressed the Gracious One, saying: “If the Vajjians, dear Gotama,
were endowed with even one or the other of these seven things which
prevent decline, surely growth is to be expected not decline, what to
say about having seven things which prevent decline? The Vajjians
cannot be overcome, dear Gotama, by the Magadhan King Ajātasattu, the
son of Lady Wisdom, by war, but only through diplomacy, or through the
breaking of an alliance. And now, dear Gotama, we shall go, as we have
many duties, and there is much which ought to be done.”
“Now is the time, brahmin, for whatever you are thinking.” Then the
Magadhan chief minister the brahmin Vassakāra, after greatly rejoicing
and gladly receiving this word of the Gracious One, rose from his seat
and departed.
3: Seven Things which Prevent Decline in the Community (1–7)
Then the Gracious One, not long after the Magadhan chief minister, the
brahmin Vassakāra, had gone, addressed venerable Ānanda, saying: “Go,
Ānanda, and whatever monks there are living in dependence on Rājagaha,
assemble them in the Attendance Hall.”
“Very well, reverend Sir,” said venerable Ānanda, and after replying
to the Gracious One, and assembling whatever monks there were living
in dependence on Rājagaha in the Attendance Hall, he approached the
Gracious One, and after approaching and worshipping the Gracious One,
he stood on one side.
While standing on one side venerable Ānanda said this to the Gracious
One: “The Community of monks have assembled, reverend Sir, now is the
time, Gracious One, for whatever you are thinking.”
Then the Gracious One after rising from his seat approached the
Attendance Hall, and after approaching he sat down on the prepared
seat. While sitting the Gracious One addressed the monks, saying:
“I will teach you seven things which prevent decline, listen to it,
apply your minds well, and I will speak.” “Very well, reverend Sir,”
those monks replied to the Gracious One, and the Gracious One said
this:
1) “For as long, monks, as the monks will assemble regularly and
assemble frequently, surely growth, monks, is to be expected for the
monks, not decline.
2) For as long, monks, as the monks will assemble unanimously, rise
unanimously, and carry out their Community duties unanimously, surely
growth, monks, is to be expected for the monks, not decline.
3) For as long, monks, as the monks do not establish new laws that
were not established, or cut off old laws that were established, and
they carry on with such training-rules as have been accepted, surely
growth, monks, is to be expected for the monks, not decline.
4) For as long, monks, as the monks will honour the elder monks, those
of long-standing, a long time gone-forth, the Fathers of the
Community, the Leaders of the Community, respect, revere, worship and
think them worth listening to, surely growth, monks, is to be expected
for the monks, not decline.
5) For as long, monks, as the monks will not come under the influence
of craving which has arisen for continued existence, surely growth,
monks, is to be expected for the monks, not decline.
6) For as long, monks, as the monks will have desire for forest
dwellings, surely growth, monks, is to be expected for the monks, not
decline.
7) For as long, monks, as the monks individually will attend to the
ways of mindfulness, so that their fellow celibates, who are
well-behaved, in the future can come, and having come to their fellow
celibates, who are well-behaved, can live comfortably, surely growth,
monks, is to be expected for the monks, not decline.
For as long, monks, as the monks will maintain these seven things
which prevent decline, and the monks will agree with these seven
things which prevent decline, surely growth, monks, is to be expected
for the monks, not decline.
4: Seven Further Things which Prevent Decline in the Community (8–14)
I will teach you a further seven things which prevent decline, listen
to it, apply your minds well, and I will speak.”
“Very well, reverend Sir,” those monks replied to the Gracious One,
and the Gracious One said this:
1) “For as long, monks, as the monks are not devoted to the pleasure
in work, the delight in work, and are not attached to work, surely
growth, monks, is to be expected for the monks, not decline.
2) For as long, monks, as the monks are not devoted to the pleasure in
speech, the delight in speech, and are not attached to speech, surely
growth, monks, is to be expected for the monks, not decline.
3) For as long, monks, as the monks are not devoted to the pleasure in
sleep, the delight in sleep, and are not attached to sleep, surely
growth, monks, is to be expected for the monks, not decline.
4) For as long, monks, as the monks are not devoted to the pleasure of
company, the delight in company, and are not attached to company,
surely growth, monks, is to be expected for the monks, not decline.
5) For as long, monks, as the monks do not have evil wishes, do not go
under the influence of evil wishes, surely growth, monks, is to be
expected for the monks, not decline.
6) For as long, monks, as the monks do not have wicked friends, do not
have wicked companions, do not have wicked comrades, surely growth,
monks, is to be expected for the monks, not decline.
7) For as long, monks, as the monks do not achieve only mundane or
incomplete attainment, surely growth, monks, is to be expected for the
monks, not decline.
For as long, monks, as the monks will maintain these seven things
which prevent decline, and the monks will agree with these seven
things which prevent decline, surely growth, monks, is to be expected
for the monks, not decline.
5: Seven Further Things which Prevent Decline in the Community (15–21)
I will teach you a further seven things which prevent decline, listen
to it, apply your minds well, and I will speak.”
“Very well, reverend Sir,” those monks replied to the Gracious One,
and the Gracious One said this:
1) “For as long, monks, as the monks will have faith, surely growth,
monks, is to be expected for the monks, not decline.
2) For as long, monks, as the monks will have a conscientious mind,
surely growth, monks, is to be expected for the monks, not decline.
3) For as long, monks, as the monks will have a sense of shame, surely
growth, monks, is to be expected for the monks, not decline.
4) For as long, monks, as the monks will be learned, surely growth,
monks, is to be expected for the monks, not decline.
5) For as long, monks, as the monks will be strenuous, surely growth,
monks, is to be expected for the monks, not decline.
6) For as long, monks, as the monks will attend to mindfulness, surely
growth, monks, is to be expected for the monks, not decline.
7) For as long, monks, as the monks will possess wisdom, surely
growth, monks, is to be expected for the monks, not decline.
For as long, monks, as the monks will maintain these seven things
which prevent decline, and the monks will agree with these seven
things which prevent decline, surely growth, monks, is to be expected
for the monks, not decline.
6: Seven Further Things which Prevent Decline in the Community (22–28)
I will teach you a further seven things which prevent decline, listen
to it, apply your minds well, and I will speak.”
“Very well, reverend Sir,” those monks replied to the Gracious One,
and the Gracious One said this:
1) “For as long, monks, as the monks will develop the factor of
Perfect Awakening that is Mindfulness, surely growth, monks, is to be
expected for the monks, not decline.
2) For as long, monks, as the monks will develop the factor of Perfect
Awakening that is Investigation of the nature of things, surely
growth, monks, is to be expected for the monks, not decline.
3) For as long, monks, as the monks will develop the factor of Perfect
Awakening that is Energy, surely growth, monks, is to be expected for
the monks, not decline.
4) For as long, monks, as the monks will develop the factor of Perfect
Awakening that is Rapture, surely growth, monks, is to be expected for
the monks, not decline.
5) For as long, monks, as the monks will develop the factor of Perfect
Awakening that is Calm, surely growth, monks, is to be expected for
the monks, not decline.
6) For as long, monks, as the monks will develop the factor of Perfect
Awakening that is Concentration, surely growth, monks, is to be
expected for the monks, not decline.
7) For as long, monks, as the monks will develop the factor of Perfect
Awakening that is Equanimity, surely growth, monks, is to be expected
for the monks, not decline.
For as long, monks, as the monks will maintain these seven things
which prevent decline, and the monks will agree with these seven
things which prevent decline, surely growth, monks, is to be expected
for the monks, not decline.
7: Seven Further Things which Prevent Decline in the Community (29–35)
I will teach you a further seven things which prevent decline, listen
to it, apply your minds well, and I will speak.”
“Very well, reverend Sir,” those monks replied to the Gracious One,
and the Gracious One said this:
1) “For as long, monks, as the monks will develop the perception of
impermanence, surely growth, monks, is to be expected for the monks,
not decline.
2) For as long, monks, as the monks will develop the perception of
non-self, surely growth, monks, is to be expected for the monks, not
decline.
3) For as long, monks, as the monks will develop the perception of the
unattractive, surely growth, monks, is to be expected for the monks,
not decline.
4) For as long, monks, as the monks will develop the perception of
danger, surely growth, monks, is to be expected for the monks, not
decline.
5) For as long, monks, as the monks will develop the perception of
giving up, surely growth, monks, is to be expected for the monks, not
decline.
6) For as long, monks, as the monks will develop the perception of
dispassion, surely growth, monks, is to be expected for the monks, not
decline.
7) For as long, monks, as the monks will develop the perception of
cessation, surely growth, monks, is to be expected for the monks, not
decline.
For as long, monks, as the monks will maintain these seven things
which prevent decline, and the monks will agree with these seven
things which prevent decline, surely growth, monks, is to be expected
for the monks, not decline.
8: Six Things which Prevent Decline in the Community (36–41)
I will teach you a further six things which prevent decline, listen to
it, apply your minds well, and I will speak.”
“Very well, reverend Sir,” those monks replied to the Gracious One,
and the Gracious One said this:
1) “For as long, monks, as the monks with friendly actions by way of
the body will serve their fellow celibates, both in public and in
private, surely growth, monks, is to be expected for the monks, not
decline.
2) For as long, monks, as the monks with friendly actions by way of
speech will serve their fellow celibates, both in public and in
private, surely growth, monks, is to be expected for the monks, not
decline.
3) For as long, monks, as the monks with friendly actions by way of
the mind will serve their fellow celibates, both in public and in
private, surely growth, monks, is to be expected for the monks, not
decline.
4) For as long, monks, as the monks in regard to those righteous
gains, received in accordance with the Teaching—whatever amount has
been received in the bowl—will divide and share such gains with those
who are virtuous, fellow celibates, and share them in common, surely
growth, monks, is to be expected for the monks, not decline.
5) For as long, monks, as the monks are endowed with those virtues
which are unbroken, faultless, unspotted, unblemished, productive of
freedom, praised by the wise, not clung to, leading to concentration,
and will live endowed with virtue amongst their fellow celibates who
themselves possess such virtue, both in public and in private, surely
growth, monks, is to be expected for the monks, not decline.
6) For as long, monks, as the monks are endowed with that which is
Ariyan View, which leads out, which leads to the complete destruction
of suffering for one who acts thus, and will live endowed with Right
View amongst those who themselves possess such Right View, both in
public and in private, surely growth, monks, is to be expected for the
monks, not decline.
For as long, monks, as the monks will maintain these six things which
prevent decline, and the monks will agree with these six things which
prevent decline, surely growth, monks, is to be expected for the
monks, not decline.”
There also the Gracious One, while living near Rājagaha on the
Vultures’ Peak Mountain, spoke frequently to the monks about the
Teaching, saying:
“Such is virtue, such is concentration, such is wisdom, when virtue is
well-developed it yields great fruit and brings great advantages in
regard to concentration, when concentration is well-developed it
yields great fruit and brings great advantages in regard to wisdom,
when wisdom is well-developed the mind is completely liberated from
the pollutants, that is to say: the pollutant of sensuality, the
pollutant of craving for continued existence, the pollutant of
ignorance.”
The declination of buddhism was because of breaking these.
Hopefully this helps. If you want to know more about this, you can try to read this book called "The concise history of buddhism".