In the cycle of existence, there is no satisfaction in sensual desires, which are like honey on a razor's edge. ~Shantideva, Bodhisattvacharyavittara
Celibacy is certainly good but it isn't easy in the modern world, like being healthy is good, but being healthy in the middle of a pandemic isn't easy.
We live in a hyper-sexualized world today where even cars need sexy lines to sell.
There's a Vietnamese Zen saying that a tiger is safe in the forest, but at risk when he enters the village. Similarly, a Yogi is safest when he's away from sense triggers: sex, money, luxurious food, liquor, bad companions - all of which can bring up base desires and aversions.
The Thai forest monks, Himalayan tapas-yogis of Hinduism, desert fathers of Christianity all sought relief from such triggers by avoiding worldly contact.
In the Buddha's age exposure to sense desires would have been fewer, and solitude would have been easier to find (no TV, social media, electricity, mobile phones, traffic). Yet, he recommended seclusion, because sense desire is a furious fire that's hard to put out once started. It's basic mental hygiene and sanitisation.
If we are going to perform surgery on our minds (i.e.) serious meditation, then we don't want to do it in an unhygienic condition. Retreats are designed to offer hygienic conditions for the mind.
The practice of "right livelihood" ensures there are few opportunities to tells lies, act dishonestly or violently. Similarly, one must find a place to live where sexual energy is not omnipresent to learn to be celibate, otherwise frustration will set in.
Such frustration arising from ignorance of causes and conditions might then turn practitioners away from the whole of Dhamma. Some teachers might have this in mind while recommending celibacy only on retreats.
It's not a universal rule, to a determined one no wall is too high, but parents often tell their little kids to not climb tall structures until they are older.
I am independent of any sectarian affiliation, so my answer draws from my interpretation of personal experience guided by Theravada, Zen, Hinduism (several kinds), Christianity, modern science and humanities etc.