A reddit commenter found the sutta I was thinking of, which is SN 22.80 (quoted below).
“And although this son of a good family has gone forth in this way, he
is covetous, with strong passion for sensual desires, with a mind of
ill will, of corrupt resolves, his mindfulness muddled, unalert,
unconcentrated, his mind distracted, loose in his sense faculties.
Just as a log from a funeral pyre, burning at both ends, smeared with
excrement in the middle, fills no use as timber either in the village
or in the wilderness: I speak of this person with this comparison. He
has missed out on the enjoyments of the householder, and yet does not
fulfill the goal of the contemplative life.
But there probably are more references too, I'll post a summary when we collect them all.