From what we can guess from the clues in the suttas, when Buddha started his quest for Enlightenment, there were many competing spiritual teachings in existence (such as e.g. the Jain's) claiming to be The Truth, which were postulating theories about things that were however impossible to see for oneself and verify by the direct testing in the here and now.
In contrast with that, Buddha's approach has been to work with the immediately observable reality. Buddha insisted that True Dharma must have a firm foundation, something that can't be reasonably denied. This approach led the Buddha to the perspective of phenomenology. In this perspective, the seeker of Enlightenment starts with "the given", which is his own direct experience - the five senses, the experience of suffering etc., and then goes on to discover "how things work" from the first-hand, not through blind faith or mediation of a priest.
This is why, when the students of other sects were trying to pull the Buddha in the direction of metaphysics, by asking about such speculative concepts like "The All" -- which is usually assumed to mean the Universe -- the Buddha insisted on staying firmly grounded in the directly observable phenomenology of our five senses plus mind.
So The All means all that can ever be experienced by a sentient being, it is sentient being's personal totality of existence. Even though this phenomenological All seems radically simpler than the All we know from natural sciences, it is equally all-encompassing and has a huge advantage over the metaphysical theories in being directly observable, describable, and verifyable by anyone.
If someone were to try and deny the primacy of the phenomenological All, and postulate some other All, as for example the totality of everything that objectively exists, or perhaps the totality of everything that ever existed and will exist, or maybe the totality of everything that potentially may exist, one would get into infinite speculative arguments about philosophical categories of existence and non-existence, and other categories lying outside of immediate experience, which would lead to endless frustrating objections, clarifications, and debates that would most likely end up in exhaustion and disappointment rather than in Peace of Liberation.