First I think that it is a good idea to examine title of this post, and point out that there is no real "should" that we encounter in Buddhism, there are encouragements, but no commandments; or more politely, because of the truth of emptiness, or lack of inherent meaning and essence, whenever we say "should," in order for it to be a valid statement, we have to add a contextual qualifier. So, "Should we do X?" always becomes, "If we want X, should we do Y?" or something to that effect. Given that, your question lacks a qualifier, you asked, "Should we do X?" without offering a "Y". So, why?
Now, about time. I would offer that, in my understanding, time is, indeed a conditioned phenomena, it is not absolute (there is no absolute time, "it" flows at different rates in the universe), it relys on other phenomena for its existence (such as space, therefore objects, gravity, ect.) Because of this, time is compounded. However, you cannot give or take it in any real sense. It is intangible, unlike other compounded phenomena, a chair for instance, it is purely abstract with no appearance to lock it into our experience with, it is not a real thing in the same way that a chair is. The best we can do is try to record and measure the passing of events with a chronometer, but there is no thing called "time" that we can actually measure, we just call the passing of events that, nominally inferring time into existence, in retrospect, or in anticipation.
The metaphor of "giving time" is merely a convention that we use for service, in the certain light of sacrificing our own agenda. I would, again rephrase the question in order to reflect the implied sacrifice of personal agenda.
So taking both points, the lack of qualifier in the first, and the inferential nature of the statement that calls for clarity in the second, I would suggest restating the question like this, "If we want to hasten our own liberation, should we practice service, sacrificing our own agenda of self centered goal seeking in order to be of benefit to others?"
So... When we put ourselves into service, when we put others, and the agenda of others, before ourselves and our own agenda, we undermine the conceptual self centered impulse. We undermine our tendency towards an ego-centric agenda. This is very helpful to our liberation, and it makes the world a better, happier place, so why not? It can eliminate negative karma, crate good karma, help see through the Samsaric delusion, and so forth. In this sense, it seems that being of service is a good personal agenda, as it has the potential to liberate us from our suffering.
Interestingly, this seems to leave us in a bind, how can I be of service to you, getting past my own self serving agenda, when that creates a self serving agenda as soon as I realize that I am doing it to get past my own suffering?
Well, this is where intention comes in. How compassionate are we being? The Dalai Lama talks about wise vs. foolish selfishness. Wise selfishness is realizing this, and being of service, giving back to the community because you know it will benefit you, and this is better than the alternative... ignoring this and being a glutton. Regardless, it is simple enough to say that the more your heart is in it, the more benefit it will be to you, and the more you practice it, the more your heart will be in it.
When we realize that helping people really means empowering them to overcome their own suffering, we become little "helper-makers", also known as teachers. In this way, we can realize selflessness giving rise to selflessness, and so forth.
TL;DR It is highly encouraged that we learn to put others before ourselves, and learn to be of true service.