Their contextual use in the suttas is the same, as follows:
And what is right effort? Katamo ca, bhikkhave, sammāvāyāmo?
It’s when a mendicant generates enthusiasm, tries, makes an effort,
exerts the mind, and strives so that bad, unskillful qualities don’t
arise. Idha, bhikkhave, bhikkhu anuppannānaṃ pāpakānaṃ akusalānaṃ
dhammānaṃ anuppādāya chandaṃ janeti vāyamati vīriyaṃ ārabhati cittaṃ
And what is the faculty of energy? Katamañca, bhikkhave,
It’s when a mendicant lives with energy roused up for giving up
unskillful qualities and embracing skillful qualities. They’re strong,
staunchly vigorous, not slacking off when it comes to developing
skillful qualities. Idha, bhikkhave, ariyasāvako āraddhavīriyo
viharati akusalānaṃ dhammānaṃ pahānāya, kusalānaṃ dhammānaṃ
upasampadāya, thāmavā daḷhaparakkamo anikkhittadhuro kusalesu dhammesu.
They generate enthusiasm, try, make an effort, exert the mind, and
strive so that bad, unskillful qualities don’t arise. So anuppannānaṃ
pāpakānaṃ akusalānaṃ dhammānaṃ anuppādāya chandaṃ janeti vāyamati
vīriyaṃ ārabhati cittaṃ paggaṇhāti padahati
The word 'vāyāma' is derived from 'vi + ā + yam' where 'yam' or 'yāma' appears to mean 'restraint', 'don't do', 'reining in' or 'control'. One fluent in Pali will need to explain how 'vi + ā' modify 'yāma'.
The word 'viriya' I guess is derived from 'vīra + ya' and is said to literally mean 'state of a strong man'. 'Vīra' means 'strong', 'brave' or 'heroic' thus 'ya' must mean 'state'.
Therefore, merely guessing, 'viriya' appears to be a 'thing', namely, 'energy'; where as 'vāyāma' appears related to acting or related to a verb, i.e., possibly a 'verbal noun'. In other words, in the Pali above, there is the phrase 'vāyamati vīriyaṃ'. Here, 'vāyamati' (due to ending in 'ati') is a verb; where as 'vīriyaṃ' is a noun. Thus, the phrase 'vāyamati vīriyaṃ' appears to mean: 'to exercise energy' or 'to exert energy'.
In summary, 'vayama' appears related to the action of acting; where as 'viriya' appears related to the quality of acting. This said, I could be wrong, here, because I am merely guessing.