That (above comment) said, there are numerous studies of what is called Open Monitoring (OM) vs Focused Attention (FA) meditation within the "mindfulness" category, for example the paper Attention regulation and monitoring in meditation, which first defined OM and FA. FA is pretty clearly related to at least some aspects of shamatha, and OA has sometimes been equated to "mindfulness" or even "vipassana", though both characterizations are in serious dispute (see below). So in that (complicated and quite likely misleading) sense, there are studies of shamatha vs vipassana, which you can find by looking up citations to the above paper.
The only study I know of comparing (parts of) two full and contrasting systems of meditation is the recent Arousal vs. Relaxation: A Comparison of the Neurophysiological and Cognitive Correlates of Vajrayana and Theravada Meditative Practices. Interestingly, that paper discusses FA and OM (or as it is called there, distributed attention) and concludes that the distinction is neither coherent from a phenomenological meditation standpoint nor supported by neuroscientific studies.