The nested planetary gear train, which has two integrated single-stage planetary gearsets, is one of the newly developed compound gear train that has been successfully applied to the automobile transmissions. In the current study, a certain type of gear fault in the nested gear train, ungrounded pinion, is investigated using a non-destructive approach monitoring its vibration levels. A novel experimental test stand with open and vertical setup has been designed to collect the vibrational data by mounting the accelerometer directly to the gear clutches. Each of the two layers of the compound gear was tested separately. The measured vibrational data were processed with several signal processing techniques, which includes (a) frequency spectrum analysis, (b) time synchronous averaging (TSA) and (c) modulation sideband analysis. The experimental results show that the existence of the ungrounded pinion can be identified with the frequency spectrum analysis of the vibrational data. In addition, the modulation sidebands are also modeled using a modified version of the traditional technique of physical signal modeling. It is shown that the relative phase of the planet and the meshing vibration strength changed by the unground gear is the critical factor for determining the modulation sideband behavior. In addition, the location of the ungrounded pinion can also be determined by the time history processed by TSA.