The classification scheme for high redshift galaxies is complex at the present time, with simple colour selection criteria, resulting in ill-defined properties regarding stellar mass and star formation rate for these distant galaxies. The goal of this work is to investigate the properties of different classes of high-z galaxies, focusing in particular on the Stellar Masses of LBGs, DRGs and BzKs, in order to derive their contribution to the total mass budget of the distant Universe. We have used the GOODS-MUSIC catalog, containing ~3000 Ks-selected (~10000 z-selected) galaxies with multi-wavelength coverage extending from the U band to the Spitzer 8 micron band, with spectroscopic or accurate photometric redshifts. We have selected samples of BM/BX/LBGs, DRGs and BzK galaxies, discussed the overlap and the limitations of these criteria, which can be overcome with a selection criterion based on physical parameters. We have then measured the stellar masses of these galaxies and computed the Stellar Mass Density (SMD) for the different samples up to redshift ~4. We show that the BzK-PE criterion is not optimal to select early type galaxies at the faint end. BzK-SF, on the other hand, is highly contaminated by passively evolving galaxies at red z-Ks colours. We find that LBGs and DRGs contribute almost equally to the global SMD at z>2 and in general that star forming galaxies form a substantial fraction of the universal SMD. Passively evolving galaxies show a strong negative density evolution from redshift 2 to 3, indicating that we are witnessing the epoch of mass assembly of such objects.