The dynamics of body movements are often driven by large and intricate low-level interactions involving various body parts. These dynamics are part of an underlying data generation process and incorporating them into data-driven activity recognition systems has the potential to enhance their robustness and data-efficiency. In this paper, we propose to model the underlying data generation process and use it to constrain training of simpler learning models via sample selection. We report on experiments conducted on the SHL dataset featuring a sensor-rich environment. Compared to the basic setting, our approach achieves substantial improvements, by simultaneously reducing the number of required data sources by one-half.