In line with contemporary constructivist pedagogies, students are frequently expected to learn through interaction in physical education (PE). There is a relatively sophisticated body of literature focusing on learning in groups, peer teaching, and cooperative learning. Current research has not, however, focused on how the body is implicated in interactional learning. This is surprising given that much learning in PE is expected to take place in the physical domain. The aim of this paper is to contribute to current theorizing by examining social interactions in PE practice. By drawing on symbolic interactionist theory, we put forward a framework for considering how inter-student interactions occur in a multimodal sense. Key ideas relate to (1) the sequential organization of interactions; (2) the ways in which semiotic resources in different fields are used to elaborate each other; (3) the importance of interpretation as a driver of interaction; (4) the creation of local environments in which participants attend to and work together within a shared world of perception; and (5) the influence of material environments on social interaction. The specific concepts employed are epistemic ecology, epistemic position, and learning trajectory. The paper includes observational data from an investigation of learning in Swedish PE to demonstrate the explanatory power and limitations of the theoretical tenets presented. The paper is concluded with practical implications of understanding group work in a multimodal manner.