In the present study, we decomposed between- and within-person effects and examined moderators of the longitudinal physical activity-cognition association. Participants (N = 1722) were drawn from the Betula study and we included four waves of data across 15 years. Bayesian multilevel modeling showed that self-reported physical activity did not predict changes in cognitive function. Physical activity positively predicted cognitive performance at baseline, and the relations were stronger for more active (compared to less active) older adults. Physical activity had a positive within-person effect on cognitive function. The within-person effect of physical activity on episodic memory recall was stronger for participants who on average engaged in less physical activity. The within-person effect on verbal fluency was stronger for participants with more education. Our results suggest that preserving cognitive functioning in old age might be more a matter of what you do in old age than reflecting what you did earlier in life.