High aerobic fitness, more moderate to vigorous physical activity (MVPA) and less sedentary behavior (SED) have all been suggested to promote cognitive functions, but it is unclear whether they are independent predictors of specific cognitive domains. This study aimed to investigate to what extent aerobic fitness MVPA and SED are independently associated with cognitive performance among middle-aged Swedish adults. We acquired device-based measures of aerobic fitness, cognitive performance and percent daily time spent in MVPA and SED in Swedish adults (n = 216; 54-66 years old). Aerobic fitness was associated with better performance at one out of two tests of speed/attention and one out of four tests of executive attention, and with worse performance at one of seven tests of memory. Increasing %MVPA was associated with better performance at one out of seven tests of memory and two out of three tests of verbal ability, whereas increasing %SED was associated with better performance at all four tests of executive attention and four out of seven tests of memory. These findings suggest that aerobic fitness, %MVPA and %SED are partly independent correlates of cognitive performance. To fully understand the association between SED and performance at several tests of cognitive function, future investigations might attempt to investigate intellectually engaging SED (such as reading books) separately from mentally undemanding SED (such as watching TV).