Older adults show relatively minor age-related decline in memory for single items, while their memory for associations is markedly reduced. Inter-individual differences in memory function in older adults are substantial but the neurobiological underpinnings of such differences are not well understood. In particular, the relative importance of inter-individual differences in the medio-temporal lobe (MTL) and the lateral prefrontal cortex (PFC) for associative and item recognition in older adults is still ambiguous. We therefore aimed to first establish the distinction between inter-individual differences in associative memory (recollection-based) performance and item memory (familiarity-based) performance in older adults and subsequently link these two constructs to differences in cortical thickness in the MTL and lateral PFC regions, in a latent structural equation modelling framework. To this end, a sample of 160 older adults (65-75 years old) performed three intentional item-associative memory tasks, of which a subsample (n = 72) additionally had cortical thickness measures in MTL and PFC regions of interest available. The results provided support for a distinction between familiarity-based item memory and recollection-based associative memory performance in older adults. Cortical thickness in the ventro-medial prefrontal cortex was positively correlated with associative recognition performance, above and beyond any relationship between item recognition performance and cortical thickness in the same region and between associative recognition performance and brain structure in the MTL (parahippocampus). The findings highlight the relative importance of the ventromedial prefrontal cortex in allowing for intentional recollection-based associative memory functioning in older adults.