Some of the most enduring social connections begin when infants first recognize their caregivers, memories that form the basis of many family relationships. It remains unknown whether these early social memories persist into adulthood in mice and, if so, which brain regions support them. Here we show that mice form memories of their mother within days after birth and that these memories persist into adulthood. Pups display greater interest in the mother than in an unfamiliar dam before weaning, after which this preference reverses. Inhibition of CA2 neurons in the pup temporarily blocks the ability to discriminate between the mother and an unfamiliar dam, whereas doing so in adulthood prevents the formation of short-term memories about conspecifics, as well as social discrimination related to long-term memories of the mother. These results suggest that the CA2 supports memories of the mother during infancy and adulthood with a developmental switch in social preference.