Maternal postpartum depression (PPD) has been found to be related to infant social withdrawal during mother–infant interaction, and this may spill over on infant interactive behavior in other social contexts and impact infant psychosocial development. We investigated whether PPD was associated with infant social withdrawal during interaction with a tester in a psychological test situation and whether infant social withdrawal in the test situation mediated the association between PPD and infant cognitive scores reported in a previous study. Participants were 28 PPD dyads and 41 control dyads. We assessed infant social behavior and cognitive development with the Alarm Distress Baby Scale and the cognitive scale of the Bayley Scales of Infant and Toddler Development, third edition, at four months. More symptoms of maternal depression were associated with more infant social withdrawal. The association between maternal depressive symptoms and cognitive scores was at most partially mediated by infant social withdrawal in the test situation (<29.6%). Our results add to the existing literature on the effects of PPD on infant social behavior in other contexts than the one constituted by the mother. More research is needed to shed light on the mechanisms through which PPD impacts infant cognitive development.