Social recognition is an essential skill for the expression of appropriate behaviors towards conspecifics in most social species. Several studies point to oxytocin (OT) and arginine-vasopressin (AVP) as key mediators of social recognition in males and females. However, sex differences in social cognitive behaviors highlight an important interplay between OT, AVP and the sex steroids. Estrogens facilitate social recognition by regulating OT action in the hypothalamus and of OT receptor in the medial amygdala (MeA). The role of OT in these brain regions seems to be essential for social recognition (SR) in both males and females. Conversely, social recognition in male rats and mice is more dependent on AVP release in the lateral septum (LS) than in females. The AVP system comprises a series of highly sexually dimorphic brain nuclei, including the bed nucleus of the stria terminalis (BNST), the amygdala and the LS. Various studies suggest that testosterone and its metabolites, including estradiol, influence SR in males by modulating the activity of the AVP at V1a receptor. Intriguingly, both estrogens and androgens can affect social recognition very rapidly, through non-genomic mechanisms. In addition, the androgen metabolites, namely 3α-diol and 3β-diol, may also have an impact on social behaviors either by interacting with the estrogen receptors or through other mechanisms. Overall, the regulation of OT and AVP by sex steroids fine tunes social recognition and the behaviors that depend upon it (e.g., social bond, hierarchical organization, aggression) in a sex dependent manner. Elucidating the sex dependent interaction between sex steroids and neuroendocrine systems is essential to understand sex differences in the normal and abnormal expression of social behaviors.