The androgen receptor (AR) ligand-binding domain (LBD) binds FXXLF motifs, present in the AR N-terminal domain and AR-specific cofactors, and some LXXLL motifs of nuclear receptor coactivators. We demonstrated that in the context of the AR FXXLF motif many different amino acid residues at positions +2 and +3 are compatible with strong AR LBD interaction, although a preference for E at +2 and K or R at +3 was found. Pairwise systematic analysis of F/L swaps at +1 and +5 in FXXLF and LXXLL motifs showed: (1) F to L substitutions in natural FXXLF motifs abolished AR LBD interaction; (2) binding of interacting LXXLL motifs was unchanged or increased upon L to F substitutions; (3) certain non-interacting LXXLL motifs became strongly AR-interacting FXXLF motifs, (4) while other non-binders remained unaffected by L to F substitutions. All FXXLF motifs, but not the corresponding LXXLL motifs, displayed a strong preference for AR LBD. Progesterone receptor LBD interacted with some FXXLF motifs, albeit always less efficiently than corresponding LXXLL motifs. AR LBD interaction of most FXXLF and LXXLL peptides depended on classical charge clamp residue K720, whereas E897 was less important. Other charged residues lining the AR coactivator-binding groove, K717 and R726, modulated optimal peptide binding. Interestingly, these four charged residues affected binding of individual peptides independent of an F or L at +1 and +5 in swap experiments. In conclusion, F residues determine strong and selective peptide interactions with AR. Sequences flanking the core motif determine the specific mode of FXXLF and LXXLL interactions.