In a paper that appeared in the journal Biology Letters, the researchers found that women tended to make that judgement based on how masculine-looking the man was.
"Women's ratings of unfaithfulness showed small-moderate, significant correlations with measures of actual infidelity," wrote the team, led by Gillian Rhodes at the ARC Centre of Excellence in Cognition and its Disorders at the University of Western Australia in Perth.
"More masculine-looking men (were) rated as more probable to be unfaithful and having a sexual history of being more unfaithful."
Attractiveness was not a factor in the women making the link.
In the study, 34 men and 34 women were shown colour photographs of 189 Caucasian adult faces and asked to rate them for faithfulness.
The researchers compared their answers to the self-reported sexual histories of the 189 individuals and found that the women participants were better able to tell who was faithful and who was not.
"We provide the first evidence that faithfulness judgements, based solely on facial appearance, have a kernel of truth," they wrote in the paper.
Men, on the other hand, seemed to have no clue.
They tended to perceive attractive, feminine women to be unfaithful, when there was no evidence that they were, the scientists noted.
Faithfulness is seen as important in the context of sexual relationships and mate choice, the scientists wrote in the paper.
Men with unfaithful partners risk raising another man's child, while women with unfaithful partners risk losing some, or even all, parental and other resources to competitors.