Barry Fitton (66) is one of three alleged victims who claim the MP abused them at a care home during the 1960s.
In an interview with a Manchester newspaper, Mr Fitton accused Smith of slapping and stroking his naked buttocks and fondling him when he was 15.
He said when he first turned up at a boys' hostel in Rochdale he was ordered to take a medical examination.
"I thought it would be a doctor, but it was Cyril. He told me to take my pants down and he started to fondle me."