"He fooled everybody," he said.
However he told how he was constantly in adult company when leading Central Remedial Clinic charity walks on 14 annual trips to Ireland.
Mr Philips spoke out to assure the public that "nothing untoward happened" while Savile participated in sponsored walks for the clinic.
He was employed to chauffeur Savile when in Ireland and he says there were always senior figures from the CRC accompanying the celebrity to his Irish engagements.
A report produced jointly by the Metropolitan Police and the Society for the Prevention of Cruelty to Children in Britain was published yesterday which outlined the extent of Savile's sex crimes.
The report found he committed 114 sex crimes between 1955 and 2009, including 34 rapes and 135 indecent assaults. The sex attacks took place in several locations, including 13 hospitals and his victims were aged between eight and 47.
The report described Jimmy Savile as a "prolific, predatory sex offender".
Mr Philips told RTE Radio One's Liveline show that no one had any idea that Savile posed a danger to young people.
"He fooled a huge amount of people, including me," he said.