The former star had "total" access to the high-security hospital in Berkshire - where he would walk in on females as they bathed, a former patient from the 1970s said.
His former personal assistant Janet Cope also claimed that Savile had free rein of Broadmoor, where she accused him of working to gain access to high-profile criminals.
She told ITV's Exposure: "Jimmy was untouchable.
"(He) was very proud of the fact that he spoke to all these very high-profile poorly criminals." Ms Cope said that although she had never seen him abuse anyone, he was a controlling person.
She added: "I knew how he operated so I was very wary of him and I felt there was an undercurrent, there was a bad side, which not many people saw."
At Stoke Mandeville Hospital, Ms Cope said Savile held a position of real authority, adding: "He controlled everything.
"He controlled me. Everybody was - I'm going to use the word frightened - of him."
Savile, a former BBC TV presenter who died last year, has been accused of hundreds of allegations of child sex abuse and rape.
A former female patient told the programme that Savile would walk in while female patients were having baths.
She said: "He would come in when you'd be having a bath...he would just walk in."
Speaking anonymously, she added the star once tried to touch her breast, and when she reported him she spent months in solitary confinement.
Lawyer Liz Duck told the programme she was representing nine women who claimed they were abused by Savile at Stoke Mandeville Hospital.
"Some of them have had their lives destroyed (by Savile)," Ms Duck said. "Several of the victims did report matters to members. No action was taken."