The Portuguese made the revelation during a wide-ranging interview with French radio station RMC Sport, also admitting he passed up the chance to manage England in 2007, after Steve McClaren's departure and before Fabio Capello was appointed.
Mourinho arrived at Stamford Bridge in 2004 after winning the Champions League with Porto, and he would quickly claim two Premier League titles before leaving London.
A year's sabbatical was followed by a two-year stint at Inter Milan, which came to an end with a Serie A, Champions League and Coppa Italia treble before Mourinho accepted the top job at the Bernabeu.
The 49-year-old admitted it had been an emotional wrench to leave the San Siro, but he felt the opportunity to coach in the Spanish capital would not come round a third time.
"I had the opportunity to come here in 2006 or 2007," he told RMC Sport. "I was at Chelsea and I couldn't have said yes at the time.
"When they came back a second time, it was very difficult to say no.
"It was a moment that everyone at Inter understood would be hard for me to refuse. Everyone was crying, but they opened the door for me.
"It was difficult for me too but I wanted to have this experience. There are positive and negative things (about it), like at all clubs.
"I'm glad I came here and, having had the opportunity to win competitions in Spain, I'm glad to have had this experience."
However, Mourinho has been linked with a return to England in the summer as his tenure at the Bernabeu continues to go sour, with Real struggling to close a 15-point gap on league leaders Barcelona amid reports of in-fighting between the outspoken coach, his playing staff and several club directors.
Indeed, a last-16 trip to face Manchester United in the Champions League on March 5 will provide a welcome distraction, after February's first leg in Madrid.
"The La Liga title is impossible now, there is too big a gap," he added. "We have to focus on other goals like the Copa del Rey and the Champions League.
"I know that everyone is waiting for the 10th (Champions League title), but it does no good to obsess over these things, obsession only makes things more complicated.
"Playing in Manchester is fantastic. I've been there to watch and I have a house in England.
"It's a pleasure to have to travel for work. My 'brother' (Alex) Ferguson is a formidable example. At 71 years of age, he came to watch us in Madrid."
Mourinho could only ever consider leading Portugal should he venture into international management, but he admits England remains his "first love" in the world of football.
"Portugal is my home," he added. "After my experience with Porto I wanted to try something different.
"It was the right time to leave in 2004. I will go back one day but not for a club. (I will go) for the national team, or to go home and retire.
"I would love to be the national team coach one day. I think the Portuguese people are waiting for that to happen.
"I want to experience coaching my country at a World Cup or at the European Championships.
"I almost did it with England, in 2007 or 2008, but for me the national team is Portugal.
"I can never say never but I think that it would be with Portugal.
"England is my first love. Well, my wife is my first love but England is my first love from a footballing perspective."