"I was uncomfortable to see what was happening on my land," Mr Graham told the Herald, explaining his decision to ask her to move on. "I wasn't sure who she was. Someone told me how big and important she was when she got here, but she's not really my cup of tea."
An admission which probably puts the Ulster farmer in a minority of one.
Four years ago she exploded on to the world stage with her smash hit single Umbrella, and since, Rihanna has become one of the most powerful and successful women in the music business. She has had more US No 1 singles than Beyonce and Lady Gaga combined. She has released five platinum-selling albums, with a sixth out shortly. And she's about to play a series of sell-out stadium gigs in England and Ireland over the next weeks, with a Dublin show on Monday night.
Like her idol, Madonna, Robyn Rihanna Fenty is one of those few performers famous enough to be known by just one name. And also like Madonna, wherever Rihanna goes controversy is sure to follow.
Earlier this year she caused a storm when her song S&M was banned on radio as it contained the lyrics: "Sticks and stones may break my bones but chains and whips excite me".
The video, which was banned in 11 countries, featured Rihanna and a host of lithe models, writhing around in PVC bondage suits, brandishing riding crops. Now you can only watch it on YouTube if you sign an electronic disclaimer.
Rihanna understands the power of shocking her audience. In the promo for another video, Man Down, she played a woman who kills a man after he rapes her.
And, of course, who could forget her performance on the X Factor at the end of last year that had the critics up in arms as she crotch-thrusted her way around the stage wearing extremely tight hot pants and a strapless bra.
But while Rihanna has always been a firebrand, some have pinpointed a specific moment as the catalyst for her change into the feisty performer that she now is. The night before the 2009 Grammy Awards, where she was due to perform, Rihanna's former boyfriend Chris Brown viciously assaulted her in the front seat of their rented sports car. Leaked photographs showed Rihanna's face bruised, cut and swollen.
Although Brown later pleaded guilty and was sentenced to six months' community service and five years' probation, Rihanna refused to let herself be painted as the victim. She withdrew from public life for almost a year, but re-emerged with the highly charged album, Rated R.
"I've put everything I've wanted to say for the past eight months into my music," she said in an interview. The songs were not your usual frothy pop. Russian Roulette tells of a near-death experience at the hands of a brutal lover, while G4L is about a female revenge fantasy. In interviews around the release of the album, Rihanna got more risque, telling one journalist that she was "a bit of a masochist . . . It's not something I'm proud of . . . It's common for people who witness abuse in their household. They can never smell how beautiful a rose is until they get pricked by a thorn".
The eldest of three children, Rihanna was born in February 1988 on the south-west side of Barbados, to an Afro-Guyanese mother and a Barbadian-Irish father. The island may be mistaken for paradise, but her parents' marriage was anything but idyllic, with her father Ronald reportedly battling a crack cocaine addiction for much of Rihanna's youth. Her parents eventually split when she was barely 10 years of age.
"Everyone has something that makes them stronger in their life," she said. "My childhood experiences helped make me a stronger person and a stronger woman."
Her school days were not easy either. Her light skin tone apparently made her a target for bullies. But at the age of 15, she entered a beauty pageant and won, inspiring her to pursue a music career. She formed a band with school friends and won an audition run by producer Evan Rodgers. He later commented that, although the trio auditioned for him with a Destiny's Child song, "the minute Rihanna walked into the room, it was like the other girls didn't exist".
A year later, she moved to the US after catching the attentions of Jay-Z. "She was obviously nervous," Jay-Z said of their first encounter. "Now she has a big personality, but I didn't get that in the meeting. What I did get was her eyes, this determination. She was fierce. I knew she was a star."
Jay-Z has been one of the defining factors in her success. She joined his management company, Roc Nation, last year. Since she has collaborated with the likes of Eminem, Kanye West and Drake, which has sent her career into overdrive.
Not surprisingly, men have readily fallen for her charms. Aside from Brown, she has also dated Shia LaBeouf, Josh Hartnett and baseball star Matt Kemp. Colin Farrell was apparently very taken with her after they met on The Graham Norton Show last year, and they reportedly dated for a while.
Rihanna herself has complained that she doesn't give herself time for a relationship, and that men are intimidated by her. "I think people are a little afraid. It can turn out bad."
Even if she's not in the mood for settling down, Rihanna's packed quite a bit into her 23 years -- 60 million singles and 20 million album sales, as well as more headlines than stars twice her age could manage.
Only one thing's for certain, though, by the time Rihanna touches down in Dublin on Monday, she'll be the only show in town.