Creator Simon Cowell and fellow producers are discussing replacements for Spears and another panellist, record producer Antonio 'LA' Reid, said the source.
Reid previously announced he was leaving after two seasons. Demi Lovato appears likely to return, joining Cowell on the revamped panel.
The departures are yet another shake-up for X Factor, Cowell's attempt to strike gold in the US.
Cowell replaced panellists Paula Abdul and Nicole Scherzinger and host Steve Jones after a disappointing first season, and ratings dropped in the second season.
The December season finale drew 9.7m viewers, compared to more than 12m for the first year's season-ender.
Although the series has failed to match Cowell's bold pre-debut predictions, and certainly hasn't eclipsed sister Fox show American Idol, it has helped to shore up the network's autumn schedule.
Pop star Spears, however, proved less than a one-season wonder.
Combining fame and a history of erratic behaviour, she was seen as a potential spark for better ratings.
But reviewers criticised her for appearing inexpressive to the point of boredom and for a repetitively dull use of the word 'awesome'.