Siptu bosses are due to meet their architects next week to decide whether to develop a scaled-down version of the skyscraper.
A Siptu spokesman said the project's backers "haven't a clue" what steps will be taken next.
"We will analyse the decision and the inspection report, but at this stage we cannot rule anything in or out," he said.
Sources close to the project admitted this weekend that the planning authority's decision to refuse permission has left them "shell-shocked".
"The worst we expected was the go-ahead with a number of strict conditions. We have to go back to the drawing board. We're gutted," a well-placed source said.
Independent councillor Christy Burke told the Herald that he is in "no doubt" that the current Liberty Hall building "has to go".
"Dubliner after Dubliner knows it's ugly and it's an eyesore. I desperately would like to see the building dramatically refurbished or removed," he said.
But the north inner city representative admitted that many people are opposed to the plan being completed "in the current climate". "I've had people ring me and say 'I don't want to see Siptu paying all this money when things are so tough'. That is certainly a view that's out there."
Fianna Fail councillor Mary Fitzpatrick expressed disappointment at An Bord Pleanala's decision and said she feared Siptu would decide to vacate the building.
"My understanding is that part of the plans involved improving health and safety aspects of the tower. I fear now that Siptu will be left with no option but to vacate and that could result in the building becoming dilapidated," she said.
Siptu had planned to demolish the Liberty Hall building and replace it with a skyscraper that was 35pc higher.
The designs included placing a public area on top of the structure which would be accessible by a glass-fronted lift.
But An Bord Pleanala rejected the proposal as being visually intrusive.
It said the proposed development would seriously detract from the setting and character of the Custom House, and would intrude on O'Connell Street and Grafton Street architectural conservation areas.
Dublin City Council had granted Siptu approval in February for the plan but attached 19 conditions.
Heritage Group An Taisce welcomed An Bord Pleanala's decision, adding that that there was no justification for the scale of the development.