Permission is being sought by the Office of Public Works (OPW) to build a commemoration to the victims of institutional abuse.
But the Irish Georgian Society said it "strongly disagrees" with the proposals, which include a new landscaped civic area at the Parnell Square site.
The garden, which was visited by Queen Elizabeth in 2011, commemorates those who died in pursuit of Irish freedom.
However, the new installation would be a tribute to children who suffered at the hands of religious orders.
In a letter of objection to Dublin City Council, the Georgian society said Parnell Square is "one of the finest of Dublin's 18th-century urban set-pieces".
It has the potential to "become a catalyst for regeneration in the city's north Georgian core", it added.
The application "gives no regard" to plans for the "proper and sensitive" development of Parnell Square, the society said.
The memorial, which is "subterranean", would provide a "dank, unsafe space which will attract anti-social behaviour", said group conservation manager Emmeline Henderson.
Independent councillors Mannix Flynn and Nial Ring also objected to the plans, believing the development would detract from the Garden's original purpose.
The OPW has sought approval for a covered walkway connecting a new hard and soft landscaped civic area at the western end of the site.
The space is to incorporate seating, water installations, feature lighting and sculptured elements. Council planners are due to decide within days.