The factory, outside the capital Dhaka, is owned by Tazreen Fashions, a subsidiary of the Tuba Group, which makes products for Wal-Mart and other companies in the US and Europe.
Protesters blocked the streets of a Dhaka suburb today in a protest that turned violent at times.
Angry workers threw stones at factories, smashed vehicles and blocked a major highway in the area.
About 200 factories were closed for the day after the protest erupted in Savar, the industrial zone where the deadly fire occurred.
Firefighters pulled out at least 100 bodies and 12 more people died at hospitals after jumping from the building.
Unions in Bangladesh have long complained about unsafe factories where workers work in dismal sweatshop conditions.
Maj Mohammad Mahbub, the fire brigade operations director, said: "Had there been at least one emergency exit through outside the factory, the casualties would have been much lower," he said.
Local media reported that up to 124 people were killed. The cause of the blaze that began on Saturday night was not immediately clear and authorities ordered an investigation.
Soldiers and border guards were helping keep order as thousands of onlookers and anxious relatives of the factory workers gathered.
Relatives of the workers frantically looked for their loved ones. Sabina Yasmine said she saw the body of her daughter-in-law, but had seen no trace of her son, who also worked there.
"Oh, Allah, where's my soul? Where's my son?" wailed Ms Yasmine, who works at another factory in the area.
"I want the factory owner to be hanged. For him, many have died."
Tazreen was given a "high risk" safety rating after a May 16, 2011 audit conducted by an "ethical sourcing" assessor for Wal-Mart, according to a document posted on the Tuba Group's website.