Speaking at Habitat for Humanity's 29th annual Carter Work Project in Leogane, 20 miles south-west of the capital, Port-Au-Prince, Mr Carter said: "It's a disgrace to the international community to still have people living in tents.
"The Haitian government seems to be functioning better than the US government regarding this situation and the EU has failed to pass a budget on this issue.
"The Haitian government has to be given some breathing room and support as long as half of the international funding is spent wisely even though some of it may be washed away."
The Irish volunteers and the Carters have also been joined by renowned Country and Western singers Garth Brooks and Trisha Yearwood who have sold millions of albums in Ireland between them.
The husband and wife team of Brooks and Yearwood are pitching in as much help as they can in building two of 100 houses this week. They are also helping to serve food to almost 700 volunteers from Ireland, the US, Canada, UK and New Zealand.
"We simply want to build and help," explained Brooks who has had hit songs such as The Dance and Friends In Low Places . This is the second year that the two have taken part in building homes in Haiti.
"Last year, we were a little at sea with what we were doing here but this time we know more about is needed to be done," added Yearwood known for her hit song, How Do I Live.
The homes that are being built are approximately 11ft by 15ft and made with earthquake and hurricane cement along with timber.
Of the almost $4.5bn (?3.46bn) pledged by international governments, in the immediate aftermath of the 7.0 magnitude earthquake in January 2010, so far only $1.5bn (?0.76bn) has been delivered.
Geraldine Monaghan, from Dublin's Castleknock, said: "I feel as if I'm doing some good. This may not be everyone's idea of a holiday but I feel as if I'm doing a small bit to help. Some friends and work colleagues think I'm nuts."