Concern is growing that inflammatory rhetoric about Haitians could escalate into actual violence – something that many have traveled hundred of miles to escape.