Man charged with slapping toddler loses his job