Should I be afraid of him anyways?

My boyfriend is Wiccan and told me that if he gets back into the habit of things, he's afraid he might accidentally hurt me. For him, it's a stress reliever and it runs in the family. What do you think? Should I be afraid even though I'm not?