Depends on how you go about it, Jerry. If you simply avoid the bloke, and are civil to bim when you can't, then it's probably fine.
It becomes bullying by exclusion if you do stuff like publicly inviting everyone else to a BBQ, but "forgetting" him...