Remove a person from your life completely if he is a considerable drain on your joy. You can do so with kindness by saying something to the person such as, "I value you, but I need a bit of a break." You can choose how much detail you provide.
Choose to limit your exposure to the person if you can’t remove him entirely. You can easily do this by saying some like, "Thank you very much for the kind invitation but I unfortunately can’t make it."