Can you half or double damage twice?

Nope. If one or more effect would double or half the damage from a single attack or spell action, you only apply one modifier.