if you change I'd say you get half the benefits of your previous class and half the benefits of the new class.
That way there is a penalty and it can't be abused to some mocking advantage.
Of course you can stick with your current class and therefore, no penalty for choosing what you wanted to be and keeping to it.