Is body armour or back/chest protectors in horse riding practical?

Ok this may sound a silly question but it is genuine. I have heard the argument for and against. I've seen advertising that said you should wear regardless of your skill level and discipline. I have just started out and all I ever want to do is trek with horses - countryside, mountains, abroad etc. I do not intend jumping. Now I have heard stories of people coming off horses while nearly stationary and hurting their backs and also at a gallop and coming away with minor sprains. Please back up the reply with rational or experience. Thanks in advance.