Or will my frame die, more importantly?
if anything is going to fail, it will be the brake lever.
its very dependant on frame shape, but quite common. worst case scenario is a minor ding in the top tube and/or a broken lever. Very unlikely though.
I tend to run my levers a shade loose anyway. Apparently a motocross trick. As mentioned above, tighten them up, then loosen them just enough. you want to be able to use them safely, and not have them move under normal loads, but with a good bash they will swivel out the way. same for shifters too. if you crank them right up you are likely just to snap them off with contact with the floor/trees/your nuts, in a crash.