Could be symptoms of bonking. Simply put, your body needs energy to keep warm / exist. Don't be fooled by surving on the RDA of calories AND then exercising, you need to fuel your body over and above the 2500 recommended cals.

You may be exhausting your energy supply and at the end of your ride your body is unable to sustain its effort at keeping your warm. Thats quite dangerous BTW.

Are you eating a lot pre-ride / during the ride? You should be. Try eating carb based energy once per hour during the ride - oats / flapjack / rice, energy bars...