Why would you even care if someone had access to your medical records? Have you ever seen one?
If it's hospital based it's probably a paper file, tatty, with nice stickers all over the front and with completely out-of-date results sheets falling out of the back pockets. The medical history will be virtually unreadable, and pretty useless even if it were legible. If you're taken into hospital some junior doc will clerk you in and take your relevant medical history anyway. Very few doctors will bother to read your notes in detail. Stealing patient data is possible, but only on a small scale. It's great strength in terms of security it that it's so damn cumbersome.
Conversely, if it's GP based it will be an electronic patient record. And yes, it WILL likely contain all sorts of stuff you wouldn't care to have your best friend knowing. Electronic patient records are highly insecure and impossible to protect given the current patient administrative systems in place in most GP surgeries. In short, any member of practice staff could access and download patient data without much chance of being discovered. A really naughty person could download the complete practice patient record simply by modifying the backup routine. It has, to the best of my knowledge, already been done.
But the point is, why worry about it? You can't change anything. Worry about things you can influence, like what you're going to have for dinner. That makes sense:-)