If contactless cards were a significant security risk, you'd know about it by now... there'd have been a major fuss by the British Public.
It might feel less secure than Chip and PIN but it's clearly more secure than cash, which it's designed to replace...
If you read the info from Visa below, you'll see many safeguards are in place. I would check with your bank what their fraud policy is as suggested below and basically not worry about it
Info from Visa
Each participating contactless bank sets a limit on the number or value of Visa contactless transactions that can be made before a cardholder is asked to do a normal chip and PIN transaction. If you are a regular user of Visa contactless, you may have noticed this happening – as, every now and then, you will be asked to verify your transaction with your PIN to prove that it is you in possession of your card.
And, just like any other Visa transaction, your bank is routinely looking out for any unusual transactions that may show that it is not you using your card. You will get the same level of protection as any other Visa Debit or Visa Credit transaction. And, if your card is lost or stolen, the card will be blocked by your bank instantly.
If your card is lost or stolen you should notify your bank as soon as possible. If anyone has fraudulently used your contactless card to make a payment, providing you take reasonable precautions to protect your card and let your bank know as soon as you realise it’s gone, you will not be responsible for any losses incurred (subject to your bank’s terms and conditions).