Silicon sealant for kitchens/bathrooms contains allsorts of nasty chemicals and shouldn't have any contact with your skin. It may bring you out in all sorts of nasty rashes
From the COSHH sheet for Dow Corning Silicon Sealant
Avoid contact with skin and eyes
MAY CAUSE IRRITATION TO THE RESPIRATORY SYSTEM
MAY BE HARMFUL IF INGESTED IN QUANTITY
MAY CAUSE EYE IRRITATION
SKIN - IRRITATION AND DERMATITIS MAY RESULT FROM PROLONGED CONTACT
Do not breathe in vapour
Although shorts have silicon bands on them they will not be made of the same stuff (similar but not the same)
This time of year bib shorts are the way to go