Hi. Don’t make this a personal issue, I would imagine you are not alone in receiving this ‘sound’ advice from a manager who manages people in a passive\aggressive manner.
Continue to remain professional, which it sounds like you are and just continue with your role – seek clarification to any matters via email.
Needless worrying will ruin the quality time you have with your family, which is far more important!