Could it be the ‘apology’ that’s the issue? In inverted commas because it’s clear you don’t think you said anything offensive and don’t think this individual should have taken offence.
acknowledged his interjection, apologised if he took offence at anything I said,
Apologising “if any offence was given” is not apologising.
If what you’re saying is you didn’t give any offence (but sorry if you did, but you didn’t) then it’s really not an apology.
I was not aware of any negative connotation of the phrase.
Me neither until now for what it’s worth, but one lives and learns.
He rejected my offering and left the premises.
If you’re going to apologise for something you’ve done (whether or not you are at fault) do it like you mean it or not at all.
(Editing to say that talk of jungle this that and the other have been tropes of racist language and white guys like presumably you and certainly I don’t get to decide who should be offended.)