I tried once and failed, so took it to a shop who also failed. For me it was a temporary borrow of the headset I was after as the wrong race came with a new frame that I wanted to ride, so I was able to just leave it.
I did ask on here about getting it out as there’s not enough room between the aluminium insert and the top of the headset to drift it out, as you’ve just found.
Brant replied and said that the way to do it is to put the headset into a vice and to then twist/wiggle the frame off. I never needed to give this a shot though, so I don’t know how hard it is.