Both sides have fragmented positions which mean that anti-immigration rhetoric is attractive.
For conservatives (snmall and big “c”) with an idea of “English -(or British)ness” endless immigration and a tendency to fawn over chicken tikka masala as an imporvement over anything the yokels cook is bad, while for labour/union folk immigration increasing the size of the labour force reduces the buying power of labour and benefits capital.
Meanwhile the cuddly and diverse left and the free-trading capitalist right share an enthusiasm for immigration.