Sounds like either

Damaged/grotty cabling

cable is clamped wrong altering the pull ratio through the arc,

Or

The hanger is bent (unlikely)

If it works fine on either the top or bottom, but never on both, i cant quite see past something happening around the middle where youve taken out the cog to make space for the 42. I know you say you've followed instructions but looking at the cassette, are the gears all equally spaced around the middle bit? Can you swap out spacers / turn the round, anything that might make that tiny adjustment that maybe isn't obvious to the eye?

I haven't done a conversion myself, or read the how to properly, but reading your post you 'removed 15&17 & spacer' and 'replaced with 16&42' - surely you also need a spacer somewhere around the 16?

That's what I thought bigger gap between the sprockets in the middle would cause this.
You did remove the 17t and the spacer in front of it .

Have you got the cable clamped around the right side of the clamp bolt on the mech. I've had exactly the problem you describe when didn't pay enough attention and clamped it on the wrong side.

I'd suggest you get the hanger alignment checked properly first(i'm amazed how many are actually bent, even new ones)as no amount of new bits and adjustments will solve a bent one. Once you know that's correct, move on to the cable, poor shifting in only one direction usually points to cable problems, replace inner and outer with good quality kit, and make sure you have smooth routing.Incorrect B-tension can also cause poor shifting, and unfortunately, setting it for a sprocket it wasn't designed to accomodate, will move it away from the rest of the block, maybe by enough for it to become a problem like yours, maybe not, but worth considering nonetheless.

had the same set up and it worked fine….wrecked the mech a few weeks back got a new one and could not get it to go through all the range…tried everything.. then admitted to myself indexing gears is a dark art and took it to LBS… PROB SOLVED.

the B-Tension adjuster leaves the mech as close to the 42T as I dare

One up you say, did that include the free 16t sprocket.

I think I remember reading recently other people having shifting problems where the one up 16t was found to be the cause.

I've the exact same problem. Only thing is that I've not got a 16 tooth fitted,just the original 15. It's getting a bit annoying.

The main reason why I chose the hope expander over the one up was the fact hope made theirs dedicated shimano or SRAM, as I thought a universal sprocket may suffer when shifting.

Thanks for the feedback. I will go and check my cassette again just to make absolutely sure that I have all the spacers in the right place. Have to admit, it does sound like there is a spacer wrong, even though I am almost certain I got it right!

In terms of checking if the alignment is out on the mech, how do I do that? Visually looking at the top and bottom jockey wheel, they seem to be dead in line. Is there a more scientific way of doing it?

Maybe I will try removing the 16T as well and putting the OEM 15T back in.