I’d get the same again if racing seriously and as you say, swap tyres. But you can get some serious gains with lighter wheels
If not, a light pair of wheels will make a big difference. You may save about a pound overall. I like Mavic Ksyriums, but not used their disc offerings. I would also change cassettes for road. I run 11-23 on the road wheels and 12-32 on cross. With a 38T narrow wide chainring. On the road it is a bit undwrgeared but I’m a spinner and it’s not a huge issue. I coast down hills on that bike
Tryes make a huge difference. Schwalbe Ones with latex tubes in 25c are lovely tyres for road. I’ve gone back to tubes from tubeless, but still have tubeless cross.
There will be no shifting issues.