Vehicles riding byways have to be road legal don't they?
Yes, taxed, insured, compliant with the construction and use regulations, etc. As far as those sorts of things are concerned, all the trackways that motorbikes can legally use (e.g. Byway Open To All Traffic) are essentially roads that just happen to be unsurfaced, so most of the same rules apply.
Enduro motorbikes generally are constructed to be road legal as they can be required to use public roads as liaisons between stages at some events, e.g. the Enduro World Championship. Course, that doesn't mean they've been kept legal after they were bought, and they still need number plate, insurance, etc. Motocross bikes are not usually road legal when bought as they have no need to be, but they can be made road legal.
Rough guide - if it has a number plate and a tax disk (if you get close enough to see that) the bike is likely to be road legal (no guarantees of course). If it has no number plate it's clearly not road legal and can't be used on any public right of way. If it's being ridden on a restricted byway, bridleway or footpath it's not allowed to be there, so is being ridden illegally even if it has tax, insurance etc.
If you see something that's clearly illegal it is worth reporting it. I trail ride motorbikes but I have no sympathy for anyone riding an illegal bike, riding where they shouldn't be, or generally making a nuisance of themselves, because all it does is reinforce the impression that all off road bikers are hooligans and even more of the 5% of routes that we can currently use end up closed to us (which is actually pointless as the idiots causing the trouble are by definition people who don't care about breaking the law, so it doesn't stop them anyway).
D0NK has a good point though. If you report motorbikes for being somewhere they're not supposed to be, but you also use it and bicycles aren't allowed either, there is a danger of getting caught up in any action the police take (assuming they do anything at all...). Caveat Reportor, or something.