I would go in and have a friendly word, I think teachers should expect a little parental concern if their children are getting dropped a set.
But from your OP I would be prepared for a couple of things
1, you might have a biased view of your kids previous performance (maybe even encouraged by some misinformation from them). So be prepared that it might be the right decision, and allow them to give their justification.
2. The teacher who said it was nothing to do with them might have been passing the book so they don’t have to deal with any fallout.