What's the basis of people saying there is a problem with running a business from home?
Because in simple terms whilst there is nothing wrong with a home office, if you are running something from your house that becomes big, noisy or popular enough then it can be seen as a change of use from a 'dwelling house' to a business use and that requires planning permission.
Hence council will look into it seriously if miserable next door neighbour complains that you are running a mountain bike business.
One of the tests of whether its become too big etc. is whether it harms residential amenity of your next door neighbours. So unfortunately you can't tell her to p!33 off or own her with bombers as that only harms your case.
However, having said all that from the way the OP described his business set up at the moment it doesn't sound like he's in running enough of a business form his home for the planners to be interested yet.