There is swimming/paddling in the river at Brokenhurst (used to do that myself as a kid in the 70’s 😳 ). Otherwise not much of what you are looking for in terms of rivers/swimming holes as people go to the sea.
I quite like the pub/hotel at Bucklers Hard, the Master Builder. There is a small ship building museum next door which can be interesting. The car park for pub/hotel is through an electric gate but it’s free. You can picnic on the grass (perhaps unofficially) and watch the boats on the river.
We used to rent bikes form the Car Museum car park and ride down via Bucklers Hard to a little beach on the Solent (officially requires payment but we where never ever checked)
There is the Lepe Park
If yu want a day dtip you can condider getting the lymington/Yrmouth ferry (foot passengers) pretty village, quite a nice little beach which you can walk too, lots of pubs/restarants and boats to look at.
pay attention if swimming int he sea as the tides can be strong and some shelve quite steeply.