You will need to speed up the rebound, compression damping, just leave it where it is, though it may need a little lowering.
When you are moving the fork, all that is happening is energy conversion. Kinetic energy is converted by the spring in strain energy in the spring material, or we can consider it and potential energy, the energy the spring converts is not lost from the system, it is there ready to be converted to kinetic energy again to extend the fork. The damper disperses the kinetic energy, the damper converts the kinetic energy to heat energy and is lost from the system.
Bit horrible that paragraph.
Kinetic energy converted to potential energy (spring, stored) + heat energy (damper, lost).
So softer spring, lower load, less stored energy extending the fork again, less damping required to disperse the stored spring energy on extension.
The compression side of things, less kinetic energy compressing the fork, less kinetic energy needing to be dispersed, less damping required i reckon.