look back to A-level maths were you have the 2 snooker balls and working out the energy transfer.
You need the weight of the hammer (head will do) and the speed it is being swung at. (you actually want the whole weight applied with an angular velocity to give an angular momentum but if we assume low angle, ie not a huge sledge hammer, we can ignore this).
Going back to the snooker balls there will be a transfer of energy to the nail and a rebound of the hammer but the elasticities of the system will be accounted for using a coefficient of restitution. This will depends on the resistance of the substrate to the nail penetrating, change in momentum of the nail (tiny) and global deformation of the substrate – ie a concrete wall will be solid but a beam of timber will bend.
The best way to work it out would be to have the substrate horizontal. The nail vertical just in the substrate and then pivot the hammer from the end of it’s handle so the shaft is horizontal when it strikes the nail. Allow the hammer to fall and strike the nail and then see how high it bounces up (use a camera or something to record it). Catch the hammer before it double strikes (or you have more work to do).
You can then get out all the figures you need, ignoring energy lost to the surroundings, supports and air resistance using the calculations here:
http://www.efm.leeds.ac.uk/CIVE/CIVE1140/section04/mechanics_sec04_full_notes02.html