Removing drums via EQ is not that easy.
Rather depends on the song to be honest, but as a general rule you are correct.
Audacity has a noise removal feature which works by you sampling the offending noise – in this case the drums – then removing any matching frequencies in the whole track. There’s going to be some collateral damage though no doubt.
There’s always going to be a bit of collateral damage, it just depends how important or not that is to the individual.
FWIW, my approach to doing this correctly would be to load up the song you want to remove the drums from in Logic, find a part of the song where there is a drum loop playing solo (or as close to solo as possible), and copy the loop into another track. Then start experimenting with various plugins in logic to isolate said frequencies, and see if you can remove them, before then applying these parameters onto the whole song.
Or… If you know that the song has synthesised drums (therefore always perfectly keep time), create a loop of the drums then create it’s inverse waveform on another track. Then play the two together and you should have total silence as the waveforms cancel each other out!
You would then just play the entire original track with the inverse waveform loop over the top of it, and hey presto, all you should hear is everything except the drums! 😀