2005 Fuel EX used a bonded in bearing. Often it's not the bearing thats worn, but the bearing disbonding from the frame.
If the bearing just "drops" out when you move the swing-arm away then the bonding has broken down. if yo're tight – sometimes you'll get away with re-bonding the original bearing.
when you re-install, you need to clean the area thoroughly, removing any trace of older loctite (fine wet'n'dry paper is good), and then re-bond with loctite 638 and re-assemble bike. Leave overnight resisting the temptation to compress the suspension.