It's the Hallgrimskirche tower you can ascend, not the cathederal. Try to time it for when the bells ring:)
Mooching around the harbour is great, the fleamarket is always good for a laugh.
Tjorninn (the lake in the park) is fun if you like being mobbed by tame ducks- take some bread. Then go and visit the Radhaus (Town Hall) to look at the enormous 3d map of where you have spent the previous week.
The National Museum is very good. The displays about the Settlement are quite evocative, esp the outline of the boat on the floor.
There is a fair bit of interesting architechture to be seen, Harpa (on the waterfront), The Hallsgrimkirche on the hill and Perlan (Oskjahlid), the Hot water tanks all spring to mind. the latter has a exhibition inside, a great viewing platform and stunningly expensive cakes. Good coffee (like everywhere else). The willy museum has now moved to Husavik.
Arbaer, out of town, is the 'Museum of life' type place, old buildings and similar.
And, as everyone else has said- nightlfe starts later than here and is absurdly expensive! Nurse that Viking Gul. Viking is the local beer, Gul= Gold, the brew.