There are quite a few ways that all corroborate each other loosely.
One is to use red-shift. Because of the way the universe is expanding (like a balloon being inflated) the further away something is from any point, the faster it is moving away. So the more its light is red-shifted (due to the Doppler effect). We can check this for closer objects by measuring their distance another way, which involves measuring their apparent position 6 months apart when we are on the opposite side of the sun and triangulating.
Measure the red-shift, measure the distance. Some stuff is x billion light years away, but nothing can travel faster than light, so the (current) universe cannot be any older than x billion years.
As for galaxies crashing into ours – let me swap the balloon analogy for a slightly more complicated one. Imagine a conical bowl. If you fill it with water from a hole in the bottom, the surface area of the water gets bigger as you fill, of course. Now as it’s filling, put some.. I dunno.. polystyrene beads on the surface, and give it a swirl. On the whole, the surface is still expanding and most of the beads are getting further apart, but locally as the water swirls some of the beads will inevitably move towards each other and some will end up sticking together (in this case because of the water’s surface tension, but in the case of galaxies gravity creates a similar effect but over a longer range).