You have some of the picture…
In a system free of resistive forces (air resistance is the main one in this case), objects will tend to accelerate at the same rate, regardless of mass.
There’s a greater force of gravity acting for more objects of higher mass, but there’s a correspondingly greater force required to accelerate.
Effectively, you freewheel faster because the air doesn’t slow you down as much.
There’s a similar air resistance acting on you both (you will have similar cross sectional areas facing the direction of motion), but it takes more force to slow you down by the same amount (because you’re heavier) – as you’re slowed down less, you roll faster 🙂