I have done exactly the same. But I end up doing it quite a lot. Never happens when with my girlfriend.
The problem isn’t that I don’t need much sleep, its that I naturally seem to sleep better during the day. A lot of people say just sleep when you are tired and wake up the same time every day and it will even out. Tried this, but my sleep time just gets later and later till after about 3 days I am going to sleep 2 hours before alarm and subsequently sleep through it till the afternoon. By this stage it makes more sense to just stay awake and reset body clock by going to bed at a reasonable time the next night (adding an hour ish to compensate for the lack of previous sleep).
I have no problems falling asleep when I am tired, and its always good sleep, so not insomnia.
I guess its caused by a lack of routine (student).
I know this isn’t much help, just thought I’d share so you know you’re not alone!