To a certain extent depends on the company, a very large one with 'hr' probably has agreement details in place (*to deal with working mothers) in a smaller/new environment ... does anybody else do this or part time etc ? - if so find out what their deal is as that can be used as a basis, either way I'd make sure everything is 'in writing' so that the agreement is clear all round and can't cause problems.
I'm afraid I do think it could affect your career - even if only at the perception level, 'small' means less cover etc, and if it's a small company it's a pita if the person if you need is not there. I work 'earlies', agreed with my manager, and the number of people who look at their wrists when I walk out at 4 is noticeable, as was the comment 'it must be 4 X is going home' from a senior manager, funnily enough nobody comments when I get in early .... but like you I'm living now, I've downsized from a London job and it's worjking for me.
*meeting == teleconference facilities exist, so dial in