The rights to various Marvel properties are owned by different studios – for example, Hulk, Iron Man, Thor, Captain America and The Avengers are all currently owned by Marvel Studios and Paramount, hence why they can have those characters crossing over between movies. The X-Men and The Fantastic Four are owned by Marvel Entertainment and 20th Century Fox, so they can have characters going between movies. Spiderman is owned by Columbia Pictures, I think. So The Avengers movie wouldn’t be able to have the Spiderman/Scarlet Witch/Wolverine in the team.
It’s already caused a few disputes, because there’s question over what characters belong to which franchise (for example, Scarlet Witch first appeared in the X-Men, but is arguably better known for being an Avenger). I one of the studios recently did a few animated features of certain story arcs, but had to swap out characters because of legal issues – in Planet Hulk, the Silver Surfer was swapped out and replaced by Beta Ray Bill.