Basically, your answer is what 404 said.
Issue 1 is contention. ADSL connections are shared at the exchange, so when lots of people use it at once, the speed drops.
Issue 2 is your ISP playing silly beggers with the connection to help minimise the effects of issue 1. They'll do things like "packet shaping" and artificially restricting your bandwidth so that the heavy users don't cause the light users to disconnect completely.
The solution in part is LLU. Most traditional ADSL reuses BT's equipment, so whether you go with ISP A or ISP B, you're essentially buying the same product that they've bought wholesale from BT. LLU providers user their own kit, so they're not tied to BT's technical restrictions (or their pricing models).
Have a look at the checker on http://www.samknows.com and it'll tell you what LLU providers are available in your area. Some are better than others, and there's opportunities for bundle bargains (eg, O2 give you a discount if you already have an O2 mobile, Sky discount if you have Sky TV, etc.) If you report back here, I can perhaps advise further.
The other thing is to make sure your home wiring is in order. DIY phone extensions can have a massive negative impact on your broadband performance for instance, but this is a topic in itself and probably should be another post.
(note to techies, I've intentionally oversimplified here for clarity)