As mentioned above, cleanliness is crucial. Basically, the fermentation conditions which best suit yeast also suit bacteria (and random wild yeasts which may be present in the air), and these can lead to 'off' flavours in your beer.
Bottling is a bit of a hassle, as you have to clean, sanitise and rinse all the bottles, but it's a cheap way to get into home-brewing. You can always get into kegs later on - if you're going to get into it long-term then kegs are much less hassle (but relatively expensive to get set up with, hence the need for long-term use to get your money's worth).
You can get some very drinkable beer from kits. Ale is generally more forgiving than lager, so stick with that until you get more experience. Usually the yeast that comes with kits is a generic one that they chuck in with every kit, lager or ale. As a general rule, chuck out the yeast that comes with it, visit your local homebrew shop and buy some decent yeast specific to the type of beer you are brewing.
Avoid using table sugar if possible, buy the same weight of light unhopped spray malt (aka light dry malt extract, or LDME). This is also a sugar, but is derived from malted barley rather than sugar cane. Table sugar can add unwanted flavours, whereas LDMA will add extra malt flavour to your beer.
Finally, try to maintain a constant temperature during fermentation. Big fluctuations can often lead to various by-products which can affect the flavour of your beer. Usually 18-20deg C is good for ales, or about 10C for lagers.