A bike that fits well, is a bike that you will love riding. A bike that is not set up correctly or fits poorly can lead to muscle and joint aches and pains. The key is to fit the bike to you.

Saddle height

Your knee should have a slight bend in it when your foot is at the bottom of the pedal stroke. The saddle should be parallel to the ground.  If the saddle is uncomfortable, change it. For short distances in an upright position (i.e. for more recreational cyclists), you need a fatter, softer saddle.

Handlebar height

As a rule of thumb your handle bars should be roughly the same height as your saddle or can be slightly higher for recreational cyclists or for those with back pain. Your elbows should be comfortably bent to allow for shock absorption and a relaxed upper body.

Foot position

Bike clips, shoes and toe straps are a personal choice. For the recreational rider who won't be wearing clips, make sure the ball of your foot is on the pedal, not the arch of your foot. If you are choosing to go for a longer ride, you may find shoes with a stiffer sole beneficial as they will help distribute the weight over the whole pedal. 

Other factors to consider

Make sure you have the correct frame size.
Your pressure should be evenly distributed between your hands and your saddle.
You should be able to reach your handlebars comfortably. A slight bend in your back is normal, but you want to avoid being hunched over.