(Also see "Obtaining Letters of Recommendation" section in the Mayfield Handbook (MIT access only) and my advice on writing recommendation letters.)
Advancement in your professional career (such as obtaining a job or fellowship) often requires that you obtain letters of recommendation from previous colleagues, supervisors, etc. In general, letters tend to be more important in academic jobs than for industrial ones. You shouldn't feel that you are imposing when you ask for such letters: the letter-writers recognize that it is part of their professional responsibility. However, you should also do everything in your power to ease the burden on your references.
Choose well. The ideal letter-writer is knowledgeable about you, about the place you are applying, and about the norms of letter-writing. Knowledge of you permits the letter to include specific examples and anecdotes, which are much more powerful than generic bromides or undifferentiated praise. Knowledge of the place you are applying permits the letter to specifically play up your strengths and qualifications. Knowledge of the norms of letter writing prevents the letter from being ignored because it is completely inappropriate. A good letter-writer also should not have a conflict of interest (such as being the advisor of another applicant). You should only request letters from people who will write you a good letter; you probably know whether this is the case but may want to double-check. Many applications request 3 letters. Including 4 strengthens your application; however, it's better to have 3 very strong letters than to have 4 letters, only 3 of which are very strong. Don't omit obvious people such as past supervisors and advisors from your list; these glaring omissions will lead those evaluating your recommendation to the conclusion that things did not work out and that person would have written a negative recommendation. Do try to include people with multiple backgrounds or who have seen you in multiple lights (corporate and academic supervisors, and teachers), but remember that people most similar to the letter readers will be able to write most persuasively: that is, for an academic or research job, focus on academics and researchers as references, and for a corporate job, your industrial bosses may will be best.
Start early. Give letter-writers a minimum of two weeks, and preferably a month or more, to write letters. (This is particularly true for the crucial job recommendation letters.) The letter-writers have other responsibilities and deadlines, and you don't want them to do a shoddy rush job or to resent a demand for immediate gratification.
Provide full information.
It is traditional for letters to be sent directly to the entity evaluating the application; when that isn't the case, the evaluation will be sealed before being given to you. Don't be paranoid about this: it is completely standard (and an ethical letter-writer would warn you before writing a negative recommendation in any case). You may be asked on an application whether you waive your right to view the letters of recommendation. You should always agree to waive that right. Many writers by policy do not write letters unless that clause is waived. (I know of some people who never write recommendation letters to some universities because those universities do not guarantee confidentiality of recommendation letters.)
Letters sometimes get lost, and people sometimes forget to write them. You should double-check that the letter has arrived, and if not, ask the writer (more likely, the writer's secretary) to send another copy.
Michael Ernst