From The London Times:
Because we are mammals and blessed with body hair, three middle ear bones, and the ability to nourish our young with milk that females produce in modified sweat glands called mammary glands. Although females have the mammary glands, we all start out in a similar way in the embryo. During development, the embryo follows a female template until about six weeks, when the male sex chromosome kicks in for a male embryo. The embryo then begins to develop all of its male characteristics. Men are thus left with nipples and also with some breast tissue.
Men can even get breast cancer and there are some medical conditions that can cause male breasts to enlarge. Abnormal enlargement of the breasts in a male is known a gynecomastia. Gynecomastia can be caused by using anabolic steroids. So, if your favourite athlete suddenly develops man boobs and starts winning gold medals, you know the reason why.
More here.