The best time to travel is during the second trimester of the pregnancy. This is when you are most likely to be enjoying your pregnancy, have increased energy, and feel sick less often. During your first trimester you might feel nauseous and tired, and there is generally a greater risk of miscarrying your baby. During your third trimester you are likely to feel more uncomfortable during travel and most tired.
On top of that, if anyone thinks there is a risk you might go into labor, you may have trouble arranging your travel to and from your destination as airlines might be reluctant to have you on board a flight. In general airlines will not take you on board after around 32 weeks pregnant, or around 28 weeks if you are pregnant with twins or multiples. Some airlines will make an exception providing you have a doctor's letter saying that you can travel.
Avoid any holiday activities that might put a lot of physical strain on your body. For example, scuba diving and skiing. Not only because of the altitude but because of the risk for impact. Even if you work out regularly, when you traveling your body needs to adapt to the different weather and time zone. Therefore it is better to make each day's activity less strenuous than your pregnant body can cope with.