Though we might not have the answer, let's look at the positive and negatives.
Milk is rich in electrolyte and carb content
Fluid and electrolytes are lost when we sweat, and must be replaced to prevent dehydration and performance dips during exercise. Fluid uptake in the small intestine is enhanced by the presence of glucose and sodium, both of which are apparently found in ideal quantities in milk.
Milk contains calcium and might assist in weight loss
Dairy is high in calcium. Under calorie constricted conditions, high dairy diets are associated with reductions in body fat and weight loss. On the contrary, laboratory models show that low calcium intakes stimulate fatty acid deposition and suppress fat oxidation. Because calcium supplementation alone does not have the same effects, it is believed that the combination of calcium, protein and other bioactive compounds present in milk might be responsible for dairy-affiliated weight loss.
Milk contains nutrients that can aid in workout recovery
We know that milk is rich in protein and that protein acts as an effective recovery aid by helping to rebuild damaged muscle. Additionally, milk is comprised of 80 percent casein, which is a digested protein thought to prolong muscular protein synthesis. Milk is also rich in branch chain amino acids that have been shown to have anabolic effects on muscles when consumed during recovery.
Milk has an acidic effect on body
Although milk is an alkaline base, it actually has an acid-forming effect on the body. After a hard training session, the accumulation of lactic acid caused by muscle contractions calls for alkalizing foods to expedite recovery. Many varieties of beans, fruit and some nuts (cashews) can have an alkalizing effect on the body.
Milk has lots of fat
49% of the calories in whole milk are from fat. Plus, a large part of the fat in whole milk is saturated fat. Although you can buy skim or reduced fat milk, these defatted versions have higher sugar contents which will convert to fat in the body.
Milk contains potentially-harmful growth hormones
Milk contains 59 hormones, one of which is IGF-1 (Insulin-like Growth Factor ONE.) This growth hormone, which functions to make newborn calves grow quickly, is considered to be a "fuel-cell" that promotes human cancers.
Milk is bacteria-ridden
One cubic centimeter of commercial cow's milk can contain up to 750,000 somatic cells and 20,000 bacteria before it is kept off the market, which is equivalent to 20 million live bacteria.
Though we might not have the answer, it's best to measure how you feel after consuming milk, and more relevantly, how you feel after working out on a day you've consumed milk.