Over-kneading whole wheat bread dough results in a tough, dense loaf because it damages the gluten structure and releases too much gluten, making the dough slack and unable to hold its shape.
Whole wheat flour contains bran, which has sharp edges that cut through the developing gluten strands during kneading. Gluten is the protein complex that gives bread its structure and elasticity. When you over-knead whole wheat dough, you excessively develop and then break down the gluten network. Initially, the dough might seem strong and elastic, but continued kneading weakens the gluten, causing it to become slack and sticky. This damaged gluten structure is unable to trap the gases produced by the yeast during fermentation, leading to a bread that doesn't rise properly and has a dense, tough texture. Furthermore, over-kneading can also oxidize the dough, affecting the flavor and color of the final product. The dough will feel very soft and almost soupy.
To avoid over-kneading whole wheat dough, use the windowpane test. Gently stretch a small piece of dough; if it stretches thin enough to see light through it without tearing, the gluten is sufficiently developed. With whole wheat dough, you may not get a perfect windowpane, but aim for a thin, translucent stretch before the dough starts to tear.