Momentum keeps increasing as the arrow gets heavier. At what point does the momentum reach its peak? When is the arrow too heavy for the bow?

Momentum continues to increase long after trajectory becomes unacceptable.

At some point, the arrow becomes too heavy to be pushed forward by the bow, but for that to happen, the arrow would need to weigh more than the holding weight of the string at full draw. For perspective, a 70 pound compound with 90% let-off has a holding weight of 7 lbs, which is a massive 49,000 grains.

So choose the heaviest arrow possible which has an acceptable trajectory.