In the optimization of whirl milling parameters, traditional methods often suffer from chatter instability due to ignoring dynamic physical boundaries, which makes it difficult to coordinate processing efficiency, quality and stability. To this end, the study proposes an optimization solution that integrates physical models and improved intelligent algorithms. At the model level, a stability hard constraint based on the regenerative flutter mechanism is constructed, and the physical safety boundary is internalized as the search premise of the algorithm. At the algorithm level, an improved multi-objective gray wolf optimization algorithm is designed, which improves search diversity by integrating chaotic mapping, and uses an adaptive penalty mechanism to deal with strong nonlinear constraints. The results show that the typical solution without constraints has an energy peak of 0.95 at the chatter frequency of 1500Hz, while the energy distribution of the constrained optimization solution remains flat. In the algorithm comparison, the inverted generational distance distribution range of the improved optimization algorithm is 0.03-0.07, and the average vibration amplitude of the finally obtained Pareto solution set is suppressed to about 0.18mm. Meanwhile, the solution set quality of the improved optimization algorithm after about 150 iterations has surpassed the final results of other algorithms after 500 iterations. In summary, the proposed optimization scheme integrating physical constraints and intelligent algorithms provides a new idea for solving the stability conflict of highly dynamic whirl milling, and achieves efficient and stable parameter optimization results under the given computational and experimental conditions for this category of parts.