The object of this paper is the differentiation of the infimum of parametrized objective functions with respect to the parameters as in Danskin [The theory of max-min, with applications, SIAM J. on Appl. Math. (4) 14 (1966), 641-644] who obtained a semidifferential equal to the infimum over the set of minimizers of the directional derivative with respect to the parameters. Yet, in applications to the topological and shape derivatives of the compliance, examples reveal the possible occurrence of an extra negative term: the so-called polarization term in Mechanics. The object of this paper is to introduce new theorems that can accommodate the occurrence of an extra term. For the shape derivative, the associated technique is a change of variable to work on the fixed initial domain; for the topological derivative, it is an extension over the hole created by the topological perturbation of the domain.