The topological derivative of cost functionals that depend on the stress (through the displacement gradient, assuming a linearly elastic material behavior) is considered in a quite general 3D setting where both the background and the inhomogeneity may have arbitrary anisotropic elastic properties. The fact that the strain perturbation inside an elastic inhomogeneity remains finite for arbitrarily small inhomogeneities makes the small-inhomogeneity asymptotics of stress-based cost functionals quite different than that of the more usual displacement-based functionals. Such functionals include energy-based functionals such as the error in constitutive relation, which is a very effective tool for many material or defect identification problems. The asymptotic cost functional perturbation is shown to be of order O(a^3) for a wide class of stress-based cost functionals having smooth densities. Several 2D and 3D numerical examples are presented, demonstrating the proposed formulation on cases involving anisotropic elasticity, flaw identification, and non-quadratic cost functionals such as regularized versions of yield criteria.