Dietary specialization is conditionally associated with increased ant predation risk in a temperate forest caterpillar community

The enemy‐free space hypothesis (EFSH) contends that generalist predators select for dietary specialization in insect herbivores. At a community level, the EFSH predicts that dietary specialization reduces predation risk, and this pattern has been found in several studies addressing the impact of individual predator taxa or guilds. However, predation at a community level is also subject to combinatorial effects of multiple‐predator types, raising the question of how so‐called multiple‐predator effects relate to dietary specialization in insect herbivores. Here, we test the EFSH with a field experiment quantifying ant predation risk to insect herbivores (caterpillars) with and without the combined predation effects of birds. Assessing a community of 20 caterpillar species, we use model selection in a phylogenetic comparative framework to identify the caterpillar traits that best predict the risk of ant predation. A caterpillar species' abundance, dietary specialization, and behavioral defenses were important predictors of its ant predation risk. Abundant caterpillar species had increased risk of ant predation irrespective of bird predation. Caterpillar species with broad diet breadth and behavioral responsiveness to attack had reduced ant predation risk, but these ant effects only occurred when birds also had access to the caterpillar community. These findings suggest that ant predation of caterpillar species is density‐ or frequency‐dependent, that ants and birds may impose countervailing selection on dietary specialization within the same herbivore community, and that contingent effects of multiple predators may generate behaviorally mediated life‐history trade‐offs associated with herbivore diet breadth.