Multibody interactions can reveal higher-order dynamical effects that are not captured by traditional two-body network models. In this work, we derive and analyze models for consensus dynamics on hypergraphs, where nodes interact in groups rather than in pairs. Our work reveals that multibody dynamical effects that go beyond rescaled pairwise interactions can appear only if the interaction function is nonlinear, regardless of the underlying multibody structure. As a practical application, we introduce a specific nonlinear function to model three-body consensus, which incorporates reinforcing group effects such as peer pressure. Unlike consensus processes on networks, we find that the resulting dynamics can cause shifts away from the average system state. The nature of these shifts depends on a complex interplay between the distribution of the initial states, the underlying structure, and the form of the interaction function. By considering modular hypergraphs, we discover state-dependent, asymmetric dynamics between polarized clusters where multibody interactions make one cluster dominate the other.