This is a difficult question without a straight-forward answer. In general you have to perform a test to decide whether the harmonic approximation is sufficient or whether you need to include higher order anharmonic terms in the potential expansion. Due to the computational cost of including anharmonic terms, very often systems are assumed to be harmonic without further checks, which can be problematic.
Having said this, here are a few thoughts one can also use as a starting point. At equilibrium, a material sits at a local minimum of the potential energy surface. The harmonic approximation is then based on the assumption that the atomic nuclei/ions do not move very far away from this minimum, and that a second order expansion of the energy around the minimum is sufficient to describe atomic vibrations. Therefore, the harmonic approximation will break down when the atoms move significantly far away from equilibrium. A few examples include:
- High temperature. At sufficiently high temperature, solids melt, and all materials will behave in an anharmonic manner sufficiently close to melting. But what temperature is that? It is strongly material dependent. A starting estimate could be to use the Lindemann criterion which, roughly speaking, states that the melting temperature of a material corresponds to the atomic vibrational amplitudes approaching 15-30% of the interatomic distance. Therefore, if your atoms are vibrating anywhere near these amplitudes, it is likely anharmonic terms are important.
- Light elements. The vibrational amplitudes of an element are larger the smaller the mass. This means that anharmonic terms tend to be larger for the lighter elements, and indeed in some like hydrogen (lightest of all elements) they can dominate even at zero temperature (quantum fluctuations are anharmonic).
- Structural phase transitions. Even if your system is well below the melting point or not made of light elements, structural phase transitions can be dominated by anharmonic vibrational terms. The most well-known example of this are perhaps the perovskite family, which typically exhibits a series of temperature-induced structural phase transitions from high symmetry high temperature cubic to lower temperature lower symmetry tetragonal, orthorhombic, etc. The higher symmetry structures correspond to saddle points of the potential energy surface (rather than minima), and the structures are stabilized to those points through anharmonic vibrations. A purely harmonic description would lead to the presence of imaginary modes that would drive you towards the lowest energy structure, and fail to describe the stability of the higher symmetry higher temperature structures.