This is an interesting question. The Hessian diagonalization one is general, however.
Group theory is helpful when dealing with small symmetric molecules.
Although we have monograph for this topic, I can provide a concrete example.
Say the molecule H2O. We know that it has three vibrational modes, one for bending, and one for symmetric stretching and another for antisymmetric. From the point group point of view, the former two are symmetric ($A_1$), and the last one is antisymmetric ($B_2$).
If we use the group theory trick, which should work if, according to the orthogonality of normal modes, only one mode belongs to an irreducible representation,
otherwise modes in the same irreducible rep will mix with each other,
which cannot be separated with only group theory.
We also know that Hessian relies on the underlying quantum chemistry (QC) computation.
So if we compute the frequencies of H2O, but with different QC method/basis set, we should have the displacements in the normal mode of antisymmetric stretching remain exactly the same. However, because the symmetric stretching and bending are both of $A_1$, the displacement vectors of those modes will change with the choice of the QC method/basis set.
Note that when doing this test, we should NOT perform the geometry optimization,
which ensure that the "equilibrium" geometry is the same (although not, optimized or not will only raise a warning in most QC programs).
Here are the results from Molpro.
HF/6-31g:
Normal Modes
1 A1 2 A1 3 B2
Wavenumbers [cm-1] 1826.82 3645.93 3770.62
Intensities [km/mol] 104.40 1.26 33.92
Intensities [relative] 100.00 1.21 32.49
OX1 -0.00000 0.00000 -0.00000
OY1 -0.00000 -0.00000 -0.06913
OZ1 -0.07127 -0.04375 -0.00000
HX2 0.00000 0.00000 0.00000
HY2 -0.36845 0.60026 0.54867
HZ2 0.56568 0.34722 0.39634
HX3 -0.00000 -0.00000 0.00000
HY3 0.36845 -0.60026 0.54867
HZ3 0.56568 0.34722 -0.39634
HF/6-311g:
Normal Modes
1 A1 2 A1 3 B2
Wavenumbers [cm-1] 1834.52 3613.18 3733.07
Intensities [km/mol] 108.14 0.08 26.20
Intensities [relative] 100.00 0.07 24.23
OX1 0.00000 -0.00000 0.00000
OY1 -0.00000 -0.00000 -0.06913
OZ1 -0.07144 -0.04347 0.00000
HX2 0.00000 -0.00000 -0.00000
HY2 -0.36610 0.60169 0.54867
HZ2 0.56703 0.34501 0.39634
HX3 -0.00000 0.00000 -0.00000
HY3 0.36610 -0.60169 0.54867
HZ3 0.56703 0.34501 -0.39634
You will find our theory is correct.