The question is too broad to be answered directly so I will provide a somewhat general scheme.
Basically in an integral like
$$
\int d\mu A B C
$$
one would seek to expand each part in irreducible representations of a given group, say for instance
\begin{align}
B=\frac{1}{\vert \mathbf{r}-\mathbf{r}^\prime\vert}
=\frac{1}{r} \sum_{\ell} \left(\frac{r'}{r}\right)^\ell \sqrt{\frac{4\pi}{2\ell+1}}Y^{\ell}_0(\theta,\varphi)
\end{align}
where here the group would be $SO(3)$ and the irreducible representations are labelled by $\ell$. Doing the same for $C$ and $A$, v.g.
\begin{align}
C&=\sum_{\ell m} c_{\ell m}Y^\ell_{m}(\theta,\varphi)\, ,\\
A&=\sum_{\ell m} a_{\ell m}Y^\ell_{m}(\theta,\varphi)\, .
\end{align}
The integral then becomes
\begin{align}
\frac{1}{r}\sum_{\ell_1m_1;\ell_2m_2;\ell}
a_{\ell_1m_1}c_{\ell_2m_2}\left(\frac{r'}{r}\right)^\ell
\sqrt{\frac{4\pi}{2\ell+1}}
\int Y^{\ell_1}_{m_1}(\theta,\varphi)Y^\ell_0(\theta,\varphi)
Y^{\ell_2}_{m_2}(\theta,\varphi) \tag{1}
\end{align}
and the last term is automatically $0$ unless we have
\begin{align}
\ell_1\otimes\ell\otimes\ell_2&=\mathbf{0}+\ldots...\, , \tag{2a}\\
m_1+m_2&=0 \tag{2b}
\end{align}
where (2a) comes from angular momentum coupling to the representation $\mathbf{0}$ (i.e. total $L=0$) and (2b) is the condition on $SO(2)\sim U(1)$ that the resulting magnetic quantum number is $0$.
There is nothing a priori to restrict the sum over $\ell_1,\ell_2,\ell$ in (1) unless you have some prior knowledge of $A$, $B$ and $C$.
The same general principle holds for point groups. In the case of
point groups you would expand each $A$, $B$, $C$ in terms of representation of the specific point group, and use the great orthogonality theorem of representations (also called Schur orthogonality relations). Probably the integral would be broken in group elements multiplied by cosets, i.e .the integration would be written as $g\cdot h$ where $g$ is in the group, and some sums over $g$ would be $0$ if the combination of representations contained in the decomposition of $A$, $B$ and $C$ can be combined to the identity (or trivial) representation. There would then remain integration over cosets. This is a bit of what happens in the example above: writing a rotation as $R_z(\varphi) R_y(\theta)$ (there is no third angle here) the condition $m_1+m_2=0$ gets rid of the $R_z(\varphi)$ integration and the result is an integration over $R_y(\theta)$ only.
Prof. Mildred Dresselhaus of MIT still has coursenotes available, and co-wrote an excellent textbook on the general topic.
Edit:
So it seems your “real solid harmonics“ are basically the same as my spherical harmonics, up to some linear combinations.
- I do not understand your comment re: Hilbert space. The Hilbert space here is the space of all 2-particle states (as you have written your states as products of two states).
So a more or less general procedure would be as follows.
Find the linear combinations of your basis sets that transform by irreducible representations of your points group. For instance, if you “only” need axial symmetry, then combinations of the type $Y^\ell_m\pm Y^\ell_{-m}$ will produce cosine and sine pieces that are symmetric or antisymmetric w/r to reversal of the $\hat z$ axis. There are systematic ways of finding these, using projection operator techniques (someone already pointed that out).
This decomposition is usually not that bad if the group has few representations but then some irreps may occur more than once and it can be a computational headache unless one is careful. In other words, the projection technique might provide you with multiple solutions which you have to specialize and properly normalize. The projection gives you (usually) one state in the irrep and you may have to work a little more to construct the remaining states, although with point groups the matrix representations are well known so it’s not that bad.
Basically the step above means you are no longer working with functions $\boldsymbol{\phi_3}\boldsymbol{\phi_4}$ in your original basis set but some combinations of states. You also need to expand the Coulomb term in this way.
The last step is to use orthogonality of group functions to eliminate some terms. The non-zero terms that survive are those for which the tensor product $\Gamma^*_k\otimes \Gamma_r\otimes \Gamma_i$ contains the identity representation. Here, $\Gamma^*_k$ is one piece in the sum of the expansion of $\phi_1\phi_2$, $\Gamma_r$ is one piece in the sum of the expansion of $1/\vert\mathbf{r}-\mathbf{r'}\vert$, and $\Gamma_i$ is one piece in the expansion of $\phi_3\phi_4$. This type of triple product may occur more than once for $(k,r,i)$ if the irreps $\Gamma_k$ etc occur more than once in the decomposition of the old basis set in the basis set.
You get to decide if finding these combinations ends up saving time over simply evaluating the original integrals.