$\sqrt{3}$ Case
$ \left( \sqrt{3} \times \sqrt{3}\right)R30$ is a short way of representing a supercell in Woods notation (1, 2), which essentially means that each of the basis vectors of the supercell are $\sqrt{3}$ times the magnitude of that of the primitive cell.
Woods Notation is given by: X(hkl) m × n - Rφ.
Here m and n are both $\sqrt{3}$, φ=30°
Now let us look at the $ \left( \sqrt{3} \times \sqrt{3}\right)R30$ supercell from the answer with some annotations that I added.
Here a1, a2 are the basis vectors of the primitive cell marked in green.
And b1, b2 are the basis vectors of the supercell marked in purple.
So from the image you can see that, to calculate b1, from the triangle law of vectors we first need to traverse 2 units of a1 and then 1 unit of a2.
Corresponding to [2 1 0], the first line of matrix.
Similarly, to calculate b2, we need to traverse negative 1 unit in a1 and 1 unit in a2.
Corresponding to [-1 1 0], the second line of matrix.
The additional row and column are introduced to represent a 3D rotation matrix, where the third dimension remains unaffected. In this case, the third row and column are [0, 0, 1], indicating that the z-axis remains the same.
Thereby, this gives us the transformation matrix:
\begin{bmatrix}2 & 1 & 0\\-1 & 1& 0 \\0 & 0& 1\end{bmatrix}
Generic Case
Let b1/a1= m, b2/a2=n as per the Woods notation.
Let us assume |a1|=|a2|=1 since we start with unit cell.
Let $G_{11}, G_{12}, G_{21}, and \space G_{22}$ be the factors that a1 and a2 are multiplied in the Matrix form of Woods notation to obtain b1 and b2.
That is,
$b_{1} = G_{11}*a_{1} + G_{12}*a_{2}$
$b_{2} = G_{21}*a_{1} + G_{22}*a_{2}$
Therefore by Cosine law of triangles we have:
$m^2 = G_{11}^2+ G_{12}^2 - 2*G_{11}*G_{12}*cos(φ)$
$n^2 = G_{21}^2+ G_{22}^2 - 2*G_{21}*G_{22}*cos(φ)$
In above equations, we know m,n and φ. So we can solve the equations to obtain relationships between $G_{11} and G_{12}$ and
$ G_{21} and \space G_{22}$ with that we construct the transformation matrix like:
\begin{bmatrix} G_{11} & G_{12} & 0\\G_{21} & G_{22}& 0 \\0 & 0& 1\end{bmatrix}
For example, in the $\sqrt{3}$ case, since m and n are the same it would result in same equation for both rows of the matrix.
Observe the triangle with the φ, we would get:
$\sqrt(3)^2 = G_{11}^2+ G_{12}^2 - 2*G_{11}*G_{12}*cos(90-φ)$
Here we take 90-φ because, we need angle between the two lines with known values in this case 2 and 1.
Solving the equation of this ellipse would give us many possible sets of solutions:
$G_{11} = -2, G_{12} = -1$
$G_{11} = -1, G_{12} = -2$
$G_{11} = -1, G_{12} = 1$
$G_{11} = 1, G_{12} = -1$
$G_{11} = 1, G_{12} = 2$
and similarly for G21, and G22.
This calculation is shown in Wolfram Alpha here
So similarly we can solve for any m,n and φ combination.