One can start with the definition of the usual, integer coordination number (which as the OP said is not a continuous function of the geometry), and smoothen its discontinuities.
The (integer) coordination number of an atom $A$, $CN_A$, is the number of atoms that are coordinated to $A$. When an atom $B$ is sufficiently close to $A$, $B$ is coordinated to $A$; when it is sufficiently far from $A$, it is said to be not coordinated to $A$. This suggests a distance-based criterion, i.e. $B$ is coordinated to $A$ if and only if the distance between $A$ and $B$ ($R_{AB}$) is smaller than a cutoff, usually taken as a constant ($k$, often slightly larger than 1) times the sum of atomic (or ionic) radii of $A$ and $B$ (i.e. $R_A$ and $R_B$):
$$
CN_A = \sum_{B\neq A} CN_{AB} \tag{1}
$$
$$
CN_{AB} = \left\{ \begin{array}{cc} 1, & R_{AB} < k(R_A + R_B) \\ 0, & otherwise \end{array} \right. \tag{2}
$$
Of course, if we have access to the wavefunction of the system, we can define coordination numbers by e.g. bond orders instead, a possibility that I'll not discuss here.
Now, we can smoothen the Heaviside function Eq. (2):
$$
CN_A' = \sum_{B\neq A} CN_{AB}', \quad CN_{AB}' = f\left(\frac{R_{AB}}{k(R_A + R_B)}\right) \tag{3}
$$
where $f(x)$ is a continuous, monotonic function that quickly approaches 1 when $x<1$ and quickly approaches 0 when $x>0$. There are countless choices for such a function $f(x)$. One example, proposed and used in the famous DFT-D3 paper by Grimme et al., reads
$$
f(x) = \frac{1}{1+exp(-K(1/x-1))} \tag{4}
$$
where $K$ is a positive parameter. However, $f(x)$ does not decay to exactly zero when $x\to +\infty$, but approaches a very small positive number, meaning that the coordination number of an atom in a bulk environment is always infinite, although in realistically sized finite systems this divergence may not be numerically obvious. In the revised DFT-D4 model, they used the following form
$$
f(x) = \frac{C}{2}(1+erf(-K(x-1))) \tag{5}
$$
where, again, $K$ is a positive parameter, and $C$ is a parameter that depends on the electronegativity difference of the atoms $A$ and $B$. One may argue that only when $C=1$ can the resulting $CN_A'$ be interpreted as a coordination number, since only in this case does $CN_A'$ reduce to $CN_A$ in the $K\to +\infty$ limit. The reason Grimme et al. allowed $C$ to be different than 1 is that they use $CN_A'$ solely for calculating the DFT-D dispersion coefficients $C_6$, so it does not matter if $CN_A'$ does not satisfy all properties that an ideal "smooth coordination number" has to satisfy. In any case, one notices that Eq. (5) correctly vanishes in the $x\to +\infty$ limit, meaning that it is safe to use in huge finite systems or periodic systems.
Finally, it is clear that the continuous coordination number $CN_A'$ defined by Eq. (5) can be used in contexts other than the determination of $C_6$. For example, the very DFT-D4 paper itself used Eq. (5) in the calculation of atomic charges. Using the coordination numbers in yet other contexts may or may not have literature precedents, but since there is considerable flexibility in tuning the parameters $k$ and $K$ (or even the function $f(x)$), it is plausible that for most applications one can define a continuous coordination number that suits the particular application. However one may need to perform some fitting or hand-tuning to determine the optimum form of $f(x)$. For example, the $K$ in Eq. (5) controls how smooth the coordination number changes when $B$ dissociates from $A$, and the $k$ in Eq. (3) determines the bond distance at which $B$ contributes half a coordination number to $A$. The fitted parameters are not necessarily transferable to other applications, since different applications may have different requirements on the smoothness, turning point etc. of the coordination number function.
