It's not talking about either, although it is related to those. They are restraining their system by adding a harmonic potential at the "boundary", which will prevent atoms from escaping.
A restraint is not the same as a constraint. We use the word constraint to mean a certain condition is strictly enforced. If we constrained the atoms to be within $r_B$ of the origin, they would never be allowed to have $r>r_B$. We use the word restraint to mean a softer condition; in this work, they add an energy penalty which disfavours configurations with $r>r_B$ (because they are high in energy) but doesn't disallow them.
Restraints mean that the desired situation, in this case that $r \leq r_B$, is favourable, but not enforced. A boundary constraint is like having a hard wall at the edge; the authors in this work are considering their boundary to be elastic, so atoms can actually deform the boundary by exchanging energy with it.
Restraints have some desirable properties, compared to constraints. They are smooth, continuous and differentiable, and this makes time-integrating the atomic dynamics quite straightforward. If the finite timestep causes an atom to go too far past $r_B$ here, for example, it just ends up with a slightly greater potential energy, and a restoring force acts to bring the atom back.
In contrast, a constraint is like having a "hard wall" potential, which is zero for $r<r_B$ and $\infty$ for $r\geq r_B$. This is neither smooth nor differentiable, makes the dynamics difficult to time-integrate and if an atom ever has $r\geq r_B$ the simulation essentially breaks down because the energy becomes infinite and there is no restoring force to bring the atom back inside the boundary. This is such a problem that in practice, hard wall boundary conditions are handled by special case sub-time-step code to try to determine the boundary collision time and reflect particles, but it can still cause problems.
