When you're doing an electronic structure calculation, the input will include the following things:
- Geometry
- Basis set
- Number of electrons
- Spin multiplicity
If the spin multiplicity is not 1 (i.e. you are not dealing with a singlet state), then you're certainly dealing with an open-shell system.
The only case where this becomes highly non-trivial is when you're dealing with an open-shell singlet, meaning that there's a bunch of unpaired electrons, but overall the entire system still has a net spin multiplicity of 1.
Open shell singlets can be notoriously complicated. They arise (for example) in studies of one of the most difficult problems in all of quantum chemistry, which is to accurately study the electronic structure of FeMoco (arguably the most complicated enzyme known) as we discussed open-shell singlets in this paper. The answers to the question "Why is "open-shell singlet" not an oxymoron" are a good starting point to understand why such states arise.
So how do you know whether or not the system that you're studying needs to be modeled as an open-shell singlet?
Sometimes you know based on spectroscopic experiments what the occupation numbers of the valence orbitals should be (see the "configuration" column in my answer to: How to determine occupied and closed orbitals for a Molpro CASSCF calculation?). For example, you might already know by looking through the experimental literature, that the ground electronic state of the molecule in which you're interested has an equal (and non-zero) number of unpaired spin-up versus unpaired spin-down electrons.
What if there's not enough experimental data?
Then often the computational scientist's task is to do calculations for various values of the spin multiplicity (try making the molecule a singlet, triplet, etc.) and various electronic configurations to see which one gives the lowest energy. For very simple molecules you can infer what the ground state electronic configuration will be, but my answer to Total spin and/or multiplicity for transition metal ions? shows that even for a molecule with only two atoms (like the diatomic iron dimer) we don't know experimentally nor theoretically what the spin multiplicity of the ground electronic state is! My answer to the above-linked CASSCF question also shows examples where the ground state electronic configuration is still unknown. Another related thread is here: How to find out the multiplicities of molecules containing d and f block species?. So often we do not know in advance whether the system's ground state electronic configuration corresponds to an open-shell system or a closed-shell system, in fact we often don't even know the spin multiplicity so we frequently find ourselves doing calculations for many different configurations and attempting to determine which one has the lowest energy so that scientists in the future don't have to. In this regard, a literature search can be helpful: see if others have studied the same molecule and what they said about the open-shell vs closed-shell character of the ground electronic state.
Apart from what I've discussed above for ground electronic states, many people are also just interested in excited state chemistry, meaning that they're doing electronic structure calculations for many different states (including open-shell singlets, for example). In this case the decision is a lot easier :)
