It depends a lot on your application. All problems can be solved on a CPU, some problems can be solved much faster on a GPU.
One thing to consider is the effort required to write your code. If you're writing your own code from scratch, adapting it for the GPU can be a lot of work, and potentially for little to no reward. On the other hand, if you're using software packages or certain libraries, enabling GPU support might be as simple as checking a box.
Are you talking about configuring your own personal/work computer? How much do you plan on running your code on your machine itself (rather than a cluster, or dedicated workstation). If you're going to be running your code on a cluster, the amount of simulation you can get done on your personal machine won't matter much. In that case, you might want just enough CPU and GPU to do development and short tests.