Are there critical mistakes to avoid when creating a workstation (32-128 cores)?

Question

What are practical concerns for creating a workstation that will likely be expanded in the future to have more workstations/nodes?

I am interested in running atomistic and quantum mechanical calculations.

Most jobs would be 12 or fewer cores (nearly all would be 4-6 core jobs), but there would be times when many would be running simultaneously. It would be nice to control priority, but this isn't crucial.

The most used software would be psi4, Gaussian, GPAW, Quantum ESPRESSO and GROMACS. These would all be run on CPU's. In the future perhaps I would try GPU's for plane wave DFT and MD, but, Gaussian basis set electronic structure programs are not well suited for GPU's, yet.

My feeling is that a default Linux distribution would probably be okay, or perhaps Quantum Mobile in which case Windows would also be okay, but I don't know how well Quantum Mobile is for production runs, it advertises on desktops as educational and tutorial.

Answer 1

One mistake you already described is using virtualization. Nothing is more reliable than running your code directly on your hardware, without intermediaries (even when the virtualization software and Windows are evolving each day). It will be better to have a dual-boot system if you really need Windows than virtualization.

To avoid mistakes, you need to start reading the specifications of the code(s) you pretend to use.

Codes that use planewaves (like QuantumEspresso, ABINIT, etc.) are known to be huge RAM consumers (and I mean, they eat LOTS of RAM). On the other hand, codes like SIESTA uses much less RAM.

If running Molecular Dynamics calculations, must of the main codes used today (LAMMP, GROMACS, NAMD, DESMOND) use GPUs (and believe me, they are much more efficient than using CPUs).

If the code uses hard disk extensively, you will need high capacity hard disk and even SSD drives (or SCSI) to avoid write/read delays.

Finally, if you will add your workstation to a cluster, you will need a high speed network cards to avoid delays in the node-node communication.

Answer 2

"Are there critical mistakes to avoid when creating a matter modeling workstation (32-128 cores)?"

32-128 is quite a big range since at 128 you'd be limited to non-mainstream chips like the Ampere and as far as I know, the most cores in an Intel processor apart from the Xeon Phi line (which was discontinued in 2020 and wasn't a conventional processor anyway) is 40 cores (80 threads). AMD has 64-core processors but AMD is far less in the "mainstream" for the type of workstation you're trying to build and you're probably better off with Intel. While it's true that AMD has their place in HPC, for example the $600 million El Capitan project, that's a custom-made system and what you'll get from the store is not going to be the same hardware that's going in El Capitan.

I would stick with 40 cores or fewer, and if you want to run more than 3 of your 12-core jobs simultaneously, perhaps get two or three 40-core workstations rather than a single node with 128 cores.

"What are practical concerns for creating a workstation that will likely be expanded in the future to have more workstations/nodes?"

You said "Most jobs would be 12 or fewer cores, but there would be times when many would be running simultaneously" so I would strongly recommend avoiding the complexity of making a multi-node system connected via OmniPath, InfiniBand or ethernet (which would be much slower than the first two, which would unfortunately add a lot to the overall cost).

At this scale (32-128 cores for multiple separate 12-core jobs) it would be best to have a few workstations rather than trying to build a multi-node supercomputer. It's true that some research groups have a "cluster" of connected nodes, but you'd be making things much more complicated and expensive (for the inter-node connectivity). If you're running separate 12-core jobs up to 128-cores worth, that's only 10 jobs and I'd rather manage 10 jobs on three 40-core workstations or four 32-core ones than pay so much more for InfiniBand and deal with the hassle of a multi-node system.

"My feeling is that a default Linux distribution would probably be okay, or perhaps Quantum Mobile in which case Windows would also be okay, but I don't know how well Quantum Mobile is for production runs, it advertises on desktops as educational and tutorial."

You said you're going to be "running atomistic and quantum mechanical calculations" for which all mainstream software is primarily tested on Linux machines, with Windows or MacOS functionality sometimes possible but not guaranteed. You may consider a dual-boot with Windows as Camps suggested for the few programs for which you need Windows, or if you're really considering to expand to 128-cores then you'll need three 40-core or four 32-core processors anyway, so one can be your Windows machine and the other two Linux. Quantum Mobile and others are mentioned in this previous question we had: Which Linux distribution is best for Matter Modeling?

Answer 3

I would suggest to stay away from niche solutions and start with one mainstream machine.

The most direct reason is that you get much more computing power for the same money if you buy mass-produced hardware.

A more indirect reason is that everything you buy next year will be 20% cheaper or faster, your choice, cumulative for each year. If you want optimal performance, cater to the jobs you want to run now. In particular, as everybody knows a herd of greedy morons have created a GPU shortage. While I am not clairvoyant the situation may improve before you buy again, either because cryptocurrencies crash — in which case you'll be able to buy real cheap because the market will be flooded — or because the chip supply adapts.

The most long-term reason is that you'll avoid locking yourself in. If you buy in a granular fashion you can cater to the workloads as they appear. A mainstream machine will be so cheap compared to your 128 core server that you may still be able to buy one next year if and when you really need it — and not if you don't. (I suppose you need to strike a deal with your budget director, shifting unused funds to the next year. But that's outside the scope of this question.)

Answer 4

Buying an Intel CPU would be one, for sure. (unless you are buying an Ice Lake server)

I think the biggest decision to make is to build a CPU box or a GPU box. For atom-centered Gaussian basis set quantum chemistry you probably going to want a CPU box, with lots of RAM and a very fast and a high endurance SSD.

Answer 5

To invest in new hardware, just to have it crippled by Intel MKL afterwards.

This was object of a question I asked here previously.

In matter modeling, it's usual for people to compile software themselves, instead of relying in precompiled binaries, so it can be tuned to the particular processor model in their machine, for example, taking advantage of more recent instruction set vector extensions like AVX or XOP to speed up calculations. By contrast, to ensure it can be run in just about every machine found in the wild, precompiled binaries rely on ancient SSE extensions, for example, thus not taking advantage of the full potential of new CPU models.

The Intel compilers are quite popular in the field of matter modeling, and numeric computing in general. Said that, it comes with the downside of sabotaging the performance of machines equiped with hardware of competitors, by unreasonably refusing to use the new extensions, as explained in this wikipedia entry:

MKL and other programs generated by the Intel C++ Compiler and the Intel DPC++ Compiler improve performance with a technique called function multi-versioning: a function is compiled or written for many of the x86 instruction set extensions, and at run-time a "master function" uses the CPUID instruction to select a version most appropriate for the current CPU. However, as long as the master function detects a non-Intel CPU, it almost always chooses the most basic (and slowest) function to use, regardless of what instruction sets the CPU claims to support. This has netted the system a nickname of "cripple AMD" routine since 2009. As of 2020, Intel's MKL, which remains the numeric library installed by default along with many pre-compiled mathematical applications on Windows (such as NumPy, SymPy). Although relying on the MKL, MATLAB implemented a workaround starting with Release 2020a which ensures full support for AVX2 by the MKL also for non Intel (AMD) CPUs

In older versions, setting the undocumented environment variable MKL_DEBUG_CPU_TYPE=5 could be used to override the vendor string dependent codepath choice and activate supported instructions up to AVX2 on AMD processor based systems resulting in equal or even better performance when compared to Intel CPUs. Since at least Update 1 2020, the environment variable does not work anymore.

I think the effects of this sabotage problem tends to get worse, because, as pointed by @GregorMichalicek comment to @uLoop answer, many people in the high performance and scientific computing community are switching to AMD, given the strides they made with the Zen architecture, while Intel faces issues with their manufacturing process. But perhaps not everybody is aware of this problem, to be able to switch their compiler and linear algebra libraries accordingly, toward alternatives not biased against any hardware, specially in small research teams with no dedicated IT staff. As examples of open source linear algebra libraries, we have ATLAS, OpenBLAS and BLIS.

Answer 6

One critical mistake is to invest heavily in GPUs before doing a cost/benefit analysis. A single GPU may cost $2000, which can already get you 2-3 good compute nodes. Many codes (e.g. https://gaussian.com/g16/gpus.pdf) advertise speedups that are underwhelming: the GPU speedup should be at least an order of magnitude, when the CPU code is already close to optimal. If GPUs don't give you more performance than CPUs

A major point to consider is: who are you buying the workstation for? If there are several users, it may be better to buy several cheaper units than a big compute server, since the users' needs may conflict. IIRC GROMACS is CPU bound, while quantum chemistry and plane wave calculations may have big i/o demands. If you have several users trying to run large coupled-cluster calculations simultaneously, the performance will be poor since both are trying to use the limited i/o resources.