# Are there critical mistakes to avoid when creating a workstation (32-128 cores)?

What are practical concerns for creating a workstation that will likely be expanded in the future to have more workstations/nodes?

I am interested in running atomistic and quantum mechanical calculations.

Most jobs would be 12 or fewer cores (nearly all would be 4-6 core jobs), but there would be times when many would be running simultaneously. It would be nice to control priority, but this isn't crucial.

The most used software would be psi4, Gaussian, GPAW, Quantum ESPRESSO and GROMACS. These would all be run on CPU's. In the future perhaps I would try GPU's for plane wave DFT and MD, but, Gaussian basis set electronic structure programs are not well suited for GPU's, yet.

My feeling is that a default Linux distribution would probably be okay, or perhaps Quantum Mobile in which case Windows would also be okay, but I don't know how well Quantum Mobile is for production runs, it advertises on desktops as educational and tutorial.

• This discussion has been moved to chat
– Tyberius
May 24 at 2:07

One mistake you already described is using virtualization. Nothing is more reliable than running your code directly on your hardware, without intermediaries (even when the virtualization software and Windows are evolving each day). It will be better to have a dual-boot system if you really need Windows than virtualization.

To avoid mistakes, you need to start reading the specifications of the code(s) you pretend to use.

Codes that use planewaves (like QuantumEspresso, ABINIT, etc.) are known to be huge RAM consumers (and I mean, they eat LOTS of RAM). On the other hand, codes like SIESTA uses much less RAM.

If running Molecular Dynamics calculations, must of the main codes used today (LAMMP, GROMACS, NAMD, DESMOND) use GPUs (and believe me, they are much more efficient than using CPUs).

If the code uses hard disk extensively, you will need high capacity hard disk and even SSD drives (or SCSI) to avoid write/read delays.

Finally, if you will add your workstation to a cluster, you will need a high speed network cards to avoid delays in the node-node communication.

• SSD raid might be the way to go for extensive hard disk usage May 20 at 17:54
• SCSI? Talk about a blast from the past... Jun 4 at 0:08

"Are there critical mistakes to avoid when creating a matter modeling workstation (32-128 cores)?"

32-128 is quite a big range since at 128 you'd be limited to non-mainstream chips like the Ampere and as far as I know, the most cores in an Intel processor apart from the Xeon Phi line (which was discontinued in 2020 and wasn't a conventional processor anyway) is 40 cores (80 threads). AMD has 64-core processors but AMD is far less in the "mainstream" for the type of workstation you're trying to build and you're probably better off with Intel. While it's true that AMD has their place in HPC, for example the $600 million El Capitan project, that's a custom-made system and what you'll get from the store is not going to be the same hardware that's going in El Capitan. I would stick with 40 cores or fewer, and if you want to run more than 3 of your 12-core jobs simultaneously, perhaps get two or three 40-core workstations rather than a single node with 128 cores. "What are practical concerns for creating a workstation that will likely be expanded in the future to have more workstations/nodes?" You said "Most jobs would be 12 or fewer cores, but there would be times when many would be running simultaneously" so I would strongly recommend avoiding the complexity of making a multi-node system connected via OmniPath, InfiniBand or ethernet (which would be much slower than the first two, which would unfortunately add a lot to the overall cost). At this scale (32-128 cores for multiple separate 12-core jobs) it would be best to have a few workstations rather than trying to build a multi-node supercomputer. It's true that some research groups have a "cluster" of connected nodes, but you'd be making things much more complicated and expensive (for the inter-node connectivity). If you're running separate 12-core jobs up to 128-cores worth, that's only 10 jobs and I'd rather manage 10 jobs on three 40-core workstations or four 32-core ones than pay so much more for InfiniBand and deal with the hassle of a multi-node system. "My feeling is that a default Linux distribution would probably be okay, or perhaps Quantum Mobile in which case Windows would also be okay, but I don't know how well Quantum Mobile is for production runs, it advertises on desktops as educational and tutorial." You said you're going to be "running atomistic and quantum mechanical calculations" for which all mainstream software is primarily tested on Linux machines, with Windows or MacOS functionality sometimes possible but not guaranteed. You may consider a dual-boot with Windows as Camps suggested for the few programs for which you need Windows, or if you're really considering to expand to 128-cores then you'll need three 40-core or four 32-core processors anyway, so one can be your Windows machine and the other two Linux. Quantum Mobile and others are mentioned in this previous question we had: Which Linux distribution is best for Matter Modeling? • This conversation has been moved to chat. May 20 at 17:57 • If you're considering that many physical cores, a dual-socket board is an option to get more in one machine. IDK if it's changed with Skylake Xeons, but going beyond dual socket gets really expensive. (e.g. high density servers usually run dual socket in 1U, and dual-socket workstation boards are a thing, but quad-socket boards are rarer and more expensive). And yeah, Intel's advantage here is AVX-512. If you're not taking advantage of that, Zen 2 or Zen 3 are very good CPUs. (Zen 1 has worse FMA throughput, don't consider AMD any older than Zen 2). May 21 at 1:24 • Welcome back Peter! Maybe I need to offer another bounty to entice you into turning that into something of an answer?! :) May 21 at 1:32 I would suggest to stay away from niche solutions and start with one mainstream machine. The most direct reason is that you get much more computing power for the same money if you buy mass-produced hardware. A more indirect reason is that everything you buy next year will be 20% cheaper or faster, your choice, cumulative for each year. If you want optimal performance, cater to the jobs you want to run now. In particular, as everybody knows a herd of greedy morons have created a GPU shortage. While I am not clairvoyant the situation may improve before you buy again, either because cryptocurrencies crash — in which case you'll be able to buy real cheap because the market will be flooded — or because the chip supply adapts. The most long-term reason is that you'll avoid locking yourself in. If you buy in a granular fashion you can cater to the workloads as they appear. A mainstream machine will be so cheap compared to your 128 core server that you may still be able to buy one next year if and when you really need it — and not if you don't. (I suppose you need to strike a deal with your budget director, shifting unused funds to the next year. But that's outside the scope of this question.) • I believe there is an error in saying everything you buy next year will be 20% cheaper or faster. For many years in the 2010s, hard drive prices were higher than in the past, due to floods in Thailand that disrupted the production centres, and it took years to recover. Likewise, hardware prices for personal machines were at the lowest circa November 2016, then they actually went up (for the same specs) for a while. COVID also caused prices to go up in many places due to increased demand for home-offices and difficulty with import/export. Now inflation is also making X dollars worth less. May 21 at 14:05 • @NikeDattani Barring floods and pandemics. May 21 at 14:20 Buying an Intel CPU would be one, for sure. (unless you are buying an Ice Lake server) I think the biggest decision to make is to build a CPU box or a GPU box. For atom-centered Gaussian basis set quantum chemistry you probably going to want a CPU box, with lots of RAM and a very fast and a high endurance SSD. • Just to clarify, buying an intel CPU would be a mistake or recommendation? May 21 at 19:19 • A mistake, all of their current workstation-class offerings are woefully outdated due their manufacturing issues around 10nm. May 21 at 19:23 • In the large HPC centers AMD machines are becoming more popular, but in previous years this was pretty much Intel-dominated. This implies that many codes were mostly used and optimized on Intel machines. Beyond the codes this is also true for the software infrastructure of libraries and compilers. Of course, this will change with the rise of AMD machines, but at the moment I have the impression that you have to have deep insights into the AMD hardware to fully exploit it. Cache infrastructure and so on is different. Some things also have to change in the source codes. May 21 at 21:51 ## To invest in new hardware, just to have it crippled by Intel MKL afterwards. This was object of a question I asked here previously. In matter modeling, it's usual for people to compile software themselves, instead of relying in precompiled binaries, so it can be tuned to the particular processor model in their machine, for example, taking advantage of more recent instruction set vector extensions like AVX or XOP to speed up calculations. By contrast, to ensure it can be run in just about every machine found in the wild, precompiled binaries rely on ancient SSE extensions, for example, thus not taking advantage of the full potential of new CPU models. The Intel compilers are quite popular in the field of matter modeling, and numeric computing in general. Said that, it comes with the downside of sabotaging the performance of machines equiped with hardware of competitors, by unreasonably refusing to use the new extensions, as explained in this wikipedia entry: MKL and other programs generated by the Intel C++ Compiler and the Intel DPC++ Compiler improve performance with a technique called function multi-versioning: a function is compiled or written for many of the x86 instruction set extensions, and at run-time a "master function" uses the CPUID instruction to select a version most appropriate for the current CPU. However, as long as the master function detects a non-Intel CPU, it almost always chooses the most basic (and slowest) function to use, regardless of what instruction sets the CPU claims to support. This has netted the system a nickname of "cripple AMD" routine since 2009. As of 2020, Intel's MKL, which remains the numeric library installed by default along with many pre-compiled mathematical applications on Windows (such as NumPy, SymPy). Although relying on the MKL, MATLAB implemented a workaround starting with Release 2020a which ensures full support for AVX2 by the MKL also for non Intel (AMD) CPUs In older versions, setting the undocumented environment variable MKL_DEBUG_CPU_TYPE=5 could be used to override the vendor string dependent codepath choice and activate supported instructions up to AVX2 on AMD processor based systems resulting in equal or even better performance when compared to Intel CPUs. Since at least Update 1 2020, the environment variable does not work anymore. I think the effects of this sabotage problem tends to get worse, because, as pointed by @GregorMichalicek comment to @uLoop answer, many people in the high performance and scientific computing community are switching to AMD, given the strides they made with the Zen architecture, while Intel faces issues with their manufacturing process. But perhaps not everybody is aware of this problem, to be able to switch their compiler and linear algebra libraries accordingly, toward alternatives not biased against any hardware, specially in small research teams with no dedicated IT staff. As examples of open source linear algebra libraries, we have ATLAS, OpenBLAS and BLIS. • I would like to point out another difference between AMD and Intel CPUs the user should be aware of. AMD CPUs have a larger overall L3 cache than Intel CPUs. But while this cache is shared among all cores in an Intel CPU, in AMD CPUs the cores are grouped and each group gets their own part of the L3 cache. Accessing the L3 cache from another group is very slow. This implies that thread pinning is much more important on AMD CPUs than on Intel CPUs. In an HPC center the administrators might take care of that, but on your own machine you are the administrator. May 23 at 19:58 • It is worth noting that a few MKL updates later, Intel actually started to add Zen specific optimized functions to MKL, that run at very reasonable speeds. Assuming this trend continues, MKL may actually work great on AMD. YMMV, but software vendors are now well aware of such shenanigans May 25 at 17:02 One critical mistake is to invest heavily in GPUs before doing a cost/benefit analysis. A single GPU may cost$2000, which can already get you 2-3 good compute nodes. Many codes (e.g. https://gaussian.com/g16/gpus.pdf) advertise speedups that are underwhelming: the GPU speedup should be at least an order of magnitude, when the CPU code is already close to optimal. If GPUs don't give you more performance than CPUs

A major point to consider is: who are you buying the workstation for? If there are several users, it may be better to buy several cheaper units than a big compute server, since the users' needs may conflict. IIRC GROMACS is CPU bound, while quantum chemistry and plane wave calculations may have big i/o demands. If you have several users trying to run large coupled-cluster calculations simultaneously, the performance will be poor since both are trying to use the limited i/o resources.