6
$\begingroup$

Bellow is a part from the partitions of an HPC. Can someone please explain to me the meaning of each term ( mix, alloc, adle, node28, bigmem[41-44], gpu[51-52].....)

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST 
defq         up    1:00:00      1    mix node28 
defq         up    1:00:00     34  alloc bigmem[41-44],data[31-36],node[01-12,14-16,21-27,29-30] 
demoq        up   infinite      1   idle node13 
gpu-prodq    up 7-00:00:00      2    mix gpu[51-52] 

I want also to know how can I check if a partition is available for usage because a lot of times when I start a calculation I have to wait so long until it starts running.

$\endgroup$
1
  • 1
    $\begingroup$ It might be better to look at the HPC manual for your specific cluster, as the names can vary a bit depending on the exact system configuration. $\endgroup$
    – S R Maiti
    Jul 11, 2023 at 14:46

3 Answers 3

5
$\begingroup$

To answer your second question first:

I want also to know how can I check if a partition is available for usage because a lot of times when I start a calculation I have to wait so long until it starts running.

There are really only four possibilities:

  1. The cluster is very busy and everyone has to wait a long time.
  2. You have run many jobs recently, so future jobs from you will have low priority so other users get a turn;
  3. You are requesting unusual or very large allocations which are difficult to accommodate;
  4. Your cluster scheduler's settings are buggy.

For situation 1 or 2 there's nothing you can (or should) do. Sit back, read some papers, do some post-analysis, go out and smell the fresh air.

For situation 3, make sure your requests are "efficient" -- that is, you're requesting enough resources to get the job done but not too much more, and request either few cores or a short wall-time. It's a basic logistics problem: if you ask for a large amount of resources for a very long time, it will naturally be harder to fit in your requests. To do this well, you must know your code's:

  • "checkpoint" features. Most HPC codes can save a calculation checkpoint midway through and then resume the calculation in a subsequent run. This lets you break up a big calculation into smaller sub-runs which are easier to schedule.
  • resource consumption. Does your code need lots of CPU (like most molecular dynamics codes) or lots of RAM (like some bioinformatics jobs)? Requesting less of a resource that you don't need much of will make your job easier to schedule. (If I explicitly ask for very little RAM for my MD job, the scheduler might be able to squeeze my job in next to a bioinformatics job.)
  • scaling characteristics. My code might only run 1.3x as fast if I run it on two nodes instead of one, so instead of requesting a two-node job, I might be better off requesting a one-node job and being patient.

For situation 4, it's entirely out of your control too. Please make sure you try everything else before filing a bug report. And even if the cluster scheduler settings are bugged, you can usually make some headway by requesting smaller jobs. I was recently in a situation on a cluster which, for some reason, was struggling to allocate jobs longer than 12 hours. No problem -- I just split up my jobs and then waited my turn alongside everyone else.


Now, the sinfo table you've posted has a few columns, whose names seem quite straightforward:

  • The partitions are the various queues you can request jobs in which typically provide different resources or limits:

    • Most clusters will have a default main queue, usually called work or prod (production), where most jobs should go.
    • A debug queue has very fast turnaround but you can only submit small, short jobs, so you can quickly test scripts for obvious bugs before submitting them to the main queue.
    • Some clusters have copy queues reserved for moving lots of data.
    • Other queues have special resources or settings:
      • Large-RAM jobs go on bigmem or himem queues.
      • Some clusters have small long queues which let jobs run for two weeks where their default queue has a day-long limit.
      • Clusters with GPUs will usually put them in a gpu queue (one cluster I worked with had the good stuff locked away in an ai queue).
  • If the availability of a partition is down you won't be able to submit jobs there. Grant writing time!

  • Those are some interesting time limits. The defq limit of 1 hour is probably for debugging, and that 7 day time limit is the longest I've ever seen for a GPU production queue. The infinite limit probably means you're not allowed to submit to demoq at all -- it's strictly for admins to run demonstrations, I'm guessing.

  • The next column just says the number of nodes with each state:

  • The state is the one column that stays the same across every cluster. Nodes are alloc if they're fully allocated, idle if they're not allocated at all, and mixed in between.

    • DO NOT "game" the system by trying to specify idle nodes. The scheduler knows which nodes are idle. If it's not putting your job on, that's happening for some other reason. You're just making your submit scripts more complex and bug-prone for no benefit.
  • And the last column is simply the list of nodes under each state. For example the first entry in the second row says that nodes bigmem41 to bigmem44 are currently mixed-allocated.

    • Most clusters will name their nodes sensibly but in principle the names could be anything. On one cluster I use, called Bunya, the default node names are just bun000 and so on. In the info you provided, the default node name is nodeXX. Non-default node names will usually reflect node-specific resources -- you can guess what bigmem and gpu mean, and data probably denotes nodes for large data transfers.
$\endgroup$
2
  • $\begingroup$ Thank you so much, that's amazing :) . Is there a way to know how many cores are available in each node, please? $\endgroup$
    – Camilla
    Jul 18, 2023 at 11:15
  • $\begingroup$ Your cluster should have a user guide or website which will tell you about the nodes and queues available. Your cluster's admins should also be reachable via your university or national institution's IT email. $\endgroup$ Jul 18, 2023 at 11:28
3
$\begingroup$

I do not know how the specific cluster is set up in your case, but usually sinfo would give you list of all nodes currently available in the cluster, and then you can do a sinfo | grep idle to get the nodes that are currently free. Any node indicated as mix would have some part of the processors occupied by a different job. alloc is self-explanatory.

$\endgroup$
1
  • $\begingroup$ Thank you so much for the answer, that's useful :) $\endgroup$
    – Camilla
    Jul 18, 2023 at 11:19
3
$\begingroup$

What do the terms mean?

Alloc - a node state where all the cpu cores are allocated. It's "full" essentially

Mix - Node is partially full

Idle - nothing running on that node

Bigmem - a node type, these are assigned by the cluster admin, so can't be compared between clusters, but it probably has more memory than a standard node. We can see here that there are 3, and they're all in use

GPU - a node with a graphics card. Used for jobs that need GPUs. It's in a separate queue (gpu-prodq) to stop non gpu jobs using it

node - another node type - the standard node for this cluster

data - another node type. It probably has either expanded storage for large jobs, or is meant to be used for file transfer or data operation jobs, or runs a database. Look at the cluster docs to figure this out

I think where some confusion is coming in is that you're reading "node28" as a type of node - it isn't, it's the 28th instance of the standard type of node. The numbers in [] are ranges, so bigmem[41-43] means bigmem nodes 41,42 and 43

How to get your job to queue faster?

Looking at the snippet you posted, essentially everything is allocated in the cluster, barring a few cores on node28. Node13 isn't doing anything, but it's part of a different queue, probably reserved for admin, or people with special permissions to run really long jobs.

If your runtime is longer than 1 hour, your job will never get allocated, unless you are using a GPU, because no only the "demoq" allows this, and you probably don't have access.

There's a very small number of high memory nodes, and, worse, they're in the standard queue. If your job requires a bunch of memory, you have to wait until one of these nodes becomes free, but other jobs can also be allocated there. So your job wait time is, in the worst case "wait to reach the front of the queue, and then wait until a high memory node frees up" - the scheduler will try and be smart about allocation, but it's imperfect.

To get your jobs to start faster, you need to work out the resources they need - look at how much memory they use, and if they effectively make use of all the cores and memory you assign them. If not, reduce the requirements in your submission script.

It is also useful to be aware of the specifications of each class of node. If a standard node has 64Gb of memory, and you're looking for 65Gb, it'll bump your job to competing for one of the high memory nodes. Similarly, if you request 10 cores, and a standard node has 8, you'll be waiting for a node with higher core count. If you request resources no node in the cluster has, your job may enter an infinite, never running state, where it will just sit in the queue, and never be scheduled.

Do you have a support team? Generally, having been HPC support, we like answering questions about using the cluster efficiently. The sheer number of times someone's single threaded code has requested 16 cores is quite impressive. They might be able to direct you to somewhere that can show you stats for your previous jobs, and you can check if you can reduce the resources you're requesting. Reducing requirements has a huge, huge effect on how fast things will be scheduled. It's also being a good citizen, as other people's jobs will be run faster.

The big TL:DR is, I think the same as @Shen Ree Tee's - you have no control over how busy the cluster is, and pretty much the only way to speed up your job queuing is to make sure you're only requesting as much ram or cpu as you need, and not asking the queue to assign impossible jobs.

Edit: the only other possible speedup is if you have a job that can be split up. Adding multiple small jobs to the queue, rather than one big one, increases their ability to run alongside other jobs. It's a bit like tables at a restaurant - if you show up with a party of 16, it's going to take ages to get seated. If you are happy to be 8 groups of 2, you can be slotted in around everyone else, and get seated faster.

$\endgroup$
1
  • $\begingroup$ Thank you so much your answer is very helpful :) $\endgroup$
    – Camilla
    Jul 18, 2023 at 11:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .