Man Linux: Main Page and Category List


       SPANK - SLURM Plug-in Architecture for Node and job (K)control


       This  manual  briefly  describes  the capabilities of the SLURM Plug-in
       architecture for Node and job Kontrol (SPANK)  as  well  as  the  SPANK
       configuration file: (By default: plugstack.conf.)

       SPANK  provides  a  very generic interface for stackable plug-ins which
       may be used to dynamically modify the job launch code in  SLURM.  SPANK
       plugins  may  be  built  without access to SLURM source code. They need
       only be compiled against SLURM’s spank.h  header  file,  added  to  the
       SPANK  config  file  plugstack.conf, and they will be loaded at runtime
       during the next job launch. Thus,  the  SPANK  infrastructure  provides
       administrators  and  other developers a low cost, low effort ability to
       dynamically modify the runtime behavior of SLURM job launch.


       SPANK plugins are loaded in up to  three  separate  contexts  during  a
       SLURM job. Briefly, the three contexts are:

       local   In  local  context,  the  plugin  is  loaded by srun. (i.e. the
               "local" part of a parallel job).

       remote  In remote context, the plugin is loaded by  slurmd.  (i.e.  the
               "remote" part of a parallel job).

               In  allocator  context,  the plugin is loaded in one of the job
               allocation utilities sbatch or salloc.

       In  local  context,   only   the   init,   exit,   init_post_opt,   and
       user_local_init  functions  are  called. In allocator context, only the
       init, exit, and init_post_opt functions are called.  Plugins may  query
       the  context  in  which  they  are  running  with the spank_context and
       spank_remote functions defined in <slurm/spank.h>.

       SPANK plugins may be called from multiple points during the  SLURM  job
       launch. A plugin may define the following functions:

         Called just after plugins are loaded. In remote context, this is just
         after job step is initialized. This function  is  called  before  any
         plugin option processing.

         Called  at  the  same  point  as slurm_spank_init, but after all user
         options to the plugin have been processed. The reason that  the  init
         and  init_post_opt  callbacks  are  separated  is so that plugins can
         process system-wide options specified in plugstack.conf in  the  init
         callback,  then process user options, and finally take some action in
         slurm_spank_init_post_opt if necessary.

         Called in local (srun) context  only  after  all  options  have  been
         processed.   This  is  called  after  the  job  ID  and  step IDs are
         available.  This happens in srun after the allocation  is  made,  but
         before tasks are launched.

         Called  after  privileges  are  temporarily  dropped. (remote context

         Called for each  task  just  after  fork,  but  before  all  elevated
         privileges are dropped. (remote context only)

         Called for each task just before execve(2). (remote context only)

         Called  for  each task from parent process after fork(2) is complete.
         Due to the fact that slurmd does not exec any tasks until  all  tasks
         have  completed  fork(2),  this  call is guaranteed to run before the
         user task is executed. (remote context only)

         Called for each task as  its  exit  status  is  collected  by  SLURM.
         (remote context only)

         Called once just before slurmstepd exits in remote context.  In local
         context, called before srun exits.

       All of these functions have the same prototype, for example:

          int slurm_spank_init (spank_t spank, int ac, char *argv[])

       Where spank is the SPANK handle which must be passed back to SLURM when
       the  plugin  calls  functions  like  spank_get_item  and  spank_getenv.
       Configured arguments  (See  CONFIGURATION  below)  are  passed  in  the
       argument vector argv with argument count ac.

       SPANK  plugins  can  query  the  current  list of supported slurm_spank
       symbols to determine if the current version  supports  a  given  plugin
       hook.   This  may be useful because the list of plugin symbols may grow
       in the future. The  query  is  done  using  the  spank_symbol_supported
       function, which has the following prototype:

           int spank_symbol_supported (const char *sym);

       The return value is 1 if the symbol is supported, 0 if not.

       SPANK  plugins  do  not  have direct access to internally defined SLURM
       data structures. Instead, information about the currently executing job
       is obtained via the spank_get_item function call.

         spank_err_t spank_get_item (spank_t spank, spank_item_t item, ...);

       The spank_get_item call must be passed the current SPANK handle as well
       as the item requested, which is defined by the passed  spank_item_t.  A
       variable  number  of  pointer  arguments  are also passed, depending on
       which item was requested by the plugin. A list of the valid values  for
       item is kept in the spank.h header file. Some examples are:

         User id for running job. (uid_t *) is third arg of spank_get_item

         Job   step  id  for  running  job.  (uint32_t  *)  is  third  arg  of

         Exit status for exited task. Only valid  from  slurm_spank_task_exit.
         (int *) is third arg of spank_get_item.

         Complete  job  command  line. Third and fourth args to spank_get_item
         are (int *, char ***).

       See spank.h for more details, and EXAMPLES  below  for  an  example  of
       spank_get_item usage.

       SPANK   plugins  may  also  use  the  spank_getenv,  spank_setenv,  and
       spank_unsetenv functions to view  and  modify  the  job’s  environment.
       spank_getenv   searches  the  job’s  environment  for  the  environment
       variable var and copies the current value into a buffer buf  of  length
       len.  spank_setenv allows a SPANK plugin to set or overwrite a variable
       in the job’s environment,  and  spank_unsetenv  unsets  an  environment
       variable in the job’s environment. The prototypes are:

        spank_err_t spank_getenv (spank_t spank, const char *var,
                            char *buf, int len);
        spank_err_t spank_setenv (spank_t spank, const char *var,
                            const char *val, int overwrite);
        spank_err_t spank_unsetenv (spank_t spank, const char *var);

       These  are  only necessary in remote context since modifications of the
       standard  process   environment   using   setenv(3),   getenv(3),   and
       unsetenv(3) may be used in local context.

       Functions are also available from within the SPANK plugins to establish
       environment variables to be  exported  to  the  SLURM  PrologSlurmctld,
       Prolog,  Epilog and EpilogSlurmctld programs (the so-called job control
       environment).  The name of environment variables established  by  these
       calls  will  be  prepended with the string SPANK_ in order to avoid any
       security implications of arbitrary environment variable control. (After
       all, the job control scripts do run as root or the SLURM user.).

       These functions are available from local context only.

         spank_err_t spank_job_control_getenv(spank_t spank, const char *var,
                              char *buf, int len);
         spank_err_t spank_job_control_setenv(spank_t spank, const char *var,
                              const char *val, int overwrite);
         spank_err_t spank_job_control_unsetenv(spank_t spank, const char *var);

       See spank.h for more information, and EXAMPLES below for an example for
       spank_getenv usage.

       Many of the described  SPANK  functions  available  to  plugins  return
       errors  via  the  spank_err_t  error type. On success, the return value
       will be set to ESPANK_SUCCESS, while on failure, the return value  will
       be  set to one of many error values defined in slurm/spank.h. The SPANK
       interface provides a simple function

         const char * spank_strerror(spank_err_t err);

       which may be used to translate a  spank_err_t  value  into  its  string


       SPANK  plugins also have an interface through which they may define and
       implement extra job options. These options are made  available  to  the
       user  through SLURM commands such as srun(1), salloc(1), and sbatch(1).
       if the option is specified by the user,  its  value  is  forwarded  and
       registered with the plugin in slurmd when the job is run.  In this way,
       SPANK plugins may dynamically provide new options and functionality  to

       Each  option registered by a plugin to SLURM takes the form of a struct
       spank_option which is declared in <slurm/spank.h> as

          struct spank_option {
             char *         name;
             char *         arginfo;
             char *         usage;
             int            has_arg;
             int            val;
             spank_opt_cb_f cb;


       name   is  the  name  of  the  option.  Its  length   is   limited   to
              SPANK_OPTION_MAXLEN defined in <slurm/spank.h>.

              is  a  description  of the argument to the option, if the option
              does take an argument.

       usage  is a short description of the option suitable for --help output.

              0  if  option  takes no argument, 1 if option takes an argument,
              and  2  if  the  option  takes  an   optional   argument.   (See

       val    A  plugin-local value to return to the option callback function.

       cb     A callback function that is invoked when the  plugin  option  is
              registered   with   SLURM.   spank_opt_cb_f   is   typedef’d  in
              <slurm/spank.h> as

                typedef int (*spank_opt_cb_f) (int val, const char *optarg,
                                         int remote);

              Where val is the value of the  val  field  in  the  spank_option
              struct,  optarg  is  the  supplied  argument  if applicable, and
              remote is 0 if the function is being  called  from  the  "local"
              host (e.g. srun) or 1 from the "remote" host (slurmd).

       Plugin    options    may   be   registered   with   SLURM   using   the
       spank_option_register function. This function is only valid when called
       from the plugin’s slurm_spank_init handler, and registers one option at
       a time. The prototype is

          spank_err_t spank_option_register (spank_t sp,
                    struct spank_option *opt);

       This function will return ESPANK_SUCCESS on successful registration  of
       an  option,  or  ESPANK_BAD_ARG  for  errors  including invalid spank_t
       handle, or when the function is not called  from  the  slurm_spank_init
       function.  All options need to be registered from all contexts in which
       they will be used. For instance, if an option is  only  used  in  local
       (srun)  and remote (slurmd) contexts, then spank_option_register should
       only be called from within those contexts. For example:

          if (spank_context() != S_CTX_ALLOCATOR)
             spank_option_register (sp, opt);

       If,   however,   the   option   is   used   in   all   contexts,    the
       spank_option_register needs to be called everywhere.

       In  addition  to spank_option_register, plugins may also export options
       to SLURM by defining a table of struct  spank_option  with  the  symbol
       name spank_options. This method, however, is not supported for use with
       sbatch   and   salloc   (allocator   context),   thus   the   use    of
       spank_option_register is preferred. When using the spank_options table,
       the  final  element  in  the  array  must  be  filled  with  zeros.   A
       SPANK_OPTIONS_TABLE_END  macro  is provided in <slurm/spank.h> for this

       When an option is provided by the user on the local  side,  SLURM  will
       immediately  invoke  the option’s callback with remote=0. This is meant
       for the plugin to do local sanity checking of  the  option  before  the
       value is sent to the remote side during job launch. If the argument the
       user specified is invalid, the plugin should issue an error and issue a
       non-zero return code from the callback.

       On  the  remote  side,  options and their arguments are registered just
       after SPANK plugins are loaded and before  the  spank_init  handler  is
       called.   This   allows  plugins  to  modify  behavior  of  all  plugin
       functionality based  on  the  value  of  user-provided  options.   (See
       EXAMPLES below for a plugin that registers an option with SLURM).


       The default SPANK plug-in stack configuration file is plugstack.conf in
       the same directory as slurm.conf(5), though this may be changed via the
       SLURM  config  parameter  PlugStackConfig.  Normally the plugstack.conf
       file should be identical on all nodes of the cluster.  The config  file
       lists  SPANK  plugins,  one  per line, along with whether the plugin is
       required or optional, and any global arguments that are to be passed to
       the  plugin  for runtime configuration.  Comments are preceded with ’#’
       and extend to the end of  the  line.   If  the  configuration  file  is
       missing or empty, it will simply be ignored.

       The format of each non-comment line in the configuration file is:

         required/optional   plugin   arguments

        For example:

         optional /usr/lib/slurm/

       Tells  slurmd  to  load  the plugin passing no arguments.  If a
       SPANK plugin is required, then failure of any of the plugin’s functions
       will  cause  slurmd  to  terminate the job, while optional plugins only
       cause a warning.

       If a fully-qualified path is not  specified  for  a  plugin,  then  the
       currently configure PluginDir in slurm.conf(5) is searched.

       SPANK  plugins  are stackable, meaning that more than one plugin may be
       placed into the config file. The  plugins  will  simply  be  called  in
       order,  one  after  the  other, and appropriate action taken on failure
       given that state of the plugin’s optional flag.

       Additional config files or directories of config files may be  included
       in  plugstack.conf  with  the include keyword. The include keyword must
       appear on its own line, and takes a glob as its parameter, so  multiple
       files may be included from one include line. For example, the following
       syntax will load all config files  in  the  /etc/slurm/plugstack.conf.d
       directory, in local collation order:

         include /etc/slurm/plugstack.conf.d/*

       which  might  be  considered  a  more flexible method for building up a
       spank plugin stack.

       The SPANK config file is re-read on each job  launch,  so  editing  the
       config  file will not affect running jobs. However care should be taken
       so that a partially edited config file is not read by a launching  job.


       Simple SPANK config file:

       # SPANK config file
       # required?       plugin                     args
       optional                  min_prio=-10
       required          /usr/lib/slurm/

       The  following is a simple SPANK plugin to modify the nice value of job
       tasks. This plugin adds a --renice=[prio] option to  srun  which  users
       can  use  to set the priority of all remote tasks. Priority may also be
       specified via a SLURM_RENICE environment variable. A  minimum  priority
       may  be  established  via a "min_prio" parameter in plugstack.conf (See
       above for example).

        *   To compile:
        *    gcc -shared -o renice.c
       #include <sys/types.h>
       #include <stdio.h>
       #include <stdlib.h>
       #include <unistd.h>
       #include <string.h>
       #include <sys/resource.h>

       #include <slurm/spank.h>

        * All spank plugins must define this macro for the SLURM plugin loader.
       SPANK_PLUGIN(renice, 1);

       #define PRIO_ENV_VAR "SLURM_RENICE"
       #define PRIO_NOT_SET 42

        *  Minimum allowable value for priority. May be set globally
        *   via plugin option min_prio=<prio>
       static int min_prio = -20;

       static int prio = PRIO_NOT_SET;

       static int _renice_opt_process (int val, const char *optarg, int remote);
       static int _str2prio (const char *str, int *p2int);

        *  Provide a --renice=[prio] option to srun:
       struct spank_option spank_options[] =
           { "renice", "[prio]", "Re-nice job tasks to priority [prio].", 2, 0,
               (spank_opt_cb_f) _renice_opt_process

        *  Called from both srun and slurmd.
       int slurm_spank_init (spank_t sp, int ac, char **av)
           int i;

            /* Don’t do anything in sbatch/salloc
            if (spank_context () == S_CTX_ALLOCATOR)
                 return (0);

           for (i = 0; i < ac; i++) {
               if (strncmp ("min_prio=", av[i], 9) == 0) {
                   const char *optarg = av[i] + 9;
                   if (_str2prio (optarg, &min_prio) < 0)
                       slurm_error ("Ignoring invalid min_prio value: %s", av[i]);
               else {
                   slurm_error ("renice: Invalid option: %s", av[i]);

           if (!spank_remote (sp))
               slurm_verbose ("renice: min_prio = %d", min_prio);

           return (0);

       int slurm_spank_task_post_fork (spank_t sp, int ac, char **av)
           pid_t pid;
           int taskid;

           if (prio == PRIO_NOT_SET) {
                *  See if SLURM_RENICE env var is set by user
               char val [1024];

               if (spank_getenv (sp, PRIO_ENV_VAR, val, 1024) != ESPANK_SUCCESS)
                   return (0);

               if (_str2prio (val, &prio) < 0) {
                   slurm_error ("Bad value for %s: %s", PRIO_ENV_VAR, optarg);
                   return (-1);

           if (prio < min_prio)
               slurm_error ("%s=%d not allowed, using min=%d",
                   PRIO_ENV_VAR, prio, min_prio);

           if (prio < min_prio)
               prio = min_prio;

           spank_get_item (sp, S_TASK_GLOBAL_ID, &taskid);
           spank_get_item (sp, S_TASK_PID, &pid);

           slurm_info ("re-nicing task%d pid %ld to %ld", taskid, pid, prio);

           if (setpriority (PRIO_PROCESS, (int) pid, (int) prio) < 0) {
               slurm_error ("setpriority: %m");
               return (-1);

           return (0);

       static int _str2prio (const char *str, int *p2int)
           long int l;
           char *p;

           l = strtol (str, &p, 10);
           if ((*p != ’ ’) || (l < -20) || (l > 20))
               return (-1);

           *p2int = (int) l;

           return (0);

       static int _renice_opt_process (int val, const char *optarg, int remote)
           if (optarg == NULL) {
               slurm_error ("renice: invalid argument!");
               return (-1);

           if (_str2prio (optarg, &prio) < 0) {
               slurm_error ("Bad value for --renice: %s", optarg);
               return (-1);

           if (prio < min_prio)
               slurm_error ("--renice=%d not allowed, will use min=%d",
                            prio, min_prio);

           return (0);


       Copyright (C)  2006  The  Regents  of  the  University  of  California.
       Produced  at  Lawrence  Livermore National Laboratory (cf, DISCLAIMER).
       CODE-OCEC-09-009. All rights reserved.

       This file is  part  of  SLURM,  a  resource  management  program.   For
       details, see <>.

       SLURM  is free software; you can redistribute it and/or modify it under
       the terms of the GNU General Public License as published  by  the  Free
       Software  Foundation;  either  version  2  of  the License, or (at your
       option) any later version.

       SLURM is distributed in the hope that it will be  useful,  but  WITHOUT
       ANY  WARRANTY;  without even the implied warranty of MERCHANTABILITY or
       FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General  Public  License
       for more details.


       /etc/slurm/slurm.conf - SLURM configuration file.
       /etc/slurm/plugstack.conf - SPANK configuration file.
       /usr/include/slurm/spank.h - SPANK header file.


       srun(1), slurm.conf(5)