Update flux exclusive behavior modify task info passed to allocation/launcher by jwhite242 · Pull Request #467 · llnl/maestrowf

jwhite242 · 2026-03-12T07:03:13Z

Updates handling of exclusive keys to modify the batch directives and launcher rendering

Prunes everything but nodes from allocations
If set on launcher, overrides any cores per task, gpu flags, leaving nodes and tasks only
Jobspec creation via Python api now sets num_slots = nodes when exclusive is set, matching flux behavior

Updates default renderings:

cores per task omitted from launcher if not set in step keys
removes default 'nodes=1' in batch header/directives to sync with step keys
retains required >=1 for tasks/cores per task in jobspec creation but omits from rendered batch scripts to sync with flux default behaviors

Unblocks scheduling to nodes with configurable resources: subdividing gpus happens after allocation startup, and prior behavior of attaching tasks/gpus to allocation would cause flux to reject such jobs (unsatisfiable).

…ions and cores/gpus from launcher

…equirements to set nodes if exclusive also set

doutriaux1 · 2026-03-12T14:04:44Z

+                    ngpus_per_slot = int(ceil(ngpus / nodes))
+                else:
+                    ngpus_per_slot = None
+                LOGGER.warn(f"NODES recieved: {nodes}")


typo and also why is it a wraning? Shouldn't this be an info? (unless you want it in the else)

doutriaux1 · 2026-03-12T14:05:30Z


-        args = ["flux", "run", "-n", str(procs)]
-
+        args = ["flux", "run"]  #, "-n", str(procs)]


do we want to keep the commented out bit?

doutriaux1 · 2026-03-12T14:06:02Z

+        if nodes is not None and nodes != '':
+            args.append("-N")
+            args.append(str(nodes))
+        # ntasks = nodes if nodes else 1


same comment do we keep this?

doutriaux1 · 2026-03-12T14:06:55Z

+        # args.append(str(ntasks))
+
+        # NOTE: should we raise an exception here instead of just logging the error?
+        #       better pre-run validation might be more useful...


Yes I agree I'd raise an error since we log it, or downgrade the log to a warning.

doutriaux1 · 2026-03-12T14:07:43Z

+
+        if procs:
+            args.append("-n")
+            args.append(str(procs))


args.append(f"- n {procs}") ?

convention on all the adapters so far is each arg as a separate list item, following some of the subprocessing style arguments for shell commands, so going to leave it this way to be consistent

doutriaux1 · 2026-03-12T14:18:50Z

+            nodes = None
+        if nodes is not None and nodes != '':  # Have to account for empty string!
+            nodes = int(nodes)
+        # if not isinstance(nodes, int):


doutriaux1 · 2026-03-12T14:19:59Z


        # Calculate nprocs
-        ncores = cores_per_task * nodes
+        ncores = cores_per_task * processors  # What was this check doing?


seems like a big change, was it broken before?

allowing nodes to be 0 or None (never could be before) broke it.

will likely remove it entirely later.. not clear this check is useful given there's no place to eval against the machine... might only catch negative values now, which will be better checked upstream after future refactoring.

doutriaux1 · 2026-03-12T14:21:24Z

      conf:
        resource:
-          rediscover: True # Flux wants 'true', not 'True' that python bool serializes to
+          rediscover: True


removed the comment?

stray debris from prior pr: already fixed it so it's not a user facing problem anymore

doutriaux1 · 2026-03-12T14:21:36Z

    launcher_args:
      setopt:
-        fastload:
+        fastload: 


want the new space?

doutriaux1 · 2026-03-12T14:22:24Z

      conf:
        resource:
-          rediscover: "true" # Flux wants 'true', not 'True' that python bool serializes to
+          rediscover: "true" 


so string here but True above? Is it a flux issue?

either one works: it's json syntax. see 'render_additional_args' in the flux0_49_0.py for explanation

… the target

…tch script

… nodes

* Extra Flux Options (#459) * Add autoyes to cancel command (#461) * Add compact, sortable 'hashing' strategy for step ids/workspaces (#465) * Update flux exclusive behavior modify task info passed to allocation/launcher (#467)

Update flux exclusive behavior to prune tasks/cores/gpus from allocat…

9e77511

…ions and cores/gpus from launcher

jwhite242 changed the title ~~Update flux exclusive behavior to prune tasks/cores/gpus from allocat…~~ Update flux exclusive behavior modify task info passed to allocation/launcher Mar 12, 2026

Update test specs/data to reflect new exclusive behavior and flux's r…

eefd96a

…equirements to set nodes if exclusive also set

doutriaux1 approved these changes Mar 12, 2026

View reviewed changes

jwhite242 added 8 commits March 12, 2026 11:06

Fix up tables, add docs on new exclusive behavior

bfeac52

Update deprecation tags in docs to reflect impending 1.2.0 release as…

612fd78

… the target

Clean up exclusive table entry and add examples for both forms

e378b56

Document behavior change for flux allocation level exclusive flag

a98af8a

Finish exclusive overrides on python jobpsec creation to sync with ba…

3bc618e

…tch script

Cleanup testing debris

cad3b8c

Fix up exclusive test, clean up some old debugging debris

90ec1c7

Tick dev version

48d77f3

jwhite242 marked this pull request as ready for review March 13, 2026 18:36

jwhite242 mentioned this pull request Mar 13, 2026

Add controls for attaching nodes/task info to jobspecs and launchers #460

Closed

jwhite242 added 3 commits March 18, 2026 15:56

Add tests to verify exclusive flag properly modifies procs (slots) ==…

d824428

… nodes

Cleanup old code

17f7216

Minor comment fixups

254d2c8

jwhite242 merged commit 73fd565 into develop Mar 19, 2026
14 checks passed


		args = ["flux", "run", "-n", str(procs)]

		args = ["flux", "run"] #, "-n", str(procs)]

Conversation

jwhite242 commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jwhite242 commented Mar 12, 2026 •

edited

Loading