Skip to content

Release 1.1.10#437

Merged
jwhite242 merged 20 commits intomainfrom
repo_reset/release/1.1.10
Feb 6, 2024
Merged

Release 1.1.10#437
jwhite242 merged 20 commits intomainfrom
repo_reset/release/1.1.10

Conversation

@jwhite242
Copy link
Copy Markdown
Collaborator

Resetting/creating new branch structure to enable git flow/smoother release and mergebacks going forward. Main branch is replacing master due to history syncing issues on the release mergebacks into develop (1.1.7/1.1.8, etc end up on top due to some missed mergebacks/version ticks). Main has been spawned off the first 1.1.10dev commit. Master will essentially be frozen from this point on, using git flow with main and develop going forward.

jwhite242 and others added 20 commits June 2, 2022 16:39
Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.10.8 to 2022.12.7.
- [Release notes](https://github.com/certifi/python-certifi/releases)
- [Commits](certifi/python-certifi@2021.10.08...2022.12.07)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [cryptography](https://github.com/pyca/cryptography) from 37.0.1 to 38.0.3.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](pyca/cryptography@37.0.1...38.0.3)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
no scheduled scripts missig #! on lsf
* Updating versions in lock file.

* Increase minimum version of python to 3.8

* Remove missed 3.7 version from matrix.

* Update to readd 3.7 and remove fabric.

* Remove version limit on pytest.

* Updates to versions in lock.

* Add 3.11 to the test matrix

* Bump Sphinx version.
* Adds discussion on the philosophy and motivations of Maestros design: reproducible computational science

* Adds complete documentation of the current study specification

* Improved discussion of dependency types available in env block

* Documents Maestros minimal workflow language (tokens, step dependencies, ...)

* Expands documentation of how to schedule studies using various HPC schedulers

* Documentation of command line interface

* Comprehensive, progressive tutorial building Maestro workflows from scratch and running them

* Exposes multiple variants of the lulesh specs in the sample directory directly in the documentation

* Adds mini tutorial for porting existing HPC batch workflows and extending them with Maestro features

* Adds how-to guides with more advanced recipes/solutions

* Add mkdocstrings to autogenerate api docs from source as part of docs build

* Adds workflow topology diagrams using mermaid to better highlight runtime expansion of specs with parameters

* New documentation organization/layout inspired by divio documentation system

* Removes sphinx dependency

---------

Co-authored-by: Jeremy White <[email protected]>
Switch to importlib to get version for maestro's cli from pyproject.toml and remove old out of sync version source
* Pin mermaid to 9.4 due to api breaking change in 10.x

* Pin readthedocs poetry install to < 1.5 to avoid python2supports import failure in virtualenv pkg
Fixes dependabot flagged security issues:

* Updates lock file to update to certifi version that removes e-Tugra root certificate.

* Updates lock file to update requests dependency to patch unintended leak of proxy-authorization header
…it's resource/uri handling (#415)

Casts flux's JobID objects into string form to enable the dag to be pickled when conductor saves state. JobID objects are tied to flux's use of cffi and are not pickle-able themselves.

Base58 string form is used instead of the integer it's derived from so that logging is consistent with the view users have on the command line with flux reporting job id's in that format. This requires a separate re-instantiation of JobID objects upon waking to check the job status'.

Additional fixes:
* Improve the flexibility of resource specification in the flux adapter to align with the other adapters (i.e. not requiring nodes be specified).
* Change uri handling so env vars are not the only way to schedule jobs
* Updates to get working on flux only machines that require no bootstrapping or uri
  * Changes job submit call to be non-waitable by default on adapter 0.49 and newer as only owner of instance can do so (which is rarely the case on flux managed machines)
* Adds documentation and recipes for how to use Maestro with flux machines and flux instances running on non-flux scheduled machines
* Pins poetry version to 1.4.2 in CI to maintain support for python 3.7
* Fixes parsing of flux broker version to properly handle non-release versions of flux and logs that version into the flux batch script header comments and the maestro logs.
Adds pager functionality to rich rendered status layouts (not legacy layout) with ability to turn pager and theme on/off to remain compatible with systems/setups that don't have full support.
…hecking (#427)

* Adds packaging to toml file as it is now a required non-dev dependency.  Version is constrained to >=22.0 to reflect removal of LegacyVersion which can be returned from parse_version in older packaging versions.
* Improve robustness of flux adapter version error handling by testing against base_version to allow pre-release builds to succeed.
* Add version comparison unit tests for both full and base_version variants
Update v0.49 and v0.26 flux script adapters to adapt cancellation machinery to work with recent update to submit machinery that converts from flux JobID to native types.

Update general cancellation behaviors:
* Check for in progress steps before declaring cancelled successfully to delay until actual final states can be serialized
* Update cancel logic to mirror failure logic: mark all steps downstream of a cancelled step to also be cancelled
Slurm adapter bugfixes
* Adds prescribed column lists to both squeue and sacct to avoid parsing issues due to optional user environment variables 'SQUEUE_FORMAT' and 'SACCT_FORMAT'.
* Reduces some unneeded columns in sacct to make it more robust (not all machines have accounts, which broke the parsing)

Test additions
* Enabled detection of slurm scheduler for live test subset (sched_slurm marker) with auto disablement of marked tests to remain quiet in github ci
* Unit tests added for slurm adapter
  * check use of squeue and sacct job status checking options in presence of custom squeue/sacct format env vars
  * check functionality of job cancellation
* Slurm 'hello world' integration test
@jwhite242 jwhite242 merged commit 6fa62f1 into main Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants