Add mypy type hints for utils by emmuhamm · Pull Request #862 · DUNE-DAQ/drunc

emmuhamm · 2026-03-25T14:47:24Z

Description

Fixes issue #852

Update utils folder to follow mypy standards

This is the first step in introducing mypy to all of drunc. This PR only concerns itself with the utils folder.

The mypy and ruff settings that were used is in this commit: 2097249

It is highly recommended that those who want to add mypy to drunc also follows these settings.

Explanation of the ruff and mymy settings

Meaning of the ruff codes

D201 means “no blank line before class docstring.”
D202 means “no blank line after class docstring.”
D212 means “multi-line docstring summary should start at the first line.”
D205 means “blank line required between summary line and description.”

D + convention = "google" tells ruff to check docstrings against Google style

Meaning of the mypy codes

General ones

disallow_untyped_defs = true
- Every function and method must have type annotations.
disallow_incomplete_defs = true
- Partial annotations are not allowed. If you annotate one part of a function signature, the rest must be fully typed too.
check_untyped_defs = true
- Even functions without full type annotations still get type-checked internally.
no_implicit_optional = true
- A parameter is not allowed to silently mean Optional unless you spell it out.
warn_return_any = true
- mypy warns if a function returns Any.
warn_unused_ignores = true
- If a type: ignore is no longer needed, mypy will complain.
warn_redundant_casts = true
- mypy warns when a cast does nothing useful.
strict_equality = true
- mypy checks equality comparisons more carefully and flags suspicious ones.
disallow_any_generics = true
- You cannot use generic types without specifying their type arguments, like bare list or dict where a more specific type is expected.
disallow_subclassing_any = true
- You cannot subclass something mypy thinks is Any, because that usually means the base type is unknown.

Any-specific rules

disallow_any_explicit = true
- You cannot write Any directly in annotations unless you explicitly relax it somewhere else.
disallow_any_expr = false
- mypy is allowed to accept expressions that evaluate to Any. This is the important escape hatch here. It means you are not forcing the whole codebase to be clean of inferred Any yet.
disallow_any_decorated = true
- mypy rejects functions whose types become Any because of decorators.
disallow_any_unimported = true
- mypy rejects Any that comes from imports it could not properly follow or type-check.

It should be noted that we avoid changing the TOMLs in this PR. Therefore, if you run mypy right now, then these are the errors you might encounter

Details

With no changes in the toml, the output is as follows

src/drunc/utils/grpc_utils.py:15: error: Skipping analyzing "google.rpc": module is installed, but missing library stubs or py.typed marker  [import-untyped]

src/drunc/utils/configuration.py:8: error: Skipping analyzing "conffwk": module is installed, but missing library stubs or py.typed marker  [import-untyped]

src/drunc/utils/configuration.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 2 errors in 2 files (checked 6 source files)

Type of change

List of required branches from other repositories

N/A

Suggested manual testing checklist

To check if its safe to merge in develop

The usual

To check if the mypy checks work

revert 2981ad4. This reverts the revert of the toml, so you should be able to see the new and improved toml files.
Make sure you install mypy. Do this with pip install -e .[dev,test]
Run mypy src/drunc/utils/. All checks should pass
Run ruff check src/drunc/utils/. All checks should pass

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: np04-srv-028 from release Nightly from 16 April.

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker
- Without marker
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed
- Only concern yourself if failures related to drunc are in the log files
- If non-drunc failure appears:
  - Validate failure in fresh working area
  - Contact Pawel if unsure

Once the features are validated and both the unit and integration tests pass, the PRs is ready to be merged.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #daq-sw-librarians Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for #daq-sw-librarians Slack channel

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

jamesturner246 · 2026-03-26T13:28:42Z

Hey @emmuhamm. Just saw this flash past my inbox. You can also put mypy conficg inside the pypyoject.toml, so you don't have to add another file. As in

[tool.mypy] disable_error_code = ["import-untyped"]

I'd be more inclined to try and find and install the missing stubs from e.g. typeshed though.

Hi James, ah yes thanks thats useful. Didn't know about typeshed so will have a go through with that.

Yeah I was wondering if it were better to have the mypy stuff in a separate file or combined the toml. After thinking about it I think ur right, keeps everything centralised

Thank you for this discussion. For the linitng, I would suggest we leave that in a separate PR so that linting does not otherwise fail while we work on the mypy development. Having a configurattion in pyproject.toml would be better, but that can also be for a separate PR that closes #828

As for typeshed, I have left that as a suggestion in #828 as well. I think it would be very useful

emmuhamm · 2026-03-26T15:05:09Z

Note: when developing this it was discovered that ruff has two settings in drunc, one in .ruff.toml and another in the pyproject.toml.

We should probably choose which one we want to keep, and also probably propagate to mypy.

Basically

do we want settings to be all encompassed in pyproject? or do we want each thing to have its own file?

To be discussed

PawelPlesniak · 2026-03-27T14:11:19Z

This is great work, thanks @emmuhamm . I left some comments. Once they are addressed I will review, but this looks like the exact changes we need towards working with mypy.

Note: when developing this it was discovered that ruff has two settings in drunc, one in .ruff.toml and another in the pyproject.toml.

We should probably choose which one we want to keep, and also probably propagate to mypy.

Basically

do we want settings to be all encompassed in pyproject? or do we want each thing to have its own file?

To be discussed

These should be consolidated, if it can be done with the pre-commit workflows. I would expect that this can be done. The pyproject.toml contains more rigorous linting rules, I think we should go with those

PawelPlesniak · 2026-03-27T13:59:34Z

Thank you for this discussion. For the linitng, I would suggest we leave that in a separate PR so that linting does not otherwise fail while we work on the mypy development. Having a configurattion in pyproject.toml would be better, but that can also be for a separate PR that closes #828

As for typeshed, I have left that as a suggestion in #828 as well. I think it would be very useful

PawelPlesniak · 2026-03-27T14:07:27Z

 prod = ["paramiko[gssapi]"]
 dev = ["ruff", "pre-commit", "pytest", "pytest-cov", "grpcio-testing", "grpcio==1.75", "grpcio-tools==1.75", "grpcio-status==1.75"]
-test = ["pytest", "pytest-cov", "grpcio-testing", "grpcio==1.75", "grpcio-tools==1.75", "grpcio-status==1.75"]
+test = ["pytest", "pytest-cov", "grpcio-testing", "grpcio==1.75", "grpcio-tools==1.75", "grpcio-status==1.75", "mypy", "types-requests"]


Note that as per the mypy.ini comment, we will leave these until a dedicated PR for #828 gets created. In that PR, we will also need to add these to the dev settings so at commit times we can have these checks. May be worth adding something to the pre-commit config too.

This reverts commit 0ddb530b528403f637df530f6d578b818bc4ac2b.

emmuhamm · 2026-04-16T15:11:17Z

Hi @PawelPlesniak this is now ready for review. I've updated the description so hopefully thats all clear. lmk if somethings not clear :)

PawelPlesniak · 2026-04-28T13:11:48Z

Note no changes were introduced from the merge of develop in the utils dir

PawelPlesniak · 2026-04-28T15:21:01Z

Nice work on this. Integ tests are currently running. 2 questions

Was RichErrorServerInterceptor(grpc.ServerInterceptor): to RichErrorServerInterceptor: necessary for mypy?
You use the variable TYPE_CHECKING to control when some of the stubs are defined. Where is this defined, and how come it is not used in all the files?

PawelPlesniak · 2026-04-28T15:30:54Z

Manual tests passed
Integ tests passed on np04-srv-029

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Tue Apr 28 05:30:19 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260428164007.log

⮕ Running daqsystemtest/3ru_1df_multirun_test.py ⬅
=============================== 1 error 🚨 in 0.32s ===============================
⮕ Running daqsystemtest/3ru_3df_multirun_test.py ⬅
======================== 6 passed ✅ in 343.30s (0:05:43) =========================
⮕ Running daqsystemtest/example_system_test.py ⬅
======================== 12 passed ✅ in 365.53s (0:06:05) ========================
⮕ Running daqsystemtest/fake_data_producer_test.py ⬅
======================== 6 passed ✅ in 294.63s (0:04:54) =========================
⮕ Running daqsystemtest/long_window_readout_test.py ⬅
============================== 1 skipped 🟡 in 1.76s ==============================
⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
========================= 4 passed ✅ in 77.44s (0:01:17) =========================
⮕ Running daqsystemtest/readout_type_scan_test.py ⬅
======================== 33 passed ✅ in 874.98s (0:14:34) ========================
⮕ Running daqsystemtest/sample_ehn1_multihost_test.py ⬅
============================= 4 skipped 🟡 in 52.81s ==============================
⮕ Running daqsystemtest/small_footprint_quick_test.py ⬅
========================= 3 passed ✅ in 80.16s (0:01:20) =========================
⮕ Running daqsystemtest/tpg_state_collection_test.py ⬅
======================== 5 passed ✅ in 146.50s (0:02:26) =========================
⮕ Running daqsystemtest/tpreplay_test.py ⬅
======================== 6 passed ✅ in 172.60s (0:02:52) =========================
⮕ Running daqsystemtest/tpstream_writing_test.py ⬅
======================== 4 passed ✅ in 143.68s (0:02:23) =========================
⮕ Running daqsystemtest/trigger_bitwords_test.py ⬅
======================== 18 passed ✅ in 440.48s (0:07:20) ========================

emmuhamm self-assigned this Mar 25, 2026

emmuhamm linked an issue Mar 25, 2026 that may be closed by this pull request

[Feature]: mypy - utils #852

Open

emmuhamm added the Utilities label Mar 25, 2026

emmuhamm added this to the fddaq-v5.7.0 milestone Mar 26, 2026

jamesturner246 reviewed Mar 26, 2026

View reviewed changes

PawelPlesniak reviewed Mar 27, 2026

View reviewed changes

emmuhamm added 22 commits April 16, 2026 16:47

Add mypy to toml

12982b9

Fix most errors in utils.py

d4e7a9e

Add mypy to disable import-untyped

4eb841f

Fix most errors in shell_utils

93f8742

Change NoReturn to None

14b419d

Require explicit return statement

7a8866f

Fix oks key typehint

fd9f02a

[To discuss] Add protocols for shell utils

97e1243

Add stricter mypy settings

8084d67

Patch ruff toml for testing

3d44841

[fixup with pervious]

eb9f3cc

Fix utils with stricter requirements

e0a9d7a

Disable import untyped, use typeshed

45eee0b

fix grpc utils

3e12d8b

fix shell utils

de8ef0d

[to fixup] fix flask manager and config and others

dc3c047

Hopefully completely clear mypy and ruff

643fb5b

Remove missing imports for proper fixes

72fac55

Move mypy config to pyproject

bb51284

Fix some toml stuff

3a47c73

Be much stricter with Any

be97173

Minor formatting changes

24f4509

emmuhamm added 3 commits April 16, 2026 16:48

Revert all changes to tomls

dda74f8

Add tomls for typehinting

2097249

Revert "Add tomls for typehinting"

2981ad4

This reverts commit 0ddb530b528403f637df530f6d578b818bc4ac2b.

emmuhamm force-pushed the emmuhamm/mypy-utils branch from 015d0ad to 2981ad4 Compare April 16, 2026 15:09

emmuhamm marked this pull request as ready for review April 16, 2026 15:11

Merge branch 'develop' into emmuhamm/mypy-utils

b5e1f27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mypy type hints for utils #862

Add mypy type hints for utils #862
emmuhamm wants to merge 26 commits intodevelopfrom
emmuhamm/mypy-utils

emmuhamm commented Mar 25, 2026 •

edited by PawelPlesniak

Loading

Uh oh!

jamesturner246 Mar 26, 2026

Uh oh!

emmuhamm Mar 26, 2026

Uh oh!

PawelPlesniak Mar 27, 2026

Uh oh!

emmuhamm commented Mar 26, 2026

Uh oh!

PawelPlesniak commented Mar 27, 2026 •

edited

Loading

Uh oh!

PawelPlesniak Mar 27, 2026

Uh oh!

PawelPlesniak Mar 27, 2026

Uh oh!

emmuhamm commented Apr 16, 2026

Uh oh!

PawelPlesniak commented Apr 28, 2026

Uh oh!

PawelPlesniak commented Apr 28, 2026

Uh oh!

PawelPlesniak commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

emmuhamm commented Mar 25, 2026 • edited by PawelPlesniak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Meaning of the ruff codes

Meaning of the mypy codes

General ones

Any-specific rules

Type of change

List of required branches from other repositories

Suggested manual testing checklist

To check if its safe to merge in develop

To check if the mypy checks work

Developer checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for #daq-sw-librarians Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

jamesturner246 Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm commented Mar 26, 2026

Uh oh!

PawelPlesniak commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm commented Apr 16, 2026

Uh oh!

PawelPlesniak commented Apr 28, 2026

Uh oh!

PawelPlesniak commented Apr 28, 2026

Uh oh!

PawelPlesniak commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emmuhamm commented Mar 25, 2026 •

edited by PawelPlesniak

Loading

PawelPlesniak commented Mar 27, 2026 •

edited

Loading