Added geNomad tool for viral classification of assemblies by PhilPalmer · Pull Request #364 · nf-core/mag

PhilPalmer · 2022-12-05T14:51:03Z

Hi,

Thanks for developing a great pipeline!

I have been using the nf-core/mag pipeline for a project where I want to investigate what percentage of the assemblies/reads are viral. I noticed that the pipeline is currently more focused on bacteria (eg using the GTD-Tk for taxonomic classification which only supports bacteria/archaea). I, therefore, decided to add a new (optional) process using a recently developed tool geNomad (see paper) which performs viral classification i.e. predicts which assemblies are viral and assigns them to taxa.

Would you be interested in including this tool within the pipeline?

Hopefully, the PR should be most of the way there. I have followed the nf-core contributing guidelines, tested the changes myself on a full-scale dataset and added CI tests. Some things that I have not done are:

Used a BioContainers Docker/Singularity container (because the tool is not yet available in BioContainers)
Updated the workflow schema/image (because I am not sure how you usually do this)

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
- If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  ~~- [ ] If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repository.~~
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

github-actions · 2022-12-05T14:52:44Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit d57ae9a

+| ✅ 156 tests passed       |+
#| ❔   1 tests were ignored |#
!| ❗   1 tests had warnings |!

Details

❗ Test warnings:

pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-mag_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-mag_logo_light.png
files_exist - File found: docs/images/nf-core-mag_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowMag.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-mag_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.4.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-mag_logo_light.png matches the template
files_unchanged - docs/images/nf-core-mag_logo_light.png matches the template
files_unchanged - docs/images/nf-core-mag_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (223 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.9
Run at 2023-07-20 15:21:09

jfy133 · 2022-12-05T15:58:23Z

Ill let Daniel and Sabrina decide on inclusion, but updating the workflow diagram is just done in inkscape. Feel free to ping me if you don't know how/don't have time and I can do it 👍

PhilPalmer · 2022-12-05T17:45:58Z

Great, thanks @jfy133,

I can try updating the workflow diagram. Should I update the figure that includes CheckM?

Also, I'm not sure why the virus classification CI test fails when running here during the mmseqs subprocess. The same command works locally so maybe it's to do with computational resources? I will look into this also

Co-authored-by: Carson J Miller <[email protected]>

Changed GENOMAD subworkflow to VIRUS_IDENTIFICATION, and added updated schematic

CarsonJM · 2023-07-10T14:41:12Z

@PhilPalmer would you please replace the images in the docs/images directory with the attached files? I am thinking this might be easier than me opening another PR! If there is a way I can directly edit the files, please let me know!

jfy133

Just need the image updated, then it's na OK from me

Co-authored-by: James A. Fellows Yates <[email protected]>

jfy133 · 2023-07-14T06:37:58Z

@prototaxites @d4straub if we get a ✔️ from you, then I guess we are good to go for a merge?

prototaxites · 2023-07-14T07:04:11Z

+    ================================================================================
+    */
+
+    if (params.run_genomad){


One last little one - should we now call this params.run_virus_identification?

Yep, that makes sense to me. Sorry for missing that on the last pass!

prototaxites

✅ from me in general! One last little comment, and I think the output docs still need updating to make sure they list the correct info.

d4straub

LGTM

CarsonJM · 2023-07-18T15:41:27Z

Submitted another PR to @PhilPalmer !

jfy133 · 2023-07-20T07:38:54Z

@CarsonJM once the PR to the fork is in, and tests pass here, you're welcome to merge :)

changed run_genomad to run_virus_identification, and updated output docs

CarsonJM · 2023-07-21T16:21:42Z

🚀🚀🚀
Thanks for all the help with this PR!

PhilPalmer added 26 commits December 2, 2022 13:29

Added genomad module

6197529

Added genomad module to mag workflow

7c4729f

Added genomad module configuration

d57c698

Added genomad params to config with defaults

06c4b7a

Added new genomad params to nf schema

9492ddf

Tidied container comment

a08bd69

Updated genomad module script vars

cd2fa24

Combine logs (remove existing log in case of resume)

0945f85

Added more output files

fc71946

Added genomad to citations

80dc583

Added output documentation for genomad

18d3530

Updated genomad output docs

5af83bb

Format with prettier

84414df

Formatted with fixed prettier settings

5cd6212

Fixed combining log files by preventing i/o error

c9b24da

Fixed removal of fasta file extension

d8e4b96

Fixed output file names for genomad

6460617

Fixed outputting files from genomad

6a1e452

Improved genomad docs

1fbd409

Added virus classification test

dbf610c

Added test virus classification profile

01a54df

Added genomad to readme

df6d56a

Made regex to remove fasta file extension more specific

40377d2

Made gh actions file prettier

32e8a27

Fixed typo in virus_classification param name

5c765d2

Added genomad to changelog

f6486ea

Remove short reads to speed up vir class test

8d65231

CarsonJM and others added 9 commits July 7, 2023 09:14

Add files via upload

8c0c193

Delete mag_workflow.png

e20fcf8

Delete mag_workflow.svg

e70a964

Add files via upload

c6dec13

Delete mag_workflow.svg

4f47158

Add files via upload

3573190

Update README.md

f0c2ca8

Co-authored-by: Carson J Miller <[email protected]>

Apply suggestions from code review

893b84f

Co-authored-by: Carson J Miller <[email protected]>

Merge pull request #7 from CarsonJM/genomad

9479cae

Changed GENOMAD subworkflow to VIRUS_IDENTIFICATION, and added updated schematic

prototaxites reviewed Jul 10, 2023

View reviewed changes

Comment thread docs/images/mag_workflow.png

CarsonJM reviewed Jul 10, 2023

View reviewed changes

Comment thread workflows/mag.nf Outdated

Update workflows/mag.nf

e689c87

CarsonJM reviewed Jul 10, 2023

View reviewed changes

Comment thread workflows/mag.nf Outdated

Update workflows/mag.nf

8f8ff9e

jfy133 approved these changes Jul 13, 2023

View reviewed changes

Comment thread README.md Outdated

PhilPalmer and others added 3 commits July 13, 2023 10:19

Update workflow diagram to include virus identification

8415eb7

Update README.md

ae5f87a

Co-authored-by: James A. Fellows Yates <[email protected]>

Merge branch 'dev' into genomad

4231a0c

prototaxites reviewed Jul 14, 2023

View reviewed changes

prototaxites approved these changes Jul 14, 2023

View reviewed changes

d4straub approved these changes Jul 17, 2023

View reviewed changes

changed run_genomad to run_virus_identification, nad updated output docs

e4b7583

Merge pull request #8 from CarsonJM/genomad

d57ae9a

changed run_genomad to run_virus_identification, and updated output docs

CarsonJM merged commit a8674de into nf-core:dev Jul 21, 2023

CarsonJM mentioned this pull request Sep 7, 2023

Release 2.4 #499

Merged

9 tasks

Conversation

PhilPalmer commented Dec 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

github-actions Bot commented Dec 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

jfy133 commented Dec 5, 2022

Uh oh!

PhilPalmer commented Dec 5, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CarsonJM commented Jul 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfy133 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jfy133 commented Jul 14, 2023

Uh oh!

prototaxites Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

CarsonJM Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

prototaxites left a comment

Choose a reason for hiding this comment

Uh oh!

d4straub left a comment

Choose a reason for hiding this comment

Uh oh!

CarsonJM commented Jul 18, 2023

Uh oh!

jfy133 commented Jul 20, 2023

Uh oh!

CarsonJM commented Jul 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PhilPalmer commented Dec 5, 2022 •

edited

Loading

github-actions Bot commented Dec 5, 2022 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

CarsonJM commented Jul 10, 2023 •

edited

Loading