Skip to content

Move binning entry test to binrefinement test, and fix binning entry+ concoct bug#473

Merged
prototaxites merged 6 commits intonf-core:devfrom
prototaxites:assembly_entry_test
Jul 13, 2023
Merged

Move binning entry test to binrefinement test, and fix binning entry+ concoct bug#473
prototaxites merged 6 commits intonf-core:devfrom
prototaxites:assembly_entry_test

Conversation

@prototaxites
Copy link
Copy Markdown
Contributor

@prototaxites prototaxites commented Jul 10, 2023 •
edited
Loading

test_binrefinement takes a long time to run - why not skip a whole bunch of steps!

Edit: In doing this, found a binning entry bug - the pipeline works internally with uncompressed assemblies, but I supplied the assemblies on the test-data site as gzipped (which is likely what users will have from a previous mag run). Added a step in setting up the assemblies channel which checks binning_entry fasta files for gzippedness (by file extension), and uncompresses them if so. Updated the docs accordingly.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [ ] If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jul 10, 2023 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 7520020

+| ✅ 156 tests passed       |+
#| ❔   1 tests were ignored |#
!| ❗   1 tests had warnings |!
Details

❗ Test warnings:

  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy

✅ Tests passed:

Run details

  • nf-core/tools version 2.9
  • Run at 2023-07-11 13:28:55

@prototaxites
Copy link
Copy Markdown
Contributor Author

prototaxites commented Jul 10, 2023 •
edited
Loading

Error running the CI tests:

ERROR ~ Error executing process > 'NFCORE_MAG:MAG:KRAKEN2 (test_minigut-minigut_kraken)'

Caused by:
  java.util.NoSuchElementException -- Check script './workflows/../modules/local/kraken2.nf' at line: 20

Source block:
  def input = meta.single_end ? "\"${reads}\"" :  "--paired \"${reads[0]}\" \"${reads[1]}\""
  prefix = task.ext.prefix ?: "${meta.id}"
  """
      kraken2 \
          --report-zero-counts \
          --threads ${task.cpus} \
          --db database \
          --report ${prefix}.kraken2_report.txt \
          $input \
          > kraken2.kraken
      cat kraken2.kraken | cut -f 2,3 > results.krona
  
      cat <<-END_VERSIONS > versions.yml
      "${task.process}":
          kraken2: \$(echo \$(kraken2 --version 2>&1) | sed 's/^.*Kraken version //' | sed 's/ Copyright.*//')
      END_VERSIONS
      """

Same error as #470 (comment), this time with Kraken2 instead of Centrifuge. Again, line 20 in kraken2.nf is prefix = task.ext.prefix ?: "${meta.id}" - something is unsafely modifying the meta map is my best guess!

@prototaxites
Copy link
Copy Markdown
Contributor Author

prototaxites commented Jul 11, 2023

Another error:

ERROR ~ Error executing process > 'NFCORE_MAG:MAG:BINNING:FASTA_BINNING_CONCOCT:CONCOCT_CUTUPFASTA (test_minigut)'

Caused by:
  Process `NFCORE_MAG:MAG:BINNING:FASTA_BINNING_CONCOCT:CONCOCT_CUTUPFASTA (test_minigut)` terminated with an error exit status (1)

Command executed:

  cut_up_fasta.py \
      SPAdes-test_minigut_contigs.fasta.gz \
       \
      -b SPAdes-CONCOCT-test_minigut.bed \
      > SPAdes-CONCOCT-test_minigut.fasta
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAG:MAG:BINNING:FASTA_BINNING_CONCOCT:CONCOCT_CUTUPFASTA":
      concoct: $(echo $(concoct --version 2>&1) | sed 's/concoct //g' )
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  Unable to find image 'quay.io/biocontainers/concoct:1.1.0--py38h7be5676_2' locally
  1.1.0--py38h7be5676_2: Pulling from biocontainers/concoct
  a3ed95caeb02: Already exists
  77c6c00e8b61: Already exists
  3aaade50789a: Already exists
  00cf8b9f3d2a: Already exists
  7ff999a2256f: Already exists
  d2ba336f2e44: Already exists
  dfda3e01f2b6: Already exists
  a3ed95caeb02: Already exists
  10c3bb32200b: Already exists
  6d0ba9e0d1d0: Pulling fs layer
  6d0ba9e0d1d0: Verifying Checksum
  6d0ba9e0d1d0: Download complete
  6d0ba9e0d1d0: Pull complete
  Digest: sha256:d608a4ac91935bd49aaee2f4d6b3277faf0dfcc1c74b48538b0b8a7c18647b8b
  Status: Downloaded newer image for quay.io/biocontainers/concoct:1.1.0--py38h7be5676_2
  Traceback (most recent call last):
    File "/usr/local/bin/cut_up_fasta.py", line 59, in <module>
      cut_up_fasta(args.contigs, args.chunk_size, args.overlap_size, args.merge_last, args.bedfile)
    File "/usr/local/bin/cut_up_fasta.py", line 16, in cut_up_fasta
      for record in SeqIO.parse(ff, "fasta"):
    File "/usr/local/lib/python3.8/site-packages/Bio/SeqIO/Interfaces.py", line 68, in __next__
      return next(self.records)
    File "/usr/local/lib/python3.8/site-packages/Bio/SeqIO/FastaIO.py", line 199, in iterate
      for title, sequence in SimpleFastaParser(handle):
    File "/usr/local/lib/python3.8/site-packages/Bio/SeqIO/FastaIO.py", line 47, in SimpleFastaParser
      for line in handle:
    File "/usr/local/lib/python3.8/codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Dumping here in case I lose it when the CI re-runs. Looking at the module, CONCOCT needs uncompressed assemblies - the binning entry spreadsheet supplies them as gzipped.

@prototaxites prototaxites changed the title Move binning entry test to binrefinement test Move binning entry test to binrefinement test, and fix binning entry+ concoct bug Jul 11, 2023
@prototaxites prototaxites mentioned this pull request Jul 11, 2023
Closed
@prototaxites prototaxites requested review from d4straub and jfy133 July 11, 2023 13:42
jfy133
jfy133 approved these changes Jul 13, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@jfy133 jfy133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea! One question but not a blocker if it's working!

Comment thread workflows/mag.nf
@prototaxites prototaxites merged commit 87761e0 into nf-core:dev Jul 13, 2023
@prototaxites prototaxites deleted the assembly_entry_test branch July 17, 2023 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jfy133 jfy133 jfy133 approved these changes

@d4straub d4straub Awaiting requested review from d4straub d4straub is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@prototaxites @jfy133