Skip to content

metaSPAdes error 250 (OOM) triggers retry rather than finish#953

Merged
dialvarezs merged 2 commits intonf-core:devfrom
AlexHoratio:metaspades_e250
Dec 3, 2025
Merged

metaSPAdes error 250 (OOM) triggers retry rather than finish#953
dialvarezs merged 2 commits intonf-core:devfrom
AlexHoratio:metaspades_e250

Conversation

@AlexHoratio
Copy link
Copy Markdown
Contributor

@AlexHoratio AlexHoratio commented Dec 3, 2025

Hi!

When metaSPAdes fails to allocate memory due to specified limits, it crashes with an error 250. This should be caught by the retry function, so that it can do it again with a higher memory allocation. The same thing appears to be already implemented for the MEGAHIT process (which also OOMs with error code 250).

Here's a copy of the error I got:

ERROR ~ Error executing process > 'NFCORE_MAG:MAG:ASSEMBLY:SHORTREAD_ASSEMBLY:METASPADES (R004)'                                                                                                                                                            
                                                                                                                                                                                                                                                            
Caused by:                                                                                                                                                                                                                                                  
  Process `NFCORE_MAG:MAG:ASSEMBLY:SHORTREAD_ASSEMBLY:METASPADES (R004)` terminated with an error exit status (250)                                                                                                                                         
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
Command executed:                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                            
  spades.py \                                                                                                                                                                                                                                               
      --meta \                                                                                                                                                                                                                                              
      --threads 10 \                                                                                                                                                                                                                                        
      --memory 64 \                                                                                                                                                                                                                                         
       \                                                                                                                                                                                                                                                    
      -1 R004_1.merged.fastq.gz -2 R004_2.merged.fastq.gz   \                                                                                                                                                                                               
      -o ./                                                                                                                                                                                                                                                 
  mv spades.log SPAdes-R004.spades.log                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
  if [ -f scaffolds.fasta ]; then                                                                                                                                                                                                                           
      mv scaffolds.fasta SPAdes-R004.scaffolds.fa                                                                                                                                                                                                           
      gzip -n SPAdes-R004.scaffolds.fa                                                                                                                                                                                                                      
  fi                                                                                                                                                                                                                                                        
  if [ -f contigs.fasta ]; then                                                                                                                                                                                                                             
      mv contigs.fasta SPAdes-R004.contigs.fa                                                                                                                                                                                                               
      gzip -n SPAdes-R004.contigs.fa                                                                                                                                                                                                                        
  fi                                                                                                                                                                                                                                                        
  if [ -f transcripts.fasta ]; then                                                                                                                                                                                                                         
      mv transcripts.fasta SPAdes-R004.transcripts.fa                                                                                                                                                                                                       
      gzip -n SPAdes-R004.transcripts.fa                                                                                                                                                                                                                    
  fi                                                                                                                                                                                                                                                        
  if [ -f assembly_graph_with_scaffolds.gfa ]; then                                                                                                                                                                                                         
      mv assembly_graph_with_scaffolds.gfa SPAdes-R004.assembly.gfa                                                                                                                                                                                         
      gzip -n SPAdes-R004.assembly.gfa                                                                                                                                                                                                                      
  fi                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                            
  if [ -f gene_clusters.fasta ]; then                                                                                                                                                                                                                       
      mv gene_clusters.fasta SPAdes-R004.gene_clusters.fa                                                                                                                                                                                                   
      gzip -n SPAdes-R004.gene_clusters.fa                                                                                                                                                                                                                  
  fi                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                            
  if [ -f warnings.log ]; then                                                                                                                                                                                                                              
      mv warnings.log SPAdes-R004.warnings.log                                                                                                                                                                                                              
  fi                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                            
  cat <<-END_VERSIONS > versions.yml                                                                                                                                                                                                                        
  "NFCORE_MAG:MAG:ASSEMBLY:SHORTREAD_ASSEMBLY:METASPADES":                                                                                                                                                                                                  
      spades: $(spades.py --version 2>&1 | sed -n 's/^.*SPAdes genome assembler v//p')                                                                                                                                                                      
  END_VERSIONS                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                            
Command exit status:                                                                                                                                                                                                                                        
  250                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                            
Command output:                                                                                                                                                                                                                                             
    1:03:47.304    11G / 11G   INFO    General                 (kmer_splitter.hpp         : 102)   Using cell size of 4194304                                                                                                                               
    1:03:47.305    16G / 16G   INFO   K-mer Splitting          (kmer_data.cpp             :  98)   Processing "R004_1.merged.fastq.gz"                                                                                                                      
    1:04:04.757    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 2442333 reads                
    1:04:21.593    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 4813682 reads                                                                                                                                  
    1:04:38.486    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 7231632 reads                                                                                                                                  
    1:04:55.557    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 9649757 reads                                                                                                                                  
    1:05:13.246    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 12126262 reads                                                                                                                                 
    1:05:30.791    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 14619033 reads                                                                                                                                 
    1:05:47.808    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 17055434 reads                                                                                                                                 
    1:06:04.601    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 19473653 reads                                                                                                                                 
    1:06:21.265    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 21847890 reads                                                                                                                                 
    1:06:38.943    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 24329556 reads                                                                                                                                 
    1:07:48.914    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 34161579 reads                                                                                                                                 
    1:08:01.353    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             :  98)   Processing "R004_2.merged.fastq.gz"                                                                                                                      
    1:11:53.431    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 108)   Processed 68199306 reads                                                                                                                                 
    1:12:19.869    16G / 18G   INFO   K-mer Splitting          (kmer_data.cpp             : 113)   Total 71929862 reads processed                                                                                                                           
    1:12:20.258    11G / 18G   INFO    General                 (kmer_index_builder.hpp    : 264)   Starting k-mer counting.                                                                                                                                 
    1:12:58.275    11G / 52G   INFO    General                 (kmer_index_builder.hpp    : 275)   K-mer counting done. There are 1309851738 kmers in total.                                                                                                
    1:12:58.593    10M / 52G   INFO   K-mer Index Building     (kmer_index_builder.hpp    : 410)   Building perfect hash indices                                                                                                                            
    1:14:53.794   936M / 52G   INFO   K-mer Index Building     (kmer_index_builder.hpp    : 446)   Index built. Total 1309851738 kmers, 946070664 bytes occupied (5.77818 bits per kmer).                                                                   
    1:14:53.794   936M / 52G   INFO   K-mer Counting           (kmer_data.cpp             : 355)   Arranging kmers in hash map order                                                                                                                        
  mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (10485760000 bytes, address: 0x7464fe000000, alignment: 67108864, commit: 1)                                                                               
  mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (10485760000 bytes, address: 0x74628b000000, alignment: 67108864, commit: 1)                                                                               
    1:15:30.352    21G / 52G   INFO    General                 (main.cpp                  : 149)   Clustering Hamming graph.                                                                                                                                
    1:41:31.315    21G / 52G   INFO    General                 (main.cpp                  : 156)   Extracting clusters:                                                                                                                                     
    1:41:31.315    21G / 52G   INFO    General                 (concurrent_dsu.cpp        :  19)   Connecting to root                                                                                                                                       
    1:41:34.090    21G / 52G   INFO    General                 (concurrent_dsu.cpp        :  35)   Calculating counts                                                                                                                                       
  mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (36511416320 bytes, address: 0x745a07c00000, alignment: 67108864, commit: 1)                                                                               
    1:48:26.599    55G / 55G   INFO    General                 (concurrent_dsu.cpp        :  64)   Writing down entries                                                                                                                                     
  mimalloc: error: unable to allocate OS memory (10485760000 bytes, error code: 12 [Cannot allocate memory], address: (nil), large only: 0, allow large: 1)                                                                                                 
  mimalloc: process info:                                                                                                                                                                                                                                   
     elapsed:    6506.601 s                                                                                                                                                                                                                                 
     process: user: 56195.411 s, system: 3302.428 s, faults: 481                                                                                                                                                                                            
         rss: current: 61473775616, peak: 58505703424                                                                                                                                                                                                       
      commit: current: 61473775616, peak: 61540884480                                                                                                                                                                                                       
                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                            
  == Error ==  system call for: "['/opt/conda/bin/spades-hammer', 'corrected/configs/config.info']" finished abnormally, OS return value: -6                                                                                                                
  None                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                            
  In case you have troubles running SPAdes, you can report an issue on our GitHub repository github.com/ablab/spades                                                                                                                                        
  Please provide us with params.txt and spades.log files from the output directory.                                                                                                                                                                         
                                                                                                                                                                                                                                                            
  SPAdes log can be found here: spades.log                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                            
  Thank you for using metaSPAdes! If you use it in your research, please cite:                                                                                                                                                                              
                                                                                                                                                                                                                                                            
    Nurk, S., Meleshko, D., Korobeynikov, A. and Pevzner, P.A., 2017. metaSPAdes: a new versatile metagenomic assembler. Genome research, 27(5), pp.824-834.                                                                                                
    doi.org/10.1101/gr.213959.116

PR checklist

  • This comment contains a description of changes (with reason).

dialvarezs
dialvarezs approved these changes Dec 3, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@dialvarezs dialvarezs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @AlexHoratio!

@dialvarezs dialvarezs merged commit 664423a into nf-core:dev Dec 3, 2025
7 checks passed
@dialvarezs dialvarezs mentioned this pull request Dec 3, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dialvarezs dialvarezs dialvarezs approved these changes

@jfy133 jfy133 Awaiting requested review from jfy133 jfy133 is a code owner

@d4straub d4straub Awaiting requested review from d4straub d4straub is a code owner

@prototaxites prototaxites Awaiting requested review from prototaxites prototaxites is a code owner

@muabnezor muabnezor Awaiting requested review from muabnezor muabnezor is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AlexHoratio @dialvarezs