Skip to content

metrics: better seconds formatting#137

Merged
sfc-gh-sbekman merged 3 commits intomainfrom
stas/metrics-big-secs
Apr 10, 2025
Merged

metrics: better seconds formatting#137
sfc-gh-sbekman merged 3 commits intomainfrom
stas/metrics-big-secs

Conversation

@sfc-gh-sbekman
Copy link
Copy Markdown
Collaborator

@sfc-gh-sbekman sfc-gh-sbekman commented Apr 9, 2025 •
edited
Loading

Changing the secs format to do %H:%M:%S when it's more than 1 minute - e.g. 1M+ ulysses is ~3h step.

Here are examples of output after the change:

<1min:

iter:  1/50   2% | loss: 6.9812 | iter time: 40.219s | iter tflops: 2.8 | lr: 4.167E-07 | seqlen: 7.71K | step time: 38.539s | step tflops: 2.9 | epoch: 0
iter:  2/50   4% | loss: 7.0292 | iter time: 6.913s | iter tflops: 15.2 | lr: 8.333E-07 | seqlen: 7.42K | step time: 6.898s | step tflops: 15.3 | epoch: 0
iter:  3/50   6% | loss: 7.2154 | iter time: 7.107s | iter tflops: 15.3 | lr: 1.250E-06 | seqlen: 7.58K | step time: 7.062s | step tflops: 15.4 | epoch: 0

>1min:

iter:  1/50   2% | loss: 8.2158 | iter time: 0:02:35 | iter tflops: 302.4 | lr: 1.000E-06 | seqlen: 249.18K | step time: 0:02:31 | step tflops: 309.3 | epoch: 0
iter:  2/50   4% | loss: 8.1467 | iter time: 0:01:41 | iter tflops: 464.5 | lr: 2.000E-06 | seqlen: 249.18K | step time: 0:01:41 | step tflops: 464.7 | epoch: 0
iter:  3/50   6% | loss: 8.1814 | iter time: 0:01:42 | iter tflops: 458.9 | lr: 3.000E-06 | seqlen: 249.82K | step time: 0:01:42 | step tflops: 459.0 | epoch: 0

the only potential non-smooth thing will be if some training hovers around 1min - then it'd switch back and forth between 2 formats. Perhaps we could cache which format was used the first time and stick to it through? can improve in the future if it's bothersome.

cc: @sfc-gh-mwyatt

@sfc-gh-sbekman sfc-gh-sbekman enabled auto-merge (squash) April 9, 2025 17:42
@sfc-gh-sbekman sfc-gh-sbekman merged commit 283d9c8 into main Apr 10, 2025
5 checks passed
@sfc-gh-sbekman sfc-gh-sbekman deleted the stas/metrics-big-secs branch April 10, 2025 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-mwyatt sfc-gh-mwyatt sfc-gh-mwyatt approved these changes

@sfc-gh-jrasley sfc-gh-jrasley Awaiting requested review from sfc-gh-jrasley sfc-gh-jrasley is a code owner

@sfc-gh-caxu sfc-gh-caxu Awaiting requested review from sfc-gh-caxu

@sfc-gh-zhyao sfc-gh-zhyao Awaiting requested review from sfc-gh-zhyao

@sfc-gh-srajbhandari sfc-gh-srajbhandari Awaiting requested review from sfc-gh-srajbhandari

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sfc-gh-sbekman @sfc-gh-mwyatt