Fix integer overflow in RGB addframe byte-size calculation and add re…#106
Fix integer overflow in RGB addframe byte-size calculation and add re…#106rootvector2 wants to merge 1 commit intodloebl:mainfrom
Conversation
|
Hi @dloebl I know in my previous PR I made wrong assumptions and did not include a proper test. This time I rechecked everything carefully. I found an integer overflow in the RGB addframe size calculation. I reproduced it with large dimensions and added a minimal test file so you can verify the behavior directly. I ran the full test suite and everything passes. If anything still looks wrong please tell me and I will fix it properly. |
|
Hey @rootvector2 , Passing test suite(required): The tests are still failing for your changes. Please fix them. AI tooling attribution: If any AI models or tooling were used to produce this PR, please disclose that per our contributing guidelines (CONTRIBUTING.md:, item 5). |
ffec0f6 to
8e7176a
Compare
The multiplication fmtChan * numPixel in quantize_and_dither() and cgif_rgb_addframe() was computed in uint32_t arithmetic, which overflows for large RGBA images (e.g. 32768x32769x4 > UINT32_MAX). This caused undersized heap allocations followed by out-of-bounds writes. Fix by promoting the multiplication to size_t with an overflow check. Add regression test (rgb_overflow_trigger) that exercises the edge case.
8e7176a to
238a742
Compare
|
@dloebl |
…gression test