Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

`pymini` is an AST-based Python minifier for scripts and packages. It preserves
package layout by default, can emit a single-file bundle when asked, and can
shrink Python code by roughly `30%` to `90%` depending on the codebase and
whether you compare raw source or compressed archives.
shrink Python code by roughly `15%` to `70%` on the checked-in fixtures and
validated package benchmarks.

- [Getting Started](#getting-started)
- [Installation](#installation)
Expand Down Expand Up @@ -46,7 +46,7 @@ Current checked-in fixtures:
| `tests/examples/pyminifier.py` | `1,355` bytes | `511` bytes | `62.3%` |
| `tests/examples/pyminify.py` | `1,990` bytes | `981` bytes | `50.7%` |
| `TexSoup/` raw Python source (`*.py`) | `98,181` bytes | `33,107` bytes | `66.3%` |
| `TexSoup/` compressed source (`.tar.gz`) | `70,532` bytes | `11,850` bytes | `83.2%` |
| `TexSoup/` compressed source (`.tar.gz`) | `23,118` bytes | `11,368` bytes | `50.8%` |

For baseline comparisons, speed results, and TexSoup validation details, see
[benchmarks/README.md](./benchmarks/README.md).
Expand Down
45 changes: 16 additions & 29 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,39 +3,26 @@
This directory holds the current size and speed measurements for `pymini`, plus
the benchmark harness used to reproduce them.

- [Compression](#compression)
- [Speed](#speed)
- [Results](#results)
- [Reproduce](#reproduce)
- [TexSoup Validation](#texsoup-validation)

# Compression
# Results

Checked-in fixture comparison:
| Input | Original | `pymini` size | `pymini` speed | `pyminifier` size | `pyminifier` speed | `python-minifier` size | `python-minifier` speed |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `pyminifier.py` | `1,355` bytes | `511` bytes, `62.3%` | `2.4 ms` | `676` bytes, `50.1%` | `0.4 ms` | `1,020` bytes, `24.7%` | `1.6 ms` |
| `pyminify.py` | `1,990` bytes | `981` bytes, `50.7%` | `5.4 ms` | `1,605` bytes, `19.3%` | `1.2 ms` | `983` bytes, `50.6%` | `4.1 ms` |
| `TexSoup/*.py` | `98,181` bytes | `33,107` bytes, `66.3%` | `399.3 ms` | `34,643` bytes, `64.7%` | `31.3 ms` | `83,303` bytes, `15.2%` | `120.8 ms` |
| `TexSoup.tar.gz` | `23,118` bytes | `11,368` bytes, `50.8%` | `—` | `9,741` bytes, `57.9%` | `—` | `21,532` bytes, `6.9%` | `—` |

| Input | Original | `pymini` | `pyminifier` | `python-minifier` |
| --- | ---: | ---: | ---: | ---: |
| `tests/examples/pyminifier.py` | `1,355` bytes | `511` bytes, `62.3%` | `676` bytes, `50.1%` | `1,020` bytes, `24.7%` |
| `tests/examples/pyminify.py` | `1,990` bytes | `981` bytes, `50.7%` | `1,605` bytes, `19.3%` | `983` bytes, `50.6%` |
`TexSoup/*.py` compares validated package outputs. `pymini` uses package mode;
the baselines minify each file independently in the preserved package tree. All
three outputs pass the upstream TexSoup test suite (`78` tests).

TexSoup package validation:
# Reproduce

| Input | Original | `pymini` | Reduction |
| --- | ---: | ---: | ---: |
| `TexSoup/` raw Python source (`*.py`) | `98,181` bytes | `33,107` bytes | `66.3%` |
| `TexSoup/` compressed source (`.tar.gz`) | `70,532` bytes | `11,850` bytes | `83.2%` |

# Speed

Latency is machine-dependent. Recompute these with
`PYTHONPATH=. .venv/bin/python benchmarks/benchmark_speed.py`.

| Input | `pymini` | `pyminifier` | `python-minifier` |
| --- | ---: | ---: | ---: |
| `tests/examples/pyminifier.py` | `14.1 ms` | `0.4 ms` | `1.5 ms` |
| `tests/examples/pyminify.py` | `1227.6 ms` | `1.1 ms` | `4.0 ms` |
| `TexSoup/` package API | `4928.8 ms` | `—` | `—` |
| `TexSoup/` package CLI | `5062.0 ms` | `—` | `—` |

To reproduce those numbers:
Recompute the speed measurements with:

```bash
python3 -m pip install -e ".[dev]" python-minifier
Expand All @@ -48,9 +35,9 @@ PYTHONPATH=. .venv/bin/python benchmarks/benchmark_speed.py --pyminifier-root /t
`pymini` has been validated against the upstream `TexSoup` test suite in
package mode. Current validation: upstream pytest passes (`78` tests), raw
source code is `66.3%` smaller, and compressed source code (`.tar.gz`) is
`83.2%` smaller.
`50.8%` smaller when measured on clean `.py`-only package snapshots.

<!-- Raw bytes: 98,181 -> 33,107. Compressed bytes: 70,532 -> 11,850. -->
<!-- Raw bytes: 98,181 -> 33,107. Compressed bytes: 23,118 -> 11,368. -->

To reproduce that flow locally:

Expand Down
16 changes: 9 additions & 7 deletions pymini/pymini.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ def visit(self, node):
for child in ast.iter_child_nodes(node):
child.parent = node
self.visit(child)
return super().visit(node)
return node


class CommentRemover(NodeTransformer):
Expand Down Expand Up @@ -845,7 +845,6 @@ def transform(self, *trees):
)
imported.transform(tree)
append_public_aliases(tree, imported.nodes_to_append)
ParentSetter().visit(tree)
new_trees.append(tree)
return new_trees

Expand Down Expand Up @@ -918,8 +917,6 @@ def transform(self, *trees):
collector = RepeatedStringCollector()
collector.visit(tree)
RepeatedStringRewriter(self.generator, collector.repeated_strings_by_scope).visit(tree)
ParentSetter().visit(tree)
ast.fix_missing_locations(tree)
return trees


Expand Down Expand Up @@ -1080,10 +1077,7 @@ def __init__(self, generator):

def transform(self, *trees):
for tree in trees:
ParentSetter().visit(tree)
self.visit(tree)
ParentSetter().visit(tree)
ast.fix_missing_locations(tree)
return trees

def _next_safe_name(self, reserved_names):
Expand Down Expand Up @@ -1352,6 +1346,13 @@ def transform(self, *trees):
yield ast.unparse(tree)


class LocationFixer(Transformer):
def transform(self, *trees):
for tree in trees:
ast.fix_missing_locations(tree)
return trees


def module_prefixes(module: Optional[str]) -> List[str]:
if not module:
return []
Expand Down Expand Up @@ -1727,6 +1728,7 @@ def minify(sources, modules='main', keep_module_names=False,
),

# final post-processing to remove whitespace (minify)
LocationFixer(),
Unparser(),
WhitespaceRemover(),
)
Expand Down
Loading