Skip to content

fix(kernelcache): detect enosys syscalls by instruction pattern matching#1178

Open
ThePraeceps wants to merge 1 commit intoblacktop:masterfrom
ThePraeceps:fix/enosys_detection
Open

fix(kernelcache): detect enosys syscalls by instruction pattern matching#1178
ThePraeceps wants to merge 1 commit intoblacktop:masterfrom
ThePraeceps:fix/enosys_detection

Conversation

@ThePraeceps
Copy link
Copy Markdown
Contributor

@ThePraeceps ThePraeceps commented Apr 16, 2026

Summary

ipsw kernel syscall has several failure modes relating to enosys:

  1. Syscalls with the function signature int syscall(void); are invalidly set to enosys stub
    • for example syscall 2 int fork()
  2. Functions can declare a function signature but be AUE_NULL and produce an enosys stub. These are incorrectly reported as valid syscalls.
    • for example syscall 429 429 AUE_NULL ALL { int audit_session_join(mach_port_name_t port); }
  3. Some syscalls have an actual implementation but return ENOSYS anyway:
int
mlockall(__unused proc_t p, __unused struct mlockall_args *uap, __unused int32_t *retval)
{
	return ENOSYS;
}
  1. The above functions would emit nonsensical arguments once correctly detected as enosys:
    • For example 429 0xfffffff00a26bea8: enosys munge=0xfffffff009f72adc ret=int narg=1 bytes=4 int enosys(mach_port_name_t port);

This code currently only supports ARM64 / AMD64 - I am uncertain what level of support you set for armv7 / i386 / ppc so have omitted these for now. These would still be processed but fall back to whatever the built in JSON declares.

I have went with byte matching as I expect the implementation to be stable. An alternative would be to leverage emulation similar to kernel cpp but that seemed a bit heavy for this use case.

I have speculatively left the BTI/ENDBR64 hardening from AI in the code but I suspect as a leaf function the compiler would never emit those here and I'm skeptical Apple would backport ENDBR64 within Tahoe.

Testing

I have tested the new processing against several x86 and arm64 kernels:

OS Version Build Devices
iOS 15.8.3 19H364 iPhone 7
macOS 15.4 25B78 M1 Macs
iOS 17.4.1 23E246 iPhone 12
macOS 26.3.1 25D2128 KDK - x86
macOS 26.3.1 25D2128 KDK - x86 Development
macOS 26.3.1 25D2128 KDK - Development M1
macOS 26.3.1 25D2128 KDK - Development M5

I manually verified the cases mentioned above in the x86 and iPhone 13 kernels to confirm they are enosys. Example difference in output:

4c4
< 2   0xfffffe0007b3cb2c: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 2   0xfffffe0007b3cb2c: fork                             munge=0x0                ret=int      narg=0 bytes=0  int fork(void);
22c22
< 20  0xfffffe0007b53d18: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 20  0xfffffe0007b53d18: getpid                           munge=0x0                ret=int      narg=0 bytes=0  int getpid(void);
26,27c26,27
< 24  0xfffffe0007b53e1c: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
< 25  0xfffffe0007b53e50: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 24  0xfffffe0007b53e1c: getuid                           munge=0x0                ret=int      narg=0 bytes=0  int getuid(void);
> 25  0xfffffe0007b53e50: geteuid                          munge=0x0                ret=int      narg=0 bytes=0  int geteuid(void);
38c38
< 36  0xfffffe00077b52f4: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 36  0xfffffe00077b52f4: sync                             munge=0x0                ret=int      narg=0 bytes=0  int sync(void);
41c41
< 39  0xfffffe0007b53d44: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 39  0xfffffe0007b53d44: getppid                          munge=0x0                ret=int      narg=0 bytes=0  int getppid(void);
44,45c44,45
< 42  0xfffffe0007ba1198: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
< 43  0xfffffe0007b53f48: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 42  0xfffffe0007ba1198: pipe                             munge=0x0                ret=int      narg=0 bytes=0  int pipe(void);
> 43  0xfffffe0007b53f48: getegid                          munge=0x0                ret=int      narg=0 bytes=0  int getegid(void);
49c49
< 47  0xfffffe0007b53f14: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 47  0xfffffe0007b53f14: getgid                           munge=0x0                ret=int      narg=0 bytes=0  int getgid(void);
83c83
< 81  0xfffffe0007b53d58: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 81  0xfffffe0007b53d58: getpgrp                          munge=0x0                ret=int      narg=0 bytes=0  int getpgrp(void);
87c87
< 85  0xfffffe0007c1f918: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 85  0xfffffe0007c1f918: swapon                           munge=0x0                ret=int      narg=0 bytes=0  int swapon(void);
91c91
< 89  0xfffffe0007b053b0: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 89  0xfffffe0007b053b0: sys_getdtablesize                munge=0x0                ret=int      narg=0 bytes=0  int sys_getdtablesize(void);
149c149
< 147 0xfffffe0007b5410c: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 147 0xfffffe0007b5410c: setsid                           munge=0x0                ret=int      narg=0 bytes=0  int setsid(void);
326,327c326,327
< 324 0xfffffe0007b404c8: mlockall                         munge=0xfffffe000776b774 ret=int      narg=1 bytes=4  int mlockall(int how);
< 325 0xfffffe0007b404d4: munlockall                       munge=0xfffffe000776b774 ret=int      narg=1 bytes=4  int munlockall(int how);
---
> 324 0xfffffe0007b404c8: enosys                           munge=0xfffffe000776b774 ret=int      narg=1 bytes=4  int enosys(void);
> 325 0xfffffe0007b404d4: enosys                           munge=0xfffffe000776b774 ret=int      narg=1 bytes=4  int enosys(void);
329c329
< 327 0xfffffe0007b54424: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 327 0xfffffe0007b54424: issetugid                        munge=0x0                ret=int      narg=0 bytes=0  int issetugid(void);
364c364
< 362 0xfffffe0007b16174: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 362 0xfffffe0007b16174: kqueue                           munge=0x0                ret=int      narg=0 bytes=0  int kqueue(void);
369c369
< 367 0xfffffe0007ade58c: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 367 0xfffffe0007ade58c: workq_open                       munge=0x0                ret=int      narg=0 bytes=0  int workq_open(void);
437c437
< 435 0xfffffe0007b9aa8c: nosys                            munge=0x0                ret=int      narg=0 bytes=0  int nosys(int pid);
---
> 435 0xfffffe0007b9aa8c: nosys                            munge=0x0                ret=int      narg=0 bytes=0  int nosys(void);
457c457
< 455 0xfffffe00077c88c4: enosys                           munge=0x0                ret=int      narg=0 bytes=0  int enosys(void);
---
> 455 0xfffffe00077c88c4: vfs_purge                        munge=0x0                ret=int      narg=0 bytes=0  int vfs_purge(void);

I have ran the test suite and all the kernel tests continued to pass.

AI Assistance

Initial code written by Claude Opus 4.6 then tested and bug fixed by hand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ThePraeceps