Neal Fultz

Inference scaling is quietly becoming one of the most important ideas in modern LLMs. It lets you trade extra compute at inference time for major gains in accuracy, all without touching model weights.

“Usually, the first idea doesn't check out. That's just the nature of research.” For Derek Li, Senior Researcher at Noah's Ark Lab, that first idea was training all multitask RL objectives together. What went wrong: Runs underperformed baselines, with entropy collapsing and reward signals (like creative writing) diverging early. The fix: Moving from joint training to sequential training with constraints, guided by forgettability and backward transfer metrics. The lesson: In multitask RL, backward transfer, KL divergence, and entropy are the key signals that expose task interference early. _ (You can find the link to Derek’s ICML paper in the comments below)

3

1 Comment

We just published our first technical research note. "Low-Latency Event-Driven Architecture for Real-Time Options Execution on IBKR" It covers how we eliminated compounding feed lag at SPY market open — from a growing 45-second backlog to stable sub-second bar sealing — through: → Queue-first callback offloading (Python GIL problem) → Per-venue DOM isolation across ARCA, ISLAND, IEX → Event-time binning across 6 independent intervals → Single source of truth for all technical indicators No signals. No alpha. Pure infrastructure. This is the kind of work that happens before a single trade fires — and the kind that determines whether your system survives the opening bell. 📄 Download: https://lnkd.in/g-XJsYwG #AlgoTrading #OptionsTrading #MarketMicrostructure #SystematicTrading #Python #QuantitativeFinance #TradingInfrastructure

1

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Stanford Center for Research on Foundational Model's Marin project has released the first fully open model in JAX. It’s an 'open lab' sharing the entire research process - including code, data, and logs, to enable reproducibility and further innovation. Check out the project: https://goo.gle/44AMeLY

41

2 Comments

A team led by The Data Science Institute at Columbia University's postdoctoral fellow Eric Feltham set out to understand how people perceive the social ties in their communities—and how those perceptions differ from reality. Feltham's study, coauthored by Laura Forastiere and Nicholas Christakis, was published in Nature Portfolio Human Behaviour and is based on in-person interviews with more than 10,000 adults across 82 rural Honduran villages — one of the largest field studies of social perception to date. Read more about it: https://lnkd.in/e7Ywy_rX

38

😎 New blog post up summarizing our research team's work on PyGim, a highly performant, open-source library for efficient graph neural network execution. Thanks to lead author Christina Giannoula and co-authors Peiming Yang, Ivan Fernandez, Jiacheng Yang, Sankeerth Durvasula, Yu Xin Li, Mohammad Sadrosadati, Juan Gomez Luna, Onur Mutlu, and Gennady Pekhimenko. Read it now: https://lnkd.in/gUP7-68j

8

Streamline your machine learning workflows with CI/CD in Julia. From model validation to deployment, discover how to build scalable, test-driven, and reproducible ML systems using Julia’s powerful ecosystem. https://lnkd.in/esWGw8MT #JuliaLang #MLOps #CI_CD #MachineLearning #programming

17