docs(inkless:design): add diskless-managed rf feature design by jeqo · Pull Request #478 · aiven/inkless

jeqo · 2026-01-14T11:48:25Z

No description provided.

Copilot

Pull request overview

This PR adds a comprehensive design document for implementing managed replication factor (RF) for diskless topics in the Inkless system. The document proposes transitioning from the current RF=1 with faked metadata to RF=rack_count with real KRaft-managed replicas, enabling bidirectional topic migration and standard RLM integration.

Changes:

Adds new DISKLESS_MANAGED_RF.md design document with detailed approach comparison, cost analysis, and implementation plan
Updates DESIGN.md to reference the new design document for Stream 4: Multi-Replica Model for Diskless Topics

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md	New comprehensive design document proposing RF=rack_count approach with transformer-first availability, including motivation, design details, implementation path (6-8 weeks), and rejected alternatives
docs/inkless/ts-unification/DESIGN.md	Adds cross-references to the new DISKLESS_MANAGED_RF.md document in the Multi-Replica Model section and References section

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ivanyu · 2026-01-20T17:00:38Z

+
+| Trade-off | Accepted | Rationale |
+|-----------|----------|-----------|
+| KRaft metadata may show offline brokers | Yes | Availability is not blocked; eventual consistency is acceptable |


KRaft metadata may show offline brokers

Internally or to the clients?

To admin operations/metrics (e.g. topic describe), not to clients.
I'll clarify this

See 2a0ebee

Ensure cross-az is the last resort when routing requests, and that replica placement is fully decoupled.

jeqo · 2026-01-20T20:21:45Z

Moving it to draft to keep it as a live doc while this feature is developed.

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

Replace placeholder metrics with actual implemented metric names: - Controller metrics using KafkaController MBean naming - Transformer metrics using ClientAzAwarenessMetrics MBean - Add note explaining logging-based approach for placement quality to avoid high-cardinality metric concerns Co-Authored-By: Claude Opus 4.5 <[email protected]>

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) # Conflicts: # core/src/main/scala/kafka/server/ControllerServer.scala # core/src/main/scala/kafka/server/KafkaConfig.scala # metadata/src/main/java/org/apache/kafka/controller/QuorumController.java # server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) � Conflicts: � metadata/src/main/java/org/apache/kafka/controller/QuorumController.java � metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java � metadata/src/test/java/org/apache/kafka/controller/ReplicationControlManagerTest.java

Remove ~1010 lines of inline Java/SQL code and detailed tiering pipeline diagrams. Replace with bullet-point descriptions and summary tables. Keep all architecture diagrams, state machines, and read-path flow diagrams. Key changes: - Update goals: RF=rack_count decided, tiering pipeline in scope - Replace Java code blocks in sections 4.1-4.8 with bullets - Compact tiering pipeline section (4.7) to overview + tables - Replace metrics/admin code with summary tables - Remove write path and tiering pipeline data flow diagrams - Fix stale references (PROJECT_PLAN.md, config typo) - Remove broken architecture.md link Co-Authored-By: Claude Opus 4.6 <[email protected]>

…cs (#492) * test(metadata:diskless): improve test coverage for broker fencing and unregister scenarios - Add _noRacks and _withRacks test variants for consistent coverage - Fix tests that assumed broker 0 was always the leader - Get actual leader from partition registration before fencing/unregistering - Use dynamic assertions based on actual partition state - Improve assertion error messages for clarity * feat(controller:diskless): add server config for managed replicas Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) # Conflicts: # core/src/main/scala/kafka/server/ControllerServer.scala # core/src/main/scala/kafka/server/KafkaConfig.scala # metadata/src/main/java/org/apache/kafka/controller/QuorumController.java # server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java * feat(metadata:diskless): implement managed replicas for diskless topics When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) � Conflicts: � metadata/src/main/java/org/apache/kafka/controller/QuorumController.java � metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java � metadata/src/test/java/org/apache/kafka/controller/ReplicationControlManagerTest.java * fixup! feat(controller:diskless): add server config for managed replicas * fixup! feat(metadata:diskless): implement managed replicas for diskless topics

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) # Conflicts: # server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java

When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md)

…cs (#492) * test(metadata:diskless): improve test coverage for broker fencing and unregister scenarios - Add _noRacks and _withRacks test variants for consistent coverage - Fix tests that assumed broker 0 was always the leader - Get actual leader from partition registration before fencing/unregistering - Use dynamic assertions based on actual partition state - Improve assertion error messages for clarity * feat(controller:diskless): add server config for managed replicas Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) # Conflicts: # core/src/main/scala/kafka/server/ControllerServer.scala # core/src/main/scala/kafka/server/KafkaConfig.scala # metadata/src/main/java/org/apache/kafka/controller/QuorumController.java # server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java * feat(metadata:diskless): implement managed replicas for diskless topics When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) � Conflicts: � metadata/src/main/java/org/apache/kafka/controller/QuorumController.java � metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java � metadata/src/test/java/org/apache/kafka/controller/ReplicationControlManagerTest.java * fixup! feat(controller:diskless): add server config for managed replicas * fixup! feat(metadata:diskless): implement managed replicas for diskless topics (cherry picked from commit 09ba4d1)

…cs (#492) * test(metadata:diskless): improve test coverage for broker fencing and unregister scenarios - Add _noRacks and _withRacks test variants for consistent coverage - Fix tests that assumed broker 0 was always the leader - Get actual leader from partition registration before fencing/unregistering - Use dynamic assertions based on actual partition state - Improve assertion error messages for clarity * feat(controller:diskless): add server config for managed replicas Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) # Conflicts: # core/src/main/scala/kafka/server/ControllerServer.scala # core/src/main/scala/kafka/server/KafkaConfig.scala # metadata/src/main/java/org/apache/kafka/controller/QuorumController.java # server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java * feat(metadata:diskless): implement managed replicas for diskless topics When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) � Conflicts: � metadata/src/main/java/org/apache/kafka/controller/QuorumController.java � metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java � metadata/src/test/java/org/apache/kafka/controller/ReplicationControlManagerTest.java * fixup! feat(controller:diskless): add server config for managed replicas * fixup! feat(metadata:diskless): implement managed replicas for diskless topics

…cs (#492) * test(metadata:diskless): improve test coverage for broker fencing and unregister scenarios - Add _noRacks and _withRacks test variants for consistent coverage - Fix tests that assumed broker 0 was always the leader - Get actual leader from partition registration before fencing/unregistering - Use dynamic assertions based on actual partition state - Improve assertion error messages for clarity * feat(controller:diskless): add server config for managed replicas Add diskless.managed.rf.enable config (default: false) to control whether diskless topics use managed replicas with RF=rack_count or legacy RF=1. This config only affects topic creation. When enabled, new diskless topics will be created with one replica per rack using standard KRaft placement. Part of Phase 1: Diskless Managed Replicas (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) # Conflicts: # core/src/main/scala/kafka/server/ControllerServer.scala # core/src/main/scala/kafka/server/KafkaConfig.scala # metadata/src/main/java/org/apache/kafka/controller/QuorumController.java # server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java * feat(metadata:diskless): implement managed replicas for diskless topics When diskless.managed.rf.enable=true, new diskless topics are created with RF=rack_count using standard KRaft replica placement instead of legacy RF=1. Changes: - Compute RF from rack cardinality via rackCardinality() - Use standard replicaPlacer.place() for rack-aware assignment - Allow manual replica assignments when managed replicas enabled - Add checkstyle suppression for extended createTopic method Phase 1 limitations: - Add Partitions inherits RF from existing partitions (Phase 3) - Transformer not updated, uses legacy routing (Phase 2) - Integration tests deferred to Phase 2 (See #478 docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md) � Conflicts: � metadata/src/main/java/org/apache/kafka/controller/QuorumController.java � metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java � metadata/src/test/java/org/apache/kafka/controller/ReplicationControlManagerTest.java * fixup! feat(controller:diskless): add server config for managed replicas * fixup! feat(metadata:diskless): implement managed replicas for diskless topics (cherry picked from commit 09ba4d1)

jeqo · 2026-04-01T12:19:06Z

Closing as feature has been implemented and documented.

jeqo added 14 commits January 14, 2026 11:35

add diskless managed rf feature design

a3e5fa7

fixup! add diskless managed rf feature design

522c7f8

fixup! add diskless managed rf feature design

5e11830

fixup! add diskless managed rf feature design

0b0d562

fixup! add diskless managed rf feature design

e0a62ef

fixup! add diskless managed rf feature design

ba838f5

fixup! add diskless managed rf feature design

4593e7a

fixup! add diskless managed rf feature design

655a305

fixup! add diskless managed rf feature design

946acdc

fixup! add diskless managed rf feature design

6fc71de

fixup! add diskless managed rf feature design

9c35394

add cross reference

cd2d01c

fixup! add diskless managed rf feature design

1df130a

fixup! add diskless managed rf feature design

295e5d4

jeqo marked this pull request as ready for review January 14, 2026 12:03

jeqo requested a review from Copilot January 14, 2026 13:25

Copilot started reviewing on behalf of jeqo January 14, 2026 13:26 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Comment thread docs/inkless/ts-unification/DISKLESS_MANAGED_RF.md Outdated

jeqo added 4 commits January 14, 2026 15:57

fixup! add diskless managed rf feature design

1a6a585

updates from phase 0

a609b8d

results from phase 0

80ec6ad

add controller server-config to gate feature

8b2a950

ivanyu reviewed Jan 20, 2026

View reviewed changes

jeqo added 3 commits January 20, 2026 22:16

fixup! add controller server-config to gate feature

00628a5

clarify az-availability over replica placement consistency

2a0ebee

Ensure cross-az is the last resort when routing requests, and that replica placement is fully decoupled.

adjust dependencies on other work streams and phases

cedfb4d

jeqo marked this pull request as draft January 20, 2026 20:21

jeqo added 2 commits January 21, 2026 17:45

fixup! add controller server-config to gate feature

e636ef5

adjust scope for phase 1

4990f9b

jeqo mentioned this pull request Jan 22, 2026

feat(metadata:diskless): implement managed replicas for diskless topics #492

Merged

jeqo mentioned this pull request Feb 9, 2026

feat(metadata:diskless): add controller metrics for diskless topics #503

Merged

4 tasks

update controller metrics and user-defined RF

c7ad94d

jeqo closed this Apr 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(inkless:design): add diskless-managed rf feature design #478

docs(inkless:design): add diskless-managed rf feature design #478
jeqo wants to merge 27 commits intodesign/ts-unificationfrom
jeqo/design-diskless-managed-rf

jeqo commented Jan 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

ivanyu Jan 20, 2026

Uh oh!

jeqo Jan 20, 2026

Uh oh!

jeqo Jan 20, 2026

Uh oh!

jeqo commented Jan 20, 2026

Uh oh!

jeqo commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jeqo commented Jan 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

ivanyu Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

jeqo Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

jeqo Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

jeqo commented Jan 20, 2026

Uh oh!

jeqo commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants