Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions docker/examples/docker-compose-files/inkless/Makefile
Original file line number Diff line number Diff line change
@@ -1,30 +1,34 @@
DOCKER := docker

KAFKA_VERSION ?= 4.1.0-inkless-SNAPSHOT
# Default to edge (latest development build from main). Other options:
# KAFKA_VERSION=latest (latest stable release)
# KAFKA_VERSION=4.1.0-0.33 (specific Kafka+Inkless version)
# KAFKA_VERSION=local (locally built image, requires: make docker_build from repo root)
Comment thread
jeqo marked this conversation as resolved.
KAFKA_VERSION ?= edge

.PHONY: in-memory
in-memory:
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml up
KAFKA_VERSION=$(KAFKA_VERSION) $(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml up
$(MAKE) destroy

.PHONY: s3-local
s3-local:
$(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.s3-local.yml up
KAFKA_VERSION=$(KAFKA_VERSION) $(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.s3-local.yml up
$(MAKE) destroy

.PHONY: gcs-local
gcs-local:
$(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.gcs-local.yml up
KAFKA_VERSION=$(KAFKA_VERSION) $(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.gcs-local.yml up
$(MAKE) destroy

.PHONY: azure-local
azure-local:
$(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.azure-local.yml up
KAFKA_VERSION=$(KAFKA_VERSION) $(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.azure-local.yml up
$(MAKE) destroy

.PHONY: s3-aws
s3-aws:
$(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.s3-aws.yml up
KAFKA_VERSION=$(KAFKA_VERSION) $(DOCKER) compose -f docker-compose.yml -f docker-compose.monitoring.yml -f docker-compose.demo.yml -f docker-compose.s3-aws.yml up
$(MAKE) destroy

.PHONY: destroy
Expand Down
72 changes: 17 additions & 55 deletions docker/examples/docker-compose-files/inkless/README.md
Original file line number Diff line number Diff line change
@@ -1,65 +1,27 @@
# Inkless demo
# Inkless Docker Compose Demo

Contains examples of how to run Inkless with Postgres as batch coordinator and different object storage back-ends using Docker Compose.
Docker Compose examples for running Inkless with different storage backends.

Running the make task for a setup will start the necessary services and run the Inkless demo, including topic creation, producer/consumer perf clients, and monitoring with Prometheus and Grafana.
See the [Quickstart guide](../../../../docs/inkless/QUICKSTART.md#dockerized-demo) for complete setup instructions.

## Run the demo
## Quick reference

### in-memory
| Command | Description |
|-----------------------|---------------------------------------------------|
| `make s3-local` | MinIO S3-compatible storage (default) |
| `make gcs-local` | Fake GCS server |
| `make azure-local` | Azurite Azure-compatible storage |
| `make s3-aws` | Real AWS S3 (requires AWS credentials) |
| `make destroy` | Manual cleanup (automatic on Ctrl+C) |

This setup uses an in-memory object storage and Postgres as a metadata store.
## Image version

```
make in-memory
```

### S3 local

This setup uses MinIO as a local S3-compatible object storage and Postgres as a metadata store.

```
make s3-local
```

### GCS local

This setup uses `fake-gcs-server` as a local GCS-compatible object storage and Postgres as a metadata store.

```
make gcs-local
```

### Azure local

This setup uses `azurite` as a local Azure-compatible object storage and Postgres as a metadata store.

```
make azure-local
```

### AWS S3

This example demonstrates how to use Inkless with remote data and metadata stores,
using a Remote Postgres instance for metadata and AWS S3 for data storage.

#### Prerequisites

Set the environment variables:

For AWS S3:

```properties
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=
AWS_REGION=
AWS_S3_BUCKET_NAME=
```

Then run
By default, the demo pulls the `edge` image (latest development build). Override with `KAFKA_VERSION`:

```bash
make s3-aws
make s3-local KAFKA_VERSION=latest # Latest stable release
make s3-local KAFKA_VERSION=4.1.0-0.33 # Specific Kafka+Inkless version
make s3-local KAFKA_VERSION=local # Locally built image (requires: make docker_build)
```

See [Releases](../../../../docs/inkless/RELEASES.md#docker-images) for all available tags.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
services:
create_topic:
image: aivenoy/kafka:${KAFKA_VERSION}
image: ghcr.io/aiven/inkless:${KAFKA_VERSION}
restart: "no"
depends_on:
broker:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
services:
broker: &base-broker
image: aivenoy/kafka:${KAFKA_VERSION}
image: ghcr.io/aiven/inkless:${KAFKA_VERSION}
restart: unless-stopped
ports:
- "9092:9092"
Expand Down
9 changes: 5 additions & 4 deletions docker/inkless/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,12 @@ ARG kafka_version

COPY resources/distributions/kafka_2.13-$kafka_version.tgz /kafka.tgz

LABEL org.label-schema.name="kafka" \
org.label-schema.description="Apache Kafka" \
LABEL org.label-schema.name="inkless" \
org.label-schema.description="Inkless - Apache Kafka with object storage" \
org.label-schema.build-date="${build_date}" \
org.label-schema.vcs-url="https://github.com/apache/kafka" \
maintainer="Apache Kafka"
org.label-schema.vcs-url="https://github.com/aiven/inkless" \
org.opencontainers.image.source="https://github.com/aiven/inkless" \
maintainer="Aiven"

RUN set -eux ; \
apk update ; \
Expand Down
116 changes: 104 additions & 12 deletions docs/inkless/QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Quickstart

## Prerequisites

Before running Inkless, ensure you have:

- **Docker** and **Docker Compose v2** installed
- **Available ports:** 9092 (Kafka), 3000 (Grafana), 9001 (MinIO), 5432 (PostgreSQL)
- **Disk space:** ~2GB for demo images
- **For building from source:** JDK 17, Gradle (included via wrapper)

## Dockerized demo

Run:
Expand All @@ -10,23 +19,97 @@ make demo
make demo DOCKER=podman
```

It will pull the Docker image, and start the local demo with two producers, one consumer, the PostgreSQL-backed control plane, and MinIO as the object storage.
Grafana with metrics will be accessible at http://localhost:3000 (login `admin`, password `admin`).
Minio will be accessible at http://localhost:9001 (login `minioadmin`, password `minioadmin`).
It will pull the Docker image and start the local demo with two producers, one consumer, the PostgreSQL-backed control plane, and MinIO as the object storage. Press `Ctrl+C` to stop; containers and networks are cleaned up automatically.

### Demo services

| Service | URL | Credentials |
|------------------|-----------------------|-------------------------|
| Grafana | http://localhost:3000 | `admin` / `admin` |
| MinIO Console | http://localhost:9001 | `minioadmin` / `minioadmin` |
| Kafka Bootstrap | localhost:9092 | - |
| PostgreSQL | localhost:5432 | `admin` / `admin` |

If you prefer to build the image locally, you can run:
### Demo backends

The default backend is `s3-local` (MinIO). You can choose different storage backends:

```shell
make demo DEMO=s3-local # MinIO S3-compatible storage (default)
make demo DEMO=gcs-local # Fake GCS server
make demo DEMO=azure-local # Azurite Azure-compatible storage
make demo DEMO=s3-aws # Real AWS S3 (requires AWS credentials)
```

#### AWS S3 backend

For `s3-aws`, set the following environment variables before running:

```shell
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-key>
export AWS_SESSION_TOKEN=<your-session-token> # if using temporary credentials
export AWS_REGION=<your-region>
export AWS_S3_BUCKET_NAME=<your-bucket-name>

make demo DEMO=s3-aws
```

### Using a locally built image

By default, the demo pulls the `edge` image from GHCR. If the image is not available or you want to test local changes:

```shell
# make clean # to rebuild the binaries
make docker_build
make demo KAFKA_VERSION=local
```

and then run the demo.
### Monitoring the demo

The logs will print the producer and consumer stats.
To observe the metrics, go to the Kafka Inkless and Kafka Clients dashboards in Grafana.
The logs will print the producer and consumer stats. To observe the metrics, access Grafana at http://localhost:3000 and navigate to:

For more details, see the [demo docs](./../../docker/examples/docker-compose-files/inkless/README.md).
- **Kafka Inkless** - Inkless-specific metrics (batch sizes, object storage operations)
- **Kafka Clients** - Producer and consumer performance metrics

**View service logs (in a separate terminal):**
```shell
cd docker/examples/docker-compose-files/inkless
docker compose logs -f broker # Kafka broker logs (or: podman compose logs -f broker)
docker compose logs -f producer-1 # Producer performance stats
docker compose logs -f consumer # Consumer performance stats
```


## Using prebuilt binaries

If you prefer to run Inkless without Docker, download prebuilt binaries from [GitHub Releases](https://github.com/aiven/inkless/releases). See [Releases](./RELEASES.md#binary-distributions) for download instructions.

### Configure and run

Sample configurations are provided in `config/inkless/`:

| Config | Control Plane | Storage Backend |
|--------|---------------|-----------------|
| `single-broker-pg-0.properties` | PostgreSQL | S3 (MinIO) |
| `single-broker-pg-0-gcs.properties` | PostgreSQL | GCS |
| `single-broker-pg-0-azure.properties` | PostgreSQL | Azure Blob |

1. **Copy and edit a sample config** for your environment:
```shell
cp config/inkless/single-broker-pg-0.properties config/server.properties
# Edit config/server.properties with your storage and database settings
```

2. **Format storage and start broker:**
```shell
# Format storage directory (first time only)
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/server.properties

# Start the broker
bin/kafka-server-start.sh config/server.properties
```

See [configs.rst](./configs.rst) for the complete configuration reference and [Versioning Strategy](./VERSIONING-STRATEGY.md) for choosing the right version.


## Topic creation
Expand All @@ -45,9 +128,18 @@ make build_release
To run docker image:

```shell
docker run -it --entrypoint bash aivenoy/kafka:4.1.0-inkless-SNAPSHOT
4e3b0026634e:/$ cd /opt/
java/ kafka/
# Use locally built image (after running: make docker_build)
docker run -it --entrypoint bash ghcr.io/aiven/inkless:local

# Or use latest edge build from GHCR
docker run -it --entrypoint bash ghcr.io/aiven/inkless:edge

# Or use latest stable release
docker run -it --entrypoint bash ghcr.io/aiven/inkless:latest
```

Example session:
```shell
4e3b0026634e:/$ cd /opt/kafka/
4e3b0026634e:/opt/kafka$ bin/kafka-topics.sh
```
Expand Down
13 changes: 13 additions & 0 deletions docs/inkless/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,19 @@ Each release on version branches represents stable progress of the Inkless imple
> **Note:** Any S3-compatible storage backend should work in principle. MinIO has been used for integration testing, though we are evaluating alternatives given [MinIO's recent changes](https://github.com/minio/minio/commit/27742d469462e1561c776f88ca7a1f26816d69e2) to their Docker image and single-node deployment support.
- PostgreSQL (for batch coordinator) or in-memory mode for testing

## Quick Start

Choose your path based on what you want to do:

| Goal | Command / Action |
|------------------------------|---------------------------------------------------------------|
| **Try Inkless in 5 minutes** | `make demo` ([details](QUICKSTART.md#dockerized-demo)) |
| **Use prebuilt Docker image**| `docker pull ghcr.io/aiven/inkless:latest` |
| **Use prebuilt binaries** | [Binary usage guide](QUICKSTART.md#using-prebuilt-binaries) |
| **Build from source** | `make docker_build` ([details](QUICKSTART.md#using-a-locally-built-image))|

See [Releases](RELEASES.md) for available versions.

## Documentation Index

### Getting Started
Expand Down