inlets Pro

Managed HTTPS tunnels in one-click with inlets cloud

2025-04-01T00:00:00+00:00

Imagine if you could expose a local HTTP service, without TLS enabled to the public Internet with a HTTPS certificate with just one click.

This is now possible with inlets cloud, our hosted tunnel service which is live in Europe, US East and Asia, and free to use for all inlets subscribers whilst in beta.

We’ll start off by looking at the one-click, automatic option, then look at how we can use our own custom domain or even a custom Reverse proxy like Caddy, Nginx, or Traefik. I’ll also throw in some bonus material on how to expose SSH, the Kubernetes API, and an advanced option for self-hosting your own tunnel server.

For help and support, you can join our Discord server from the link in the inlets cloud dashboard, or use the contact page to get in touch.

Three options for your tunnels

We’ll focus on HTTP traffic for this post - think of a draft blog post, an API you’re working on, a webhook receiver, something in your homelab like Grafana, Wordpress, or perhaps an S3 endpoint like Minio that you can use to perform backups over the Internet to your NAS.

Let’s look at each of the three options rated from the 1-click experience (easiest) all the way down to running your own Nginx server, Caddy server, or Kubernetes Ingress controller (most flexible).

1. One-click HTTP to HTTPS - with our try-inlets.dev domain

You have a HTTP endpoint on your machine, with no TLS enabled. You can now expose it to the public Internet with a single click using HTTPS, under our domain try-inlets.dev

Create a tunnel giving it a descriptive name like “Wordpress”, “Next”, or “Grafana”, etc.

Click the “HTTP endpoint (we will terminate TLS for you)” option.

Then make sure the “Generate domain” is toggled on, this will generate a random and fun domain name for you like prickly-hedgehog.try-inlets.dev or happy-platypus.try-inlets.dev.

Create the tunnel, then scroll down to “Connect” and pick from CLI, systemd or Kubernetes YAML

Click the Copy icon and then paste the CLI command in on your local machine.

Change the --upstream flag to the HTTP endpoint on your local machine, or on a machine reachable on your local network.

For Grafana, that is likely going to be: http://127.0.0.1:3000, but if that were on your Raspberry Pi, it could be: http://192.168.0.12:3000

You’ll then be able to access your service at https://prickly-hedgehog.try-inlets.dev or whatever name you chose.

I recorded a quick video walk-through to show you just how quick and easy this approach can be:

2. HTTP to HTTPS with your own custom domain

First of all, create a new domain and verify it by creating a TXT record in your DNS provider. If you don’t have a domain yet, we’d recommend trying out Cloudflare or Namecheap, both of which are easy to set up and have a free tier.

The UI will show you how to verify your own domain, and confirm that it is working.

Next, create a tunnel again, but this time make sure the toggle for “Generate name” is off.

Enter each of the sub-domains you’d like to use, and then again scroll down to “Connect” and pick from CLI, systemd or Kubernetes YAML

I’ve added both: openfaas.selfactuated.dev and fileshare.selfactuated.dev as an example.

If those services were both running on my machine on port 8080 and 8000 respectively, then I’d change the --upstream flags as follows:

--upstream openfaas.selfactuated.dev=http://127.0.0.1:8080 \
--upstream fileshare.selfactuated.dev=http://127.0.0.1:8000

Once again, you can then run the client on your machine and expose the services to the public Internet.

Run the CLI command for the client, and then you’ll then be able to access your service at https://grafana.exmaple.com or whatever name you chose.

3. HTTPS termination - bring your own domain

This final option is the most versatile, but is also more involved than the first two.

Instead of having inlets-cloud terminate TLS and obtain certificates for you, you will run your own Reverse proxy or Kubernetes Ingress Controller on your machine or cluster.

You’ll need to create a domain and verify it before moving forward. If you already have one verified, you can use it again for the new sub-domains you want to expose.

Create a tunnel and enter the sub-domains you want to expose, but this time pick “Ingress (Reverse proxy, Kubernetes Ingress, Istio, SSH)” as the type of tunnel.

I’ve added both: openfaas.selfactuated.dev and fileshare.selfactuated.dev as an example.

Rather than having the --upstream flags point directly to the plaintext HTTP service, we have the --upstream pointing to our Reverse proxy or Ingress controller.

If you were exposing Caddy for instance, then you would then need to create a Caddyfile so it knows to answer the ACME challenges from Let’s Encrypt, and how to proxy the traffic to your local services.

openfaas.selfactuated.dev {
  reverse_proxy localhost:8080
}

fileshare.selfactuated.dev {
  reverse_proxy localhost:8000
}

For Kubernetes, the process is very similar, but you use a Kubernetes Ingress resource for each of the sub-domains you want to expose, and have the tunnel point to the Ingress controller.

Wrapping up

In this post we looked at three options for exposing HTTP services to the public Internet with a single click. We used inlets cloud, which is a managed service that’s free to all inlets subscribers during beta.

We started off with the one-click option, which is the easiest and requires the least configuration. That is instant, and gives you a HTTPS endpoint on our try-inlets.dev domain.
The second option was to use your own custom domain, but still have inlets cloud terminate TLS for you. Just verify a domain and you’re good to go.
The final option is the most flexible, and allows you to bring your own domain and run your own Reverse proxy or Ingress controller.

The tunnel client can be run directly on your machine with a CLI command, set up as a systemd service, or deployed to a Kubernetes cluster using a YAML file copied from the “Connect” section of the tunnel details.

You can register for access to inlets cloud. Just make sure you use the same email from your inlets subscription, and we’ll get you approved for access quickly.

If you have any questions don’t hesitate to reach out.

Inlets Cloud can also expose SSH and the Kubernetes API server

Inlets Cloud can also be used along with the inlets-pro snimux command to expose the SSH to as many local servers and Raspberry Pis as you like.

If you have a K3s cluster at home, or in your lab, you can tunnel out the Kubernetes API server so you can run kubectl from literally anywhere with an Internet connection.

Did you know? You can also self-host tunnel servers

Inlets Cloud is a very convenient way to set up tunnel servers instantly, with as little as one click, but for maximum flexibility and control, you can also self-host the tunnel server.

How to authenticate your HTTP tunnels with inlets and OAuth.

2025-03-10T00:00:00+00:00

In this tutorial you will learn how to secure your tunnelled HTTP services using the Inlets built-in HTTP authentication.

While inlets allows you to quickly expose any HTTP application to the public internet, you may not want everyone to be able to access it. Inlets can quickly add authentication to your application without any changes.

At the moment of writing Inlets support three forms of authentication:

OAuth
Basic authentication
Bearer token authentication

We will be showing you how to configure each of these authentication methods.

When would you want to secure your tunnel?

If you’re exposing an application for production, you may want to build authentication directly into your application.

However, during development and whilst collaborating with others, you may want to restrict access to a limited audience.

You’re working on a blog post draft, but only want your team mates to view it due to an embargo or because it contains confidential information.
You’re iterating on an API, and a reverse proxy usually provides authentication. It doesn’t make sense to run that locally, but you still need to restrict access.
When someone’s helping you with remote support. You want to expose your router’s admin interface, but need to restrict access to certain people.
You’re running internal development tools like Grafana dashboards or staging environments that shouldn’t be publicly accessible
You need to demo work to clients or stakeholders, but want to ensure only they can access it
You’re exposing temporary debugging endpoints or admin interfaces that need to be secured
You’re sharing access to local development environments like an IDE with remote team members but need to maintain security

In addition to all of the above, you can also restrict access by IP address using an IP allow list.

Prerequisites

We assume you have an Inlets HTTP tunnel server deployed. If you don’t have a tunnel yet follow our docs to create a new HTTP tunnel server.

For the tunnel client make sure you have the inlets-pro binary, version 0.10.0 or higher installed. Earlier versions do not support authentication.

Connect the tunnel client

In this example we will be exposing Prometheus, which is a popular open source tool for monitoring and alerting. We chose it because it has a web interface, and an API exposed over the same port. It does have some of its own built-in options for authentication, but when we use it with inlets, we can bypass authentication for our own local use, and only enforce it for remote users.

Expose the Prometheus upstream without any authentication enabled:

inlets-pro http client \
  --url "wss://157.180.37.179:8123" \
  --token-file "./token"  \
  --upstream prometheus.demo.welteki.dev=http://127.0.0.1:9090

Authentication for tunnels is configured through flags when connecting to the tunnel server. In the next paragraphs we will be going through the configuration for different authentication methods. The --url and --token-file flags will be left out of the commands for brevity but should be provided when connecting to your own tunnel.

OAuth with GitHub

If you want to avoid managing and distributing credentials for your application or need fine grained control over who can access the app you can use OAuth to protect tunneled applications.

In this tutorial we will be setting up OAuth with GitHub so that users can login with their GitHub account to access the tunnel.

Follow the GitHub documentation to create a new OAuth app for your tunnel.
Set the Authorization callback URL. In this example we are using the domain https://prometheus.demo.welteki.dev to expose the tunnel. The authorization callback for the tunnel will be https://prometheus.demo.welteki.dev/_/oauth/callback.

Example GitHub OAuth app configuration

Once you complete the registration of your OAuth app you will get a client id and secret. Save these in a convenient location. Both values need to be provided through the --oauth-client-id and --oauth-client-secret to start the tunnel client with OAuth enabled.

inlets-pro http client \
  --upstream prometheus.demo.welteki.dev=http://127.0.0.1:9090 \
+ --oauth-provider github \
+ --oauth-client-id $(cat ./oauth-client-id) \
+ --oauth-client-secret $(cat ./oauth-client-secret) \
+ --oauth-acl welteki

The oauth-acl flag is used to provide a list of users that are allowed to access the application. In case of the GitHub provider the ACL value can either be a GitHub username or email.

When trying to access the URl of the tunnel service, users will be asked to login with the configured provider, in this case GitHub, before they are able to access the application.

Login page for tunnels with GitHub OAuth enabled.

Basic authentication

The simplest form of authentication supported by Inlets is basic authentication. Enabling basic authentication on the tunnel will protect the HTTP service with a username and password.

When a user visits the URL of the tunneled service they will be prompted for a username and password before they are able to access the application.

Basic auth can be enabled for a tunnel by setting the basic auth flags when connecting the tunnel client.

inlets-pro http client \
  --upstream prometheus.demo.welteki.dev=http://127.0.0.1:9090 \
+ --basic-auth-username welteki \
+ --basic-auth-password $(cat ./basic-auth-password)

The --basic-auth-user flag is optional, when it is not provided the username will default to admin.

For example, the following allows access to full.name@example.com along with login1 and login2.

--oauth-acl full.name@example.com \
--oauth-acl login1 \
--oauth-acl login2

Basic auth login for a tunnel endpoint

Token authentication

The OAuth flow requires a web-browser and human interaction to authenticate. If you are tunneling a service like a HTTP API that needs to be accessed by a headless client, e.g. a script, mobile app or other non-web clients like a backend API, where it is not possible to complete the OAuth flow you can use Bearer Token Authentication.

In the case of our Prometheus server we have seen how the UI can be protected with basic auth or OAuth but the Prometheus server also exposes an HTTP API that needs to be protected but accessible by other services.

Generate a random token and store it in a file. We will use openssl to generate the token:

openssl rand -base64 16 > ./bearer-token

Start the Inlets client with the --bearer-token flag to enable token authentications.

inlets-pro http client \
  --upstream prometheus.demo.welteki.dev=http://127.0.0.1:9090 \
+ --bearer-token $(cat ./bearer-token)

Query the Prometheus API with curl and authenticate by adding the Authorization header on the request.

curl "https://prometheus.demo.welteki.dev/api/v1/labels" \
  -H "Authorization: Bearer $(cat ~/.inlets/prometheus-tunnel/token)"

The Bearer Token Authentication can be used together with both the basic auth and OAuth authentication. Just add the --bearer-token along with the flags you would need to configure AOauth or basic authentication. This makes it possible to quickly add authentication to an application like Prometheus where you have a browser based UI and HTTP API.

inlets-pro http client \
  --upstream prometheus.demo.welteki.dev=http://127.0.0.1:9090 \
  --oauth-provider github \
  --oauth-client-id $(cat ~/.inlets/prometheus-tunnel/oauth-client-id) \
  --oauth-client-secret $(cat ./oauth-client-secret) \
  --oauth-acl welteki \
+ --bearer-token $(cat ./bearer-token)

Conclusion

Inlets tunnels can be used to quickly add different types of authentication to your HTTP services without changing your applications. We showed how to configure and use the different authentication types supported by Inlets and discussed which one to pick for different use cases:

Use OAuth if you need to expose a UI or browser based application. AOauth gives you fine grained control over who can access the tunneled service through Access Control Lists without the need to share credentials.
Basic Authentication is the simplest form of authentication for your tunnels. It allows users to log in with a username and password and can be used as an alternative when using OAuth is not an option for you. Basic auth can also be used by headless clients without human interaction.
Bearer Token Authentication is recommended if you are exposing an HTTP API that needs to be accessible by headless clients. It can be used as the only authentication option or in combination with both OAuth and basic authentication.

Inlets has support for multiple OAuth providers like GitHub and Google. As a commercial user you get access to all providers. Please get in touch with the Inlets team if the OAuth provider you need is missing.

Expose ArgoCD on the Internet with Inlets and Istio

2025-02-04T00:00:00+00:00

In this tutorial, you will learn how to expose the ArgoCD dashboard on the Internet with Istio and the inlets-operator for Kubernetes.

ArgoCD is a popular tool for managing GitOps workflows and deploying applications to Kubernetes. It provides a web-based dashboard that allows you to view the state of your applications, compare them to the desired state, and sync them as needed. Another popular tool for GitOps workflows is FluxCD, which does not ship with a built-in UI, add-ons are available.

If you are running ArgoCD in a private VPC, in your homelab, or on-premises, then the inlets-operator can be used to quickly create a TCP tunnel to expose Istio’s Ingress Gateway to the Internet. This will allow you to access the ArgoCD dashboard from anywhere in the world.

ArgoCD login page exposed via Istio and Inlets

A different but related workflow we have seen with inlets tunnels, is where a number of remote Kubernetes clusters are tunneled back to a central Kubernetes cluster. From there, each can be added to ArgoCD and applications can be managed from a central location. This is a great way to manage multiple clusters and applications from a single dashboard. We covered that previously in How To Manage Inlets Tunnels Servers With Argo CD and GitOps.

Prerequisites

You will need a Kubernetes cluster running in a private network without ingress or Load Balancers. KinD, K3s, or Minikube can be a convenient way to test these steps.

We will install a number of Helm charts and CLIs during the tutorial. For convenience, arkade will be used to install these tools, but you are free to install them in whatever way you prefer.

You will also need a domain name under your control where you can create an A record to point to the public IP address of the inlets tunnel server.

Personal and commercial licenses are available from the inlets website at a similar price to a cloud load balancer service. There are no restrictions on the number of domains that can be exposed over a single tunnel, and the tunnel is hosted in your own cloud account.

Install the inlets-operator

The inlets-operator looks for LoadBalancer services and in response creates a VM in your cloud account with a public IP address. It then creates a Deployment for the inlets client within the cluster, and updates the LoadBalancer’s IP address with the public IP of the inlets server.

From that point, you have a fully working TCP tunnel to your Kubernetes cluster, just like you’d get with a LoadBalancer service from a cloud provider.

To install the inlets-operator with DigitalOcean, create an API token with read/write access and save it to ~/do-access-token:

# Create a tunnel in the lon1 region

export DO_REGION=lon1
arkade install inlets-operator \
  --provider digitalocean \
  --region $DO_REGION \
  --access-token-file ~/do-access-token

You can find instructions for Helm and other providers like AWS EC2, GCE, Azure, Scaleway, and so forth in the inlets-operator documentation.

Along with the documentation, you can find the inlets-operator Helm chart on GitHub.

Install ArgoCD

If you haven’t already installed ArgoCD, you can do so with the following command:

arkade install argocd

Now edit the argocd-server deployment and turn off its built-in self-signed certificate. We will be obtaining a certificate from Let’s Encrypt instead.

kubectl edit deployment argocd-server -n argocd

Add the --insecure flag to the args section:

      containers:
      - args:
        - /usr/local/bin/argocd-server
+       - --insecure

Install Istio

Install Istio with the following command:

arkade install istio

Create a DNS record for the ArgoCD dashboard

Verify the public IP address of the inlets tunnel server:

$ kubectl get svc -n istio-system istio-ingressgateway
NAME                   TYPE           CLUSTER-IP   EXTERNAL-IP       PORT(S)                                      AGE
istio-ingressgateway   LoadBalancer   10.43.5.77   144.126.234.124   15021:32412/TCP,80:31062/TCP,443:32063/TCP   51m

Next, create a DNS A record from argocd.example.com to the public IP address of the inlets tunnel server.

Install cert-manager

Install cert-manager with the following command:

arkade install cert-manager

Create a Let’s Encrypt Issuer and certificate

The Certificate must be created in the same namespace as the Istio Ingress Gateway, i.e. istio-system.

Create a file called letsencrypt-issuer.yaml with the following content:

export EMAIL="you@example.com"

cat > issuer-prod.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: istio-system
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: $EMAIL
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - selector: {}
      http01:
        ingress:
          class: istio
EOF

Now create a Certificate resource:

cat > certificate.yaml <<EOF
export DOMAIN="argocd.example.com"

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argocd-server-cert
  namespace: istio-system
spec:
  secretName: argocd-server-tls
  commonName: $DOMAIN
  dnsNames:
    - $DOMAIN
  issuerRef:
    name: letsencrypt-prod
    kind: Issuer
EOF

Apply the resources:

kubectl apply -f letsencrypt-issuer.yaml
kubectl apply -f certificate.yaml

Expose the ArgoCD dashboard

Create a file called argocd-gateway.yaml with the following content:

cat > gateway.yaml <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: argocd-gateway
  namespace: argocd
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "*"
      tls:
        credentialName: argocd-server-tls
        maxProtocolVersion: TLSV1_3
        minProtocolVersion: TLSV1_2
        mode: SIMPLE
        cipherSuites:
          - ECDHE-ECDSA-AES128-GCM-SHA256
          - ECDHE-RSA-AES128-GCM-SHA256
          - ECDHE-ECDSA-AES128-SHA
          - AES128-GCM-SHA256
          - AES128-SHA
          - ECDHE-ECDSA-AES256-GCM-SHA384
          - ECDHE-RSA-AES256-GCM-SHA384
          - ECDHE-ECDSA-AES256-SHA
          - AES256-GCM-SHA384
          - AES256-SHA

Create a file called argocd-virtualservice.yaml with the following content:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: argocd-virtualservice
  namespace: argocd
spec:
  hosts:
    - "*"
  gateways:
    - argocd-gateway
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: argocd-server
            port:
              number: 80

Apply the resources:

kubectl apply -f gateway.yaml
kubectl apply -f virtualservice.yaml

Access the ArgoCD dashboard

At this point you should be able to access the ArgoCD dashboard at https://argocd.example.com.

ArgoCD dashboard exposed via my own domain

You can use the command given via arkade info argocd to get the initial password for the admin user.

Wrapping up

Exposing an application behind inlets requires no additional effort or changes to the application or configuration itself. It is a drop-in replacement for a cloud LoadBalancer service, and can be used to expose any TCP service running in your Kubernetes cluster.

The majority of the steps we covered were due to the need to turn off the self-signed certificate within ArgoCD, and to obtain a certificate from Let’s Encrypt instead. This is a good practice for any application that is exposed to the Internet. The certificates are trusted by most PCs already, are free to obtain, and rotated regularly.

Both Istio and Ingress are common options for routing traffic and managing TLS termination. We covered Istio here to help customers who are already using Istio. We tend to prefer ingress-nginx ourselves for its simplicity and ease of use.

The ArgoCD documentation covers how to use ingress-nginx and other Ingress controllers: Docs: ArgoCD Ingress Configuration.

Arkade was used to install various Helm charts and CLIs purely for brevity, but you can use whatever tools you prefer to install them including Helm, brew or curl.

If you are interested in learning more about inlets, check out the inlets documentation or reach out to talk to us.

Quickstart - Automate & Scale Tunnels with Inlets Uplink

2024-12-09T00:00:00+00:00

Inlets Uplink is a complete solution for automating tunnels, that scales from anywhere from ten to tens of thousands of tunnels.

This guide get you started with deploying tunnels via the CLI, Kubernetes Custom Resource Definition (CRD), and REST API within about 30 minutes.

Uplink vs inlets

Now, you may be familiar with inlets as a stand-alone binary, and container image, and may have tried out the inlets-operator, that creates LoadBalancers for Services in your Kubernetes cluster. The stand-alone version is used to expose a few services from a private network to the Internet using HTTP or TCP tunnels, and it’s great for small teams and individuals.

Uplink was built for for DevOps teams in large companies, SaaS providers, IoT solutions, and hosting providers, who need to connect to many different remote endpoints using automation to make the process as seamless as possible. It provides packaging, APIs, and observability for automating the same inlets code that is used in stand-alone architectures.

Conceptual diagram showing management via the REST API (client-api), and a private Kubernetes API server being tunneled back to the management cluster for automation via ArgoCD.

About Uplink

It’s a scalable solution for automating tunnels
Installed via a Helm chart
Implements tenant isolation through Kubernetes namespaces
Includes a REST API, CLI and Custom Resource Definition (CRD) for managing tunnels
Includes detailed Prometheus metrics on active tunnels
Supports TCP and HTTP tunnels
Endpoints are private by default, but can be made public

Private or public HTTP & TCP endpoints

By default, each HTTP and TCP endpoint is kept private, and can only be accessed from within the Kubernetes cluster using a ClusterIP Service.

This approach is ideal for managing customer endpoints or internal services that are hosted in private or hard to reach environments.

For hosting providers, where you want some or all of the tunnels to be publicly accessible, you can turn on the “data router” component and use Kubernetes Ingress or Istio to route traffic from your custom domains to the tunnel server.

When exposing tunnels to the Internet, you can create a new Ingress record for each domain, or use a wildcard domain so that a single Ingress record and TLS certificate can serve all tunnels. Learn more in: Expose Tunnels to the Internet.

Our inlets cloud product is built on top of multiple inlets uplink installations in different regions around the world. Our UI makes use of the REST API (client-api) that’s built into inlets uplink.

Quick start

This guide is a quick start for installing inlets uplink as quickly as possible, and skips over some of the more advanced features like customizing the Helm chart, or enabling public endpoints, which are mentioned in the documentation.

We make the tutorial as fast as possible, we will use our arkade tool to install a few initial helm charts, but you are free to use helm directly if you prefer.

Bill of materials

A Kubernetes cluster with the ability to create public LoadBalancers
An Ingress controller or Istio
cert-manager to obtain TLS certificates
Helm 3
Arkade CLI
A domain under your control, where you can create a subdomain

Install the Ingress controller and cert-manager

arkade install ingress-nginx
arkade install cert-manager

Next, find the public address for your Ingress controller:

kubectl get svc ingress-nginx-controller -n ingress-nginx

This will be an IP address or a DNS name, some provides such as AWS EKS will provide a DNS name. Create DNS A records in the next step if you received an IP address, otherwise create CNAME records.

What needs to be public?

The only service that needs to be public is the client-router, which is used by the inlets-pro uplink client command via its --url wss:// flag.

The client-api can be kept private and accessed from within the cluster over HTTP, or it can be turned off completely. If you only intend to manage tunnels via the inlets-pro tunnel CLI, or the Kubernetes CRD (with Helm, ArgoCD, or kubectl), then the client-api can be disabled.

Tunneled services will only be accessible via ClusterIP from within the Kubernetes cluster, so they are private by default. If needed, you can Expose them on the Internet by following separate instructions.

Configure the uplink Helm chart

Create two DNS A or CNAME records to the IP or DNS name given in the previous step:

The first is for the client-api, this is the REST API that can be used to manage tunnels - us1.uplink.example.com
The second is for the client-router, this is the public endpoint that the inlets client will use - clientapi.us1.uplink.example.com

Next, edit values.yaml:

export LE_EMAIL="webmaster@example.com"
export CLIENT_ROUTER_DOMAIN="us1.uplink.example.com"
export CLIENT_API_DOMAIN="clientapi.us1.uplink.example.com"

cat <<EOF > values.yaml
ingress:
  class: "nginx"
  issuer:
    enabled: true
    name: "letsencrypt-prod"
    email: "$LE_EMAIL"

clientRouter:
  # Customer tunnels will connect with a URI of:
  # wss://uplink.example.com/namespace/tunnel
  domain: $CLIENT_ROUTER_DOMAIN
  tls:
    ingress:
      enabled: true

clientApi:
  enabled: true
  domain: $CLIENT_API_DOMAIN
  tls:
    ingress:
      enabled: true

The REST API provided by the clientApi section is secured with an API token that we will generate in a moment, however OIDC/Auth2 can also be used, and is best for when you have several different uplink regions.

The OIDC/OAuth2 authentication is set through the following, but is not recommended for the quick start, since it requires additional infrastructure such as Keycloak or Okta.

clientApi:
  # When using OAuth/OIDC tokens to authenticate the API instead of
  # a shared secret, set the issuer URL here.
  issuerURL: "https://keycloak.inlets.dev/realms/inlets-cloud"

  # The audience is generally the same as the value of the domain field, however
  # some issuers like keycloak make the audience the client_id of the application/client.
  audience: "cloud.inlets.dev"

Before installing the Helm chart, we need to make sure some secrets exist in the cluster.

Create the inlets namespace, do not customise this for the quick start, since you’ll have to edit every command:

kubectl create namespace inlets
kubectl label namespace inlets \
    inlets.dev/uplink=1

Create a secret with the inlets-uplink license key:

kubectl create secret generic \
  -n inlets inlets-uplink-license \
  --from-file license=$HOME/.inlets/LICENSE_UPLINK

Create the API token for the client-api:

export token=$(openssl rand -base64 32|tr -d '\n')
mkdir -p $HOME/.inlets
echo -n $token > $HOME/.inlets/client-api

kubectl create secret generic \
  client-api-token \
  -n inlets \
  --from-file client-api-token=$HOME/.inlets/client-api

Now install the chart:

helm upgrade --install inlets-uplink \
  oci://ghcr.io/openfaasltd/inlets-uplink-provider \
  --namespace inlets \
  --values ./values.yaml

Verify the installation:

# Check the deployments were created and are running with 1/1 replicas:
kubectl get deploy -n inlets

# Check the logs of the various components for any errors:
kubectl logs deploy/client-api -n inlets
kubectl logs deploy/inlets-router -n inlets
kubectl logs deploy/cloud-operator -n inlets

# Make sure that the certificates were issued by cert-manager:
kubectl get certificate -n inlets -o wide

Connect a remote HTTP endpoint

You can define a HTTP or a TCP tunnel, for this example we will use HTTP.

For a simple test, run the built-in HTTP fileserver from the inlets binary on your local machine, and share a new temporary folder:

mkdir -p /tmp/share
echo "Hello from $(whoami)" > /tmp/share/index.html

cd /tmp/share

inlets-pro fileserver \
    --webroot /tmp/share \
    --port 8080 \
    --allow-browsing

It can be created via a Kubernetes CRD, via the REST API, or via the CLI.

You’ll find examples for each in the documentation.

inlets-pro get tunnel

inlets-pro tunnel list

inlets-pro tunnel create \
    fileserver \
    --upstream 127.0.0.1:8080

Then get the connection string, you can format this as a CLI command or as Kubernetes YAML to expose a Pod, Service, etc within a private cluster.

inlets-pro tunnel connect \
    fileserver \
    --namespace inlets \
    --domain https://$CLIENT_ROUTER_DOMAIN

The default output is for a CLI command you can run on your machine:

--format cli - default
--format k8s_yaml - for Kubernetes YAML to apply to a private cluster
--format systemd - generate a systemd unit file to install on a Linux machine

Both Kubernetes and systemd will restart the tunnel if it fails, and retain logs that you can view later.

Start up the tunnel client with the command you were given.

Remember, by default Inlets Uplink uses private endpoints, so you will need to run a Pod within the cluster to access the tunnel.

kubectl run --rm -it --restart=Never \
  --image=alpine:latest \
  --command -- sh

Then install curl and access the tunnel:

# apk add curl
# curl -i http://fileserver.inlets:8000/index.html

All HTTP tunnels bind to port 8000, and can multiplex multiple services over the same port using a Host header.

Create a TCP tunnel

Perhaps you need to access a customer’s Postgres database from their private network?

In this example we’ll define the tunnel using a Custom Resource instead of the CLI.

Example Custom Resource to deploy a tunnel for a Postgres database:

apiVersion: uplink.inlets.dev/v1alpha1
kind: Tunnel
metadata:
  name: db1
  namespace: inlets
spec:
  licenseRef:
    name: inlets-uplink-license
    namespace: inlets
  tcpPorts:
  - 5432

Alternatively the cli can be used to create a new tunnel:

inlets-pro tunnel create db1 \
  --namespace inlets \
  --port 5432

The quickest way to spin up a Postgres instance on your own machine would be to use Docker:

export PASSWORD=$(head -c 16 /dev/urandom |shasum)
echo $PASSWORD > ./postgres-password.txt

docker run --rm --name postgres \
  -p 5432:5432 \
  -e POSTGRES_PASSWORD=$PASSWORD \
  -ti postgres:latest

Connect with an inlets uplink client

inlets-pro tunnel connect db1 \
    --namespace inlets \
    --domain https://$CLIENT_ROUTER_DOMAIN \
    --upstream 127.0.0.1:5432

Run the above command on your local machine to generate the tunnel client command.

Then run it on your local machine to connect to the tunnel.

Access the customer database from within Kubernetes

Now that the tunnel is established, you can connect to the customer’s Postgres database from within Kubernetes using its ClusterIP db1.inlets.svc.cluster.local:

Try it out:

kubectl run -i -t psql \
  --env PGPORT=5432 \
  --env PGPASSWORD=$(cat ./postres-password.txt) --rm \
  --image postgres:latest -- psql -U postgres -h db1.inlets

Try a command such as CREATE database websites (url TEXT), \dt or \l.

Create a TCP tunnel using the REST API

You can view the reference documentation for the REST API for Inlets Uplink here: REST API

This example will tunnel a private Kubernetes cluster to your management cluster for administration or automation through tools such as kubectl, ArgoCD, Helm, Flux, or your own Kubernetes operators.

If you don’t use Kubernetes, you can still try out the commands, then delete the tunnel without connecting to it.

If you do want to try the example and don’t have a private cluster handy, you can create one using Docker via the kind create cluster --name kube1 command. The kind tool is available via arkade get kind.

As a general rule, the upstream should be kubernetes.default.svc and the port should be 443, for K3s clusters, the port is often changed to 6443.

Retrieve the API token for the client-api from Kubernetes:

export TOKEN=$(kubectl get secret -n inlets client-api-token \
  -o jsonpath="{.data.client-api-token}" \
  | base64 --decode)

Create a new tunnel using the REST API:

export CLIENT_API=https://clientapi.us1.uplink.example.com

export NAME="kube1"

curl -s -H "Authorization: Bearer ${TOKEN}" \
  -X POST \
  $CLIENT_API/v1/tunnels \
  -d '{"name": "kube1", "namespace": "inlets", "tcpPorts": [443] }' \
  | jq

{"name":"kube1","namespace":"inlets","created":"2024-12-09T11:01:38Z"}

You can verify the result via the API or via kubectl:

kubectl get tunnels.uplink.inlets.dev/kube1 -n inlets -o wide

NAME    AUTHTOKENNAME   DEPLOYMENTNAME   TCP PORTS   DOMAINS   INGRESS
kube1   kube1           kube1            [443]   

List the tunnels we created earlier, along with the new one with:

curl -s -H "Authorization: Bearer ${TOKEN}" \
  $CLIENT_API/v1/tunnels \
  | jq

[
  {
    "name": "kube1",
    "namespace": "inlets",
    "tcpPorts": [
      443
    ],
    "connectedClients": 0,
    "created": "2024-12-09T11:01:38Z"
  }
]

You can build a connection command using the inlets-pro tunnel connect command:

export CLIENT_ROUTER=us1.uplink.example.com

inlets-pro tunnel connect kube1 \
    --namespace inlets \
    --domain https://$CLIENT_ROUTER_DOMAIN \
    --format k8s_yaml \
    --upstream kubernetes.default.svc:443 > kube1-client.yaml

Switch your Kubernetes cluster to the private cluster, then apply the YAML file for the inlets client with kubectl apply -f kube1-client.yaml.

Check that the client connected:

kubectl logs deploy/kube1-inlets-client

time="2024/12/09 11:27:03" level=info msg="Connecting to proxy" url="wss://us1.uplink.example.com/inlets/kube1"
time="2024/12/09 11:27:03" level=info msg="Connection established" client_id=51dbd4430bac4049b56f107481d25394

Now switch back to the management cluster’s context.

The cluster will be available via a ClusterIP in the inlets namespace name kube1.inlets on port 443.

Access the Kubernetes API server

Either set up the additional TLS name for the Kubernetes API server’s SAN such as kube1.inlets (K3s makes this easy via --tls-san), or update the KUBECONFIG to provide the server name as per these instructions, or use --insecure-skip-tls-verify.

Find the following section and edit it:

- cluster:
    server: https://kube1.inlets:443
    tls-server-name: kubernetes

For a quick test, run a Pod in the cluster and try to access the Kubernetes API server using --insecure-skip-tls-verify:

kubectl run -i -t kube1-connect \
  --namespace inlets \
  --image=alpine:latest --rm \
  --restart=Never -- sh

# apk add kubectl curl

# curl -i -k https://kube1.inlets:443

Put your kubeconfig into place at .kube/config, update the server name and endpoint to https://kube1.inlets:443

You can use cat > .kube/config to create the file, then paste in the contents from your machine. Hit Control+D when done. This is quicker than installing an editor such as nano or vim into the container.

# cd
# mkdir -p .kube
# cat > .kube/config

# kubectl --context kind-kube1 get node
NAME                  STATUS   ROLES           AGE   VERSION
kube1-control-plane   Ready    control-plane   19m   v1.31.0
# 

Next steps

In a short period of time, we installed inlets uplink to a Kubernetes cluster, and created public endpoints for the REST API (client-api) and the client-router. We then created three tunnels using the CLI, the CRD and the REST API. We used a single namespace for all the tunnels, but you can create a namespace per tenant, and then input the namespace into each of these approaches.

Once you have services such as Postgresql, SSH, Ollama, the Kubernetes API server, or your own TCP/HTTP services tunneled back to the management cluster, you can start accessing the endpoints as if they were directly available within the Kubernetes cluster.

This means that all CLIs, tools, and products that work with whatever you’ve tunneled can be used without modification.

Common uses-cases for inlets-uplink

Do you have an agent for your SaaS product, that customers need to run on private networks? Access it via a tunnel.
Perhaps you manage a number of remote databases? Use pgdump and pgrestore to backup and restore databases.
Do you deploy to Kubernetes? Use kubectl, Helm, ArgoCD, or Flux to deploy applications, just run them in-cluster.
Do you write your own Kubernetes operators for customers? Just provide the updated KUBECONFIG to your Kubernetes operators and controllers.
Do you want to access GPUs hosted on Lambda Labs, Paperspace, or your own datacenter? Command and control your GPU instances from your management cluster.
Do you have a powerful GPU somewhere and want to infer against it using your central cluster? Run ollama remotely, and tunnel its REST API back.
Do you have many different edge devices? Tunnel SSHD and run Ansible, Puppet, or bash scripts against them just as if they were on your local network.

In the documentation you can learn more about managing, monitoring and automating tunnels.

If you’re new to Kubernetes, and would like us to give you a hand setting everything up, we’d be happy to help you with the installation, as part of your subscription benefits.

Would you like a demo, or to speak to our team? Reach out here for a meeting.

SSH Into Any Private Host With Inlets Cloud

2024-10-17T00:00:00+00:00

When you’re away from home it’s not only convenient, but often necessary to connect back to your machines. This could be to connect to a remote VSCode instance, run a backup, check on a process, or to debug a problem. SSH can also be used to port-forward services, or to copy files with scp or rsync.

In this post I’ll show you how to use inlets cloud to get SSH access to any host on your network without needing a VPN, and without having to host a tunnel server. You’ll be able to expose one or more machines over a single tunnel, using DNS names to route traffic.

What is inlets cloud?

inlets cloud is a SaaS hosted version of inlets-pro servers, and it makes for a convenient way to share local webservices, APIs, and endpoints like a blog, database server, Ollama endpoint, or a Kubernetes cluster. The client for inlets cloud uses TCP passthrough, so when your service uses SSH or TLS, there’s no way for our infrastructure to decrypt the traffic.

You can already use a self-hosted tunnel server to expose SSH for a single host, or for many with sshmux:

But in this tutorial, we’re going to do it with inlets-cloud, without having to host, maintain or even consider any infrastructure. There are two regions for the beta - the EU and USA.

Traditionally, SSH had to be exposed on a public IP address using port-forwarding rules, or a random TCP port using a SaaS tunnel, or perhaps using a complex SaaS VPN solution.

SSH is a simple technology that is designed to be exposed to the public Internet, the main issue is that it’s hard to multiplex multiple hosts over a single port.

We built sshmux into inlets to solve this problem, and now you can use it with inlets-cloud.

The diagram below shows sshmux with a self-hosted tunnel server, but in this tutorial, we’ll be using inlets-cloud to provide the hosting instead.

sshmux is a simple way to expose multiple SSH backends over a single port, using a DNS name to route traffic. Just create a wildcard domain entry, or add one entry per host, and then configure sshmux to direct traffic accordingly.

You will of course need to follow the general security advise on hardening SSH, which is easy to find on the Internet, or via a brief chat with ChatGPT.

Prerequisites

You’ll need a domain name, and the ability to create CNAME or A records, you’ll create one entry per host that you want to access.
OpenSSH set up on one or more machines in your private network, follow the usual security precautions like disabling Root access and Password authentication
An inlets cloud account, you can sign up for free at cloud.inlets.dev

Install the inlets-pro client

Since the inlets-pro tunnel server will be hosted for us, we only need to download the inlets-pro client.

If you haven’t already, install the inlets-pro client:

curl -sLSf https://get.arkade.dev | sudo sh

arkade get inlets-pro

Alternatively, download the binary from the inlets-pro releases page.

Create a tunnel on inlets-cloud

Create a new tunnel, the name is not important, but the list of sub-domains is, this is where you add one entry for each host you want to access.

You can add more after creating the tunnel, so feel free to start with one, if that’s easier.

Create any CNAME entries in your DNS provider as directed, and verify the top-level domain with a TXT record.

Then navigate back to the tunnel details page, and copy the text under “Caddy/Nginx”:

We want to adjust it slightly, so that we can use it with sshmux instead of Nginx or Caddy. We’ll be directing in SSH traffic, not TLS traffic for webservers.

inlets-pro uplink client \
  --url="wss://cambs1.uplink.inlets.dev/alexellis/sshmux" \
  --token=******** \
-  --upstream=80=127.0.0.1:80 \
+  --upstream=443=127.0.0.1:8443

We remap any incoming requests to the hosted server on port 443, to port 8443, which is the default port for sshmux.

Run the command to start the tunnel, when connected, you’ll see the text: “Connection established”.

Start sshmmux

sshmux is an SSH multiplexer that can expose multiple SSH backends over a single port, using a DNS name to route traffic.

Typically, other solutions will require you to use a different port for each host, and then you have to memorise random numbers, or resort to clever SaaS-based VPNs.

Create a config.yaml file:

upstreams:
  - name: nuc.example.com
    upstream: 192.168.0.200:22
  - name: nas.example.com
    upstream: 10.0.0.2:2222

Then start sshmux:

inlets-pro sshmux server \
    --port 8443 \
    config.yaml

Now switch to a machine you’ll use to connect to the SSH services. This can be the same host for the sake of testing, but would probably be your laptop.

SSH does not support using a SNI header, or a TLS hostname, so we use sshmux to wrap the traffic with this header.

You can use the openssl tool for this, or our convenience command inlets-pro sshmux connect.

Edit .ssh/config:

Host *.example.com
    HostName %h
    Port 443
    ProxyCommand inlets-pro sshmux connect cambs1.uplink.inlets.dev:%p %h

The text cambs1.uplink.inlets.dev is the DNS entry you used for the CNAMES in the previous step, so if you’re using a different region, use that value here instead.

All this does is to tell SSH to use the inlets-pro sshmux connect command to connect to the remote host, and to pass the hostname and port number to the command.

Connect to one of your hosts

Now you can connect to one of your hosts:

ssh nuc.example.com

You can also use the -L flag to forward ports or services running on the remote host to your local machine.

For instance, if you were running a Node.js application on port 3000 on the remote host, you could forward it to your local machine like this:

ssh -L 3000:127.0.0.1:3000 nuc.example.com

If you have a Kubernetes cluster on the remote machine, you can port-forward services from it to your local machine, whilst on a different network. For instance, if the remote cluster is running OpenFaaS CE, and you wanted to access its Prometheus dashboard, you could do this:

ssh -L 9090:127.0.0.1:9090 nuc.example.com
# kubectl port-forward -n openfaas svc/prometheus 9090:9090

Then open a browser to http://localhost:9090 to access the Prometheus dashboard.

You can specify multiple hosts and ports, i.e. for both Prometheus and the OpenFaaS gateway:

ssh -L 9090:127.0.0.1:9090 \
    -L 8080:127.0.0.1:8080 \
    nuc.example.com
# kubectl port-forward -n openfaas svc/prometheus 9090:9090 &
# kubectl port-forward -n openfaas svc/gateway 8080:8080 &

Of course you can also copy files with scp or use rsync over the SSH connection.

Copy a single remote file to your host:

scp nuc.example.com:~/debug.log .

Rsync the code for k3sup, that you’re hacking on to the remote computer:

rsync -av -r -e "ssh" ~/go/src/github.com/alexellis/k3sup .

The port override for 443 is not necessary since the .ssh/config file will handle this, but you can explicitly add the flag if you want. ssh uses -p and scp uses -P.

Conclusion

You’ve now got a secure way to access any host on your network, without needing to host a tunnel server, or to set up a VPN. This is a great way to access your home network, or to provide support to friends and family.

Adding extra hosts

Any time you want to add or remove a host, you can do so via the inlets-cloud dashboard, by navigating to the “Tunnels” page and editing the list of domains. Then make sure you have an entry in your sshmux config file, and restart it with the new configuration.

Access is completely private, there is no way to decrypt the SSH traffic, and it gets passed directly on to your own machine inside your local network.

IP filtering/allow list

For taking things further, sshmux also supports an IP allow list, which is available for inlets-cloud and self-hosted tunnels.

If the IP for your mobile hotspot was 35.202.222.154, you could write the following to restrict access to nuc.example.com to only yourself:

upstreams:
  - name: nuc.example.com
    upstream: 192.168.0.120:22
    allowed:
    - 35.202.222.154

Then just add a --proxy-protocol argument to the inlets-pro sshmux command before restarting the command. You can use v1 or v2 as the argument, just make sure it is the same as the one you selected for the tunnel server.

Watch a video walk-through of this tutorial:

Get Real Client IPs with Ingress Nginx, Caddy or Traefik

2024-10-08T00:00:00+00:00

When you’re running a reverse proxy directly on a host, or an Ingress Controller in Kubernetes, you can get the real client IP with inlets.

The real client IP address is required for rate-limiting, effective logging, understanding where your users are coming from geographically, and to prevent abuse. Just bear in mind that if you choose to store these addresses within a database, or server logs, you may need to comply with data protection laws like GDPR.

We’ve covered how Proxy Protocol works before in the original post Get Real Client IPs with K3s and Traefik, so that’s a good refresher if you’d like to cover the fundamentals again.

In this post we’ll focus on the configuration you need, so you can come back here and copy/paste it when you need it.

The Inlets Setup

Deploy a host and install the inlets-pro tcp server, you can do this manually or via cloud-init.

The inletsctl tool can create a host using a cloud provider’s API, and pre-install the inlets-pro server for you, with a randomly generated authentication token, and will print out all the details at the end.

Whichever method you take, log into the host, and edit the systemd unit file for inlets-pro, find it via sudo systemctl cat inlets-pro.

Add --proxy-protocol=v2 to the ExecStart line, if it’s already present with an empty value, update it instead.

The v2 protocol is widely supported and more efficient than v1, since it sends text in a binary format, not in a human-readable format.

This article assumes that you are running the inlets-pro tcp server process directly on an Internet-facing host. If you are running it behind a cloud load-balancer, you’ll need to add the --lb-proxy-protocol flag to the inlets-pro server specifying the protocol version sent by the load-balancer. The rest of the article applies in the same way.

Real IPs for Caddy

Caddy can be installed quickly, including its systemd unit file, special caddy user, and extra directories with the arkade system install caddy command. You can also use a custom build, or run through all the manual steps yourself from the Caddy documentation.

I’ve included this section for when you want to run a reverse proxy in a VM, container, or directly on your machine. The other examples are focused on running a reverse proxy in Kubernetes, called an Ingress Controller. For instance, you may be running OpenFaaS via faasd CE. In that case, Caddy is a quick way to get TLS termination for your OpenFaaS functions, and anything else you are running in your setup like Grafana.

The following settings are for when you run Caddy directly on your own machine, and use an inlets TCP tunnel server to expose it to the Internet, pointing ports 80 and 443 to your Caddy instance.

{
    email "webmaster@example.com"

    acme_ca https://acme-v02.api.letsencrypt.org/directory
    http_port 80
    https_port 443

   servers {
     listener_wrappers {
       proxy_protocol {
         timeout 2s
         allow 0.0.0.0/0
       }
      tls
    }
 }
}

orders.example.com {
    reverse_proxy 127.0.0.1:8080
}

There are a number of extra settings over a basic Caddyfile for Let’s Encrypt, but the main one we need is the proxy_protocol listener wrapper.

You’ll see I’ve also included an upstream for orders.example.com which is a plain HTTP service running on port 8080. It will receive the real client IP from Caddy, and can read it from the X-Real-IP header.

Real IPs for ingress-nginx

I sent to install ingress-nginx via arkade, with arkade install ingress-nginx. This is similar to applying the static YAML that is available in the project’s documentation.

The ingress-nginx documentation site explains the various settings that can be configured for an installation of ingress-nginx. One of those options is for Proxy Protocol. You don’t need to set a version, just set it to true and either version will be accepted.

Edit the ConfigMap for ingress-nginx, when installed via arkade, it will be called ingress-nginx-controller, so:

kubectl edit configmap ingress-nginx-controller

Within the data: section, add:

data:
+  use-proxy-protocol: "true"

There are some additional related headers, which you can customise:

data:
+  compute-full-forwarded-for: "true"
+  enable-real-ip: "true"
+  proxy-protocol-header-timeout: 1s
+  set-real-ip-from: 0.0.0.0/0
+  use-forwarded-headers: "true"

Once updated, the controller will reload its settings and will only accept requests which have a Proxy Protocol header. If you send a request without the header, it will be rejected, so it must only be accessed via the inlets tunnel.

Real IPs for Traefik

This section was taken from the original blog post. You can refer there for more details.

Traefik ships with K3s by default, and is installed into the kube-system namespace.

When I create k3s clusters with k3sup, I tend to turn off Traefik in order to add ingress-nginx which I find to be simpler, broadly used in production setups, and easier to operate. I just run: k3sup install --no-extras to make sure Traefik won’t be installed.

If you want to use Traefik, you can do so by editing the deployment:

kubectl -n kube-system edit deployment traefik

Then add the following flags:

    spec:                             
      containers:                              
      - args:                                                         
+        - --entryPoints.web.proxyProtocol.insecure=true
+        - --entryPoints.web.proxyProtocol.trustedIPs=0.0.0.0/24
+        - --entryPoints.websecure.proxyProtocol.insecure=true
+        - --entrypoints.websecure.http.tls
+        - --entrypoints.web.address=:8000/tcp                
+        - --entrypoints.websecure.address=:8443/tcp

I also add - --accesslog=true to help find any potential issues with the configuration.

If Traefik doesn’t detect the settings immediately, you can restart it with kubectl rollout restart -n kube-system deployment traefik.

If you wish to swap Traefik for ingress-nginx, you can run:

kubectl delete -n kube-system deployment traefik
kubectl delete -n kube-system service traefik

Wrapping up

I wanted this article to be a short and sweet reference for you, on how to configure the most popular reverse proxies to accept the Proxy Protocol header, so that your applications can get the real client IP.

If you’re running an alternative Kubernetes Ingress Controller, Istio Gateway, or a stand-alone proxy, all you need to do after configuring the inlets-pro tcp server is to enable the Proxy Protocol support using the appropriate settings.

If you have any questions or suggestions, please feel free to reach out. Whenever you sign up for a subscription for inlets, you’ll get an invite to our Discord community. If you signed up some time, ago reach out via the form on the website and we’ll get you an invite.

Create Highly Available Tunnels With A Load Balancer

2024-09-02T00:00:00+00:00

We look at Highly Available inlets tunnels, how to integrate with Proxy Protocol to get original source IP addresses, and how to configure a cloud load balancer.

For the majority of use-cases, whether for development or production, a single tunnel server VM and client as a pair will be more than sufficient.

However, some teams mandate that all infrastructure is run in a Highly Available (HA) configuration, where two or more servers or instances of a service are running at all times. This is to ensure that if one server fails, the other can take over and continue to serve traffic.

If you’re using a public cloud offering like AWS, GCP, Hetzner, DigitalOcean, or Linode, you’ll have access to managed load balancers. These are easy to deploy, come with their own stable public IP address, and can be used to route traffic to one or more virtual machines. How does failover work? Typically, when you create the load balancer, you’ll specify a health check, which the load balancer will run on a continual basis, if it detects that one of the servers is unhealthy, it will stop sending traffic to it.

What is a cloud load balancer anyway? If you dig around, you’ll find documentation, blog posts and conference talks where cloud vendors explain how they use either HAProxy, keepalived, or Envoy to provide their managed load-balancers. For anyone who does not have access to a managed load balancer, you can configure and deploy these open source tools yourself, just make sure that your load balancer itself does not become its own Single Point of Failure (SPOF).

In this post, we’ll look at how to create a Highly Available inlets tunnel with a cloud load balancer, and Proxy Protocol support, to get the original source IP addresses.

A note from an inlets user on HA

Jack Harmon is a long time inlets user, and sent in a photo of his homelab, which is running Kubernetes (setup with kubeadm) and Traefik as an Ingress Controller. He has a HA tunnel configuration using a Global LoadBalancer on DigitalOcean.

Here’s what Jack had to say about his setup:

I used to work for a company that set up secure cloud infrastructure for governments, which is where I got interested in home labs and HA config, as well as zero trust security. I’ve since left and do financial consulting for businesses. I now have a home lab running in my closet, and another in a city apartment for redundancy.

I use my setup to host my various personal software projects, file sharing, to offer hosting to friends and family, and yes - as a secure way to store client information (or show them financial dashboards) on occasion. Mostly, though, it’s an extremely overbuilt hobby project that I’ve sunk thousands of hours into over the years. I realize there may be a slightly cheaper option with its own limitations, but I prefer the privacy and control, and to support independent developers like yourself.

Jack also wanted to get SSH access into various VMs in the lab, so I told him about our sshmux add-on for inlets where you can expose dozens of sshd servers over a single inlets tunnel. This is a great way to get access to your VMs without needing to expose them directly to the internet. Of course, if you do go down this route, make sure you disable Password login, so that you’re only using SSH keys. SSH keys are going to be almost impossible to attack with brute-force.

Proxy Protocol for real client IPs

When a TCP connection is made from one server to another, the source IP address is the IP of the server that initiated the connection. This is a problem if, like many of the servers on the Internet, they are directly exposed to their users. Whether you’re working with a Pod in Kubernetes, a VM in an autoscaling group behind a Load Balancer, a service hidden behind Nginx, or a service exposed via inlets, if you want the real IP address of the client, you will need to make use of the Proxy Protocol.

The Proxy Protocol (popularised by HAProxy) is a simple protocol that is sent at the beginning of a TCP connection, and contains the original source IP address and port of the client. This is then passed through the proxy, and can be used by the service to determine the real IP address of the client. There are two versions, v1 which is sent in plain text, and v2 which is sent in binary.

Until the October release of inlets, Proxy Protocol was supported when the inlets tunnel server was run directly on an internet-facing server via the --proxy-protocol flag. That meant the receiving end of the tunnel, the “upstream” would get a Proxy Protocol header and would need to be configured to understand it.

Now, with the new release, Proxy Protocol is now supported when the inlets tunnel server is behind a load balancer by setting the --lb-proxy-protocol flag, in addition to the existing flag.

The conceptual design

Here’s a diagram of the design we’re going to implement:

An inlets tunnel server has two parts:

The control-plane, usually served on port 8123. Clients connect here to establish a tunnel.
The data-plane, these ports can vary, but in our example are 80 and 443, to expose a reverse proxy like Nginx.

The control-plane must not be set behind a load balancer, because if that were the case, both clients could connect to the same server, negating the HA design.

The data-plane will sit behind the load-balancer, and its health checks will ensure that if either of the tunnels goes down, or either of the VMs crashes, the load balancer will stop sending traffic to it.

So, to replicate the architecture in the diagram, you’ll need to:

Deploy two VMs with the inlets-pro tcp server installed.
Set up a cloud load balancer to route traffic to the two VMs, on ports 80 and 443.

If the private service that is being exposed over the tunnel supports Proxy Protocol, then this can be used to obtain real client IP addresses. Most proxies and reverse proxies do support the protocol, but if you don’t want to configure this, or don’t need real client IPs to be sent to the private service, you can ignore all references to Proxy Protocol in this post.

Now, ensure you enable Proxy Protocol support on the load balancer itself. Some clouds allow you to specify which version of Proxy Protocol you want to use, if possible, pick v2. DigitalOcean only supports v1.

Whichever you choose, you will need to configure your inlets-pro tcp server process to use the same via the new --lb-proxy-protocol flag. Valid options are: "" (off), "v1", or "v2".

Then you’ll deploy two inlets-pro tcp clients in the private network, pointing at your upstream, i.e. an nginx reverse proxy. Each client must point to its matching server, and not to the load-balancer.

Fail-over not load-balancing

You can actually connect more than one inlets-pro tcp client to a single inlets-pro tcp server, for the sake of load-balancing, and increasing the number of connections that can be handled.

However, load-balancing is not fail-over. If the VM hosting the inlets-pro tcp server fails or crashes, then you won’t be able to serve any traffic.

In the diagram above, we can see that the VM with the private IP 10.0.0.3 failed, and is not reachable by the load balancer. It will mark this endpoint as unhealthy, and stop sending traffic to it.

The other VM with IP 10.0.0.2 is still healthy, and will continue to serve traffic.

Wrapping up

In this post, we’ve looked at how to create a Highly Available inlets tunnel with a cloud Load Balancer, and Proxy Protocol support, to get the original source IP addresses.

If you want to keep your configuration simple, whilst still having a HA setup, you can forgo the use of Proxy Protocol, however I tend to recommend it for debugging and security purposes. Knowing the source IP of your users, or the IP of the client that is connecting to your service can give you insights on where your traffic is coming from, and can be used to block or allow certain IP ranges.

If you’re happy with your current setup with inlets, then there’s nothing you need to change. But one thing you may like to try out, instead of a HA setup, is running a second inlets-pro tunnel client, connected to the same server. Each connection is load-balanced, meaning you can handle additional traffic and more connections.

If you’re exposing a Kubernetes Ingress Controller, here are the instructions for setting it up to expect a Proxy Protocol header.

Both approaches involve editing either a ConfigMap, or the flags passed to the binary in the Kubernetes Deployment.

Expose your local Kubernetes Ingress Controller via Hetzner Cloud

2024-08-15T00:00:00+00:00

There are two ways to configure inlets to expose an Ingress Controller or Istio Gateway to the public Internet, both are very similar; only the lifecycle of the tunnel differs.

For teams that are new to inlets, and set up most of their configuration with clicking buttons, installing Helm charts and applying YAML from their workstation or a CI pipeline, then the inlets-operator keeps things simple. Whenever you install the inlets-operator, it searches for LoadBalancer resources and provisions VMs for them with the inlets-pro TCP server preinstalled. It then creates a Deployment in the same namespace with an inlets TCP client pointing to the remote VM, and everything just works.

The downside to the inlets-operator is that if you delete the exposed resource, i.e. ingress-nginx, then the tunnel will be deleted too, and when recreated it will have a different IP address. That means you will need to update your DNS records accordingly.

What if you’re heavily invested in GitOps, and regularly delete and re-create your cluster’s configuration? Then you may want a more stable IP address and set of DNS records, in that case, you can create the VM for the inlets tunnel server manually or semi-automatically with Terraform, Pulumi or our own provisioning CLI called inletsctl.

With the inlets-operator, you need to pick a region and supported provider such as AWS EC2, DigitalOcean, or Hetzner Cloud and provide those options via the Helm chart. For a manual tunnel server, you can use any tooling or cloud/VPS provider you wish. We’ll be using Hetzner Cloud in this example, which is particularly good value and fast to provision.

A quick video demo of the operator

In this animation by Ivan Velichko, you see the operator in action. As it detects a new Service of type LoadBalancer, provisions a VM in the cloud, then updates the Service with the IP address of the VM.

Getting an LB in about 30 seconds with Hetzner Cloud

About 30s from creating a Service with type LoadBalancer, to a fully working endpoint. And I'm sat in a cafe running KinD with WiFi. pic.twitter.com/zUV9US7OM0
— Alex Ellis (@alexellisuk) June 12, 2024

I work from home, and once per week, if work allows, I try to get out to a coffee shop and to work on a blog post there, in different surroundings. On this occasion I decided to update the inlets-operator’s support for Hetzner Cloud, and to give it a quick test. You can see from the screenshot that a KinD cluster running on my MacBook Air M2 was able to get a public IP address in about 30 seconds flat.

So what’s needed? First you’ll need an account with Hetzner Cloud, bear in mind Hetzner Robot is the dedicated server offering and needs a different login and account.

Log into Hetzner Cloud and enter your Default project.
Click “Security” then the “API tokens” tab
Click “Generate API token” and name it “inlets-operator” and grant Read/Write access.
Click “Click to show” then copy the text and save it as ~/.hetzner-cloud-token
Now determine the available regions by clicking Add Server, don’t actually add a server but use the screen to copy the code of the region you want i.e. Helsinki (eu-central) or Ashburn, VA (us-east).

If you don’t have an inlets license yet, obtain one at inlets.dev and save it as ~/.inlets/LICENSE.

Formulate the Helm install command

# Create a namespace for inlets-operator
kubectl create namespace inlets

# Create a secret to store the service account key file
kubectl create secret generic inlets-access-key \
  --namespace inlets \
  --from-file inlets-access-key=$HOME/.hetzner-cloud-token

# Create a secret to store the inlets-pro license
kubectl create secret generic \
  --namespace inlets \
  inlets-license --from-file license=$HOME/.inlets/LICENSE

# Add and update the inlets-operator helm repo
# You only need to do this once.
helm repo add inlets https://inlets.github.io/inlets-operator/

export REGION=eu-central
export PROVIDER=hetzner

# Update the Helm repository and perform an installation
helm repo update && \
  helm upgrade inlets-operator --install inlets/inlets-operator \
  --namespace inlets \
  --set provider=$PROVIDER \
  --set region=$REGION

If you don’t have an Ingress Controller already installed and configured in your cluster, then you can add one with arkade:

curl -sLS https://get.arkade.dev | sudo sh

arkade install ingress-nginx

That will create a service of type LoadBalancer in the default namespace, watch it and you’ll see an IP appear for it:

kubectl get service --watch

How quickly did your public IP appear?

There’s also a Custom Resource Definition (CRD) for the inlets-operator, you can view it with:

kubectl get tunnels -o wide

Access your ingress-nginx service with the IP address shown in the EXTERNAL-IP column, and you’ll see the default Nginx welcome page.

curl -i http://<EXTERNAL-IP>

If you delete the tunnel CR, you’ll see it re-created with a new IP in a short period of time:

kubectl delete tunnel ingress-nginx

Then watch either the service or tunnel object again:

kubectl get tunnels -o wide --watch

Wrapping up

Using a single tunnel and a single license, you can expose dozens, if not hundreds of different websites through your Ingress Controller, all running within your private or on-premises Kubernetes cluster. The inlets-operator is a great way to get started with inlets, and it’s also a great way to expose your Ingress Controller to the public Internet.

The inlets-operator works with different clouds and can expose any TCP LoadBalancer, not just Ingress Controllers and Istio.

Bear in mind that the tunnel IP and DNS records will be tied to the lifecycle of your LoadBalancer services, so if you delete them, the VMs will be deleted too, and if you re-create them, then they’ll be re-created with a new IP address. For that reason, you may want to create the tunnel servers manually, or separately from the inlets-operator.

The inlets-pro tcp server --generate=systemd and inlets-pro tcp client --generate=k8s_yaml are two utility commands to make it easier to set up both parts of the tunnel without needing the operator.

The operator will also need credentials to provision and clean up VMs, that’s another thing to consider when deciding which approach to use.

The code for the inlets-operator is open source under the MIT license and available on GitHub: inlets/inlets-operator.

Watch a video walk-through

I recorded a video walk-through of the blog post, so you can watch it back and see the steps in action.

When you sign up for a subscription, you’ll get complimentary access to a Discord community to talk with other users and the inlets team.

Access local Ollama models from a cloud Kubernetes Cluster

2024-08-09T00:00:00+00:00

Renting a GPU in the cloud, especially with a bare-metal host can be expensive, and even if the hourly rate looks reasonable, over the course of a year, it can really add up. Many of us have a server or workstation at home with a GPU that can be used for serving models with an open source project like Ollama.

We can combine a cloud-hosted, low cost, elastic Kubernetes cluster with a local single-node K3s cluster with a GPU, and in that way get the best of both worlds.

One option may be to try and join the node on your home network as a worker in the cloud-hosted cluster, but I don’t think that makes sense:

Why you shouldn’t use a VPN to join local hosts to your cloud Kubernetes cluster

Kubernetes is designed for homogenous networking, where latency to each host is low and predictable, it is not built to work over WANs
Every time the API is accessed, or the scheduler is used, it is going to have to reach out to your home network over the Internet which will introduce unreasonable latency
A large amount of bandwidth is required between nodes, this has to go over the Internet which counts as egress traffic, and is billable
The node in your home gets completely exposed to the cloud-based cluster, there is no security boundary or granularity
You’ll need complex scheduling YAML including taints, tolerations and affinity rules to ensure that the node in your home and the ones in the cloud get the right workloads

So whilst it might look cool to run “kubectl get nodes” and see one that’s in your home, and a bunch that are on the cloud, there are more reasons against it than for it.

Nothing to see here. Only 65GB of bandwidth consumption from an idle k3s cluster running embedded etcd.

Anyone else noticed similar? pic.twitter.com/Q4BVIIHJ7n
— Alex Ellis (@alexellisuk) October 21, 2022

So how is inlets different?

Inlets is well known for exposing local HTTP and TCP services on the Internet, but it can also be used for private tunnels.

There will be minimal bandwidth used, as the model is accessed over the tunnel
The risk to your home network is minimal, as only the specific port and endpoint will be accessible remotely
It’s trivial to access the tunneled service as a HTTP endpoint with a normal ClusterIP

So rather than having to enroll machines in your local or home network to be fully part of a cloud-hosted cluster, you only tunnel what you need. It saves on bandwidth, tightens up security, and is much easier to manage.

And what if ollama isn’t suitable for your use-case? You can use the same technique by creating your own REST API with Flask, FastAPI, Express.js, Go, etc, and expose that over the tunnel instead.

There are common patterns for accessing remote APIs in different networks such as using queues or a REST API, however building heterogeneous clusters with high latency is not one of those.

If you need more dynamic workloads and don’t want to build your own REST API to manage Kubernetes workloads, then consider the OpenFaaS project provides a HTTP API and built-in asynchronous queue system that can be used to run batch jobs and long-running tasks, it can also package Ollama as a function, and is easy to use over a HTTP tunnel. For example: How to transcribe audio with OpenAI Whisper and OpenFaaS or Stream OpenAI responses from functions using Server Sent Events.

A quick look at the setup

The setup with two independent Kubernetes clusters, one running locally with a GPU and ollama, the other in the cloud running your product and making HTTP requests over the tunnel.

You’ll need to do the following:

Setup a local machine where you’ve installed K3s, and setup nvidia-containerd.
Create a Kubernetes cluster using a managed Kubernetes service like DigitalOcean, AWS, or Google Cloud, or you can setup a self-hosted cluster using a tool like K3sup on a set of VMs.
Package and deploy Ollama along with a model as a container image, then deploy it to your local K3s cluster
Create a HTTPS tunnel server using inlets on the public Kubernetes cluster
Create a HTTPS tunnel client using inlets on the local K3s cluster
Finally, we will launch curl in a Pod in the public cluster and invoke the model served by Ollama over the tunnel

Then it’ll be over to you to integrate the model into your applications, or to develop your own UI or API to expose to your users.

1. Setup a local machine with K3s and nvidia-containerd

Over on the OpenFaaS blog under the heading “Prepare a k3s with NVIDIA container runtime support”, you’ll find full instructions for setting up a single-node K3s cluster on a machine with an Nvidia GPU.

If you do not have an Nvidia GPU available, or perhaps just want to try out the tunnelling example without a K3s cluster, you can create a local cluster inside Docker using Kubernetes in Docker (KinD) or Minikube.

For example:

kind create cluster --name ollama-local

To connect to the two clusters use the: kubectl config use-context command, or the popular helper kubectx NAME available via arkade with arkade get kubectx.

2. Create a cloud-hosted Kubernetes cluster

This step is self-explanatory, for ease of use, you can setup a cloud-hosted Kubernetes cluster using a managed Kubernetes service like DigitalOcean, AWS, or Google Cloud. With a managed Kubernetes offering, load-balancers, storage, networking, and updates are managed by someone else, so it’s a good way to get started.

If you already have a self-hosted cluster on a set of VMs, or want to manage Kubernetes yourself, then K3sup provides a quick and easy way to create a highly-available cluster.

3. Package and deploy Ollama

Ollama is a wrapper that can be used to serve a REST API for interference on various machine learning models. You can package and deploy Ollama along with a model as a container image, then deploy it to your local K3s cluster.

Here’s an example Dockerfile for packaging Ollama:

FROM ollama/ollama:latest

RUN apt update && apt install -yq curl

RUN mkdir -p /app/models
RUN ollama serve & sleep 2 && curl -i http://127.0.0.1:11434 ollama pull phi3

EXPOSE 11434

Two other options for making the model available to Ollama are:

If you do not wish to package the model into a container image and push it into a remote registry, you can use an init container, or a start-up script as the entrypoint and perform that operation at runtime.
Another option is to download the model locally, then to copy it into a volume within the local Kubernetes cluster, then you can mount that volume into the Ollama container.

Build the image and publish it:

export OWNER="docker.io/alexellis2"
docker build -t $OWNER/ollama-phi3:0.1.0 .

docker push $OWNER/ollama-phi3:0.1.0

Now write a Kubernetes Deployment manifest and accompanying Service for Ollama:

export OWNER="docker.io/alexellis2"

cat <<EOF > ollama-phi3.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama-phi3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama-phi3
  template:
    metadata:
      labels:
        app: ollama-phi3
    spec:
      containers:
      - name: ollama-phi3
        image: $OWNER/ollama-phi3:0.1.0
        ports:
        - containerPort: 11434
        command: ["ollama", "serve"]
---
apiVersion: v1
kind: Service
metadata:
  name: ollama-phi3
spec:
  selector:
    app: ollama-phi3
  ports:
  - protocol: TCP
    port: 11434
    targetPort: 11434
EOF

The above is designed to work with a CPU, and can be extended to support a GPU by adding the necessary device and runtime class.

Deploy the manifest to your local K3s cluster:

kubectl apply -f ollama-phi3.yaml

You can check to see when the Pod has been downloaded and started:

kubectl get deploy/ollama-phi3 -o wide --watch
kubectl logs deploy/ollama-phi3

Check that you can invoke the model from within your local cluster.

Run an Alpine Pod, install curl and jq, then try accessing the Ollama API to see if it’s up:

kubectl run -it --rm --restart=Never --image=alpine:latest ollama-phi3-test -- /bin/sh

# apk add --no-cache curl jq

# curl -i http://ollama-phi3:11434/

Ollama is running

Next, try an inference:

# curl http://ollama-phi3:11434/api/generate -d '{
    "model": "phi3",
    "stream": true,
    "prompt":"What is the advantage of tunnelling a single TCP host over exposing your whole local network to an Internet-connected Kubernetes cluster?"
    }' | jq

The above configuration may be running on CPU, in which case you will need to wait a few seconds whilst the model runs the query. It took 29 seconds to get a response on my machine, when you have the GPU enabled, the response time will be much faster.

If you’d like to get a streaming response, and see data as it comes in, you can set stream to true in the request, and remove | jq from the command.

4. Create a HTTPS tunnel server using inlets

Now you can generate an access token for inlets, and then deploy the inlets server to your cloud-hosted Kubernetes cluster.

The inlets control-plane needs to be available on the Internet, you can do this via a LoadBalancer or through Kubernetes Ingress.

I’ll show you how to use a LoadBalancer, because it’s a bit more concise:

On the public cluster, provision a LoadBalancer service along with a ClusterIP for the data-plane.

The control-plane will be used by the tunnel client in the local cluster, and the data-plane will only be available within the remote cluster.

cat <<EOF > ollama-tunnel-server-svc.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: ollama-tunnel-server-control
spec:
  type: LoadBalancer
  ports:
  - port: 8123
    targetPort: 8123
  selector:
    app: ollama-tunnel-server
---
apiVersion: v1
kind: Service
metadata:
  name: ollama-tunnel-server-data
spec:
  type: ClusterIP
  ports:
  - port: 8000
    targetPort: 8000
  selector:
    app: ollama-tunnel-server
---
EOF

Apply the manifest on the remote cluster: kubectl apply -f ollama-tunnel-server-svc.yaml

Now, obtain the public IP address of the LoadBalancer by monitoring the EXTERNAL-IP field with the kubectl get svc -w -o wide ollama-tunnel-server-control command.

$ kubectl get svc -w
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
kubernetes             ClusterIP      10.245.0.1      <none>           443/TCP          12m
ollama-tunnel-server   LoadBalancer   10.245.26.4     <pending>        8123:32458/TCP   8s
ollama-tunnel-server   LoadBalancer   10.245.26.4     157.245.29.186   8123:32458/TCP   2m30s

Now input the value into the following command run on your workstation:

export EXTERNAL_IP="157.245.29.186"
export TOKEN=$(openssl rand -base64 32)

echo $TOKEN > inlets-token.txt

inlets-pro http server \
    --generate k8s_yaml \
    --generate-name ollama-tunnel-server \
    --generate-version 0.9.33 \
    --auto-tls \
    --auto-tls-san $EXTERNAL_IP \
    --token $TOKEN > ollama-tunnel-server-deploy.yaml

Apply the generated manifest to the remote cluster:

kubectl apply -f ollama-tunnel-server-deploy.yaml

You can check that the tunnel server has started up properly with:

kubectl logs deploy/ollama-tunnel-server

At this stage, you should have the following on your public cluster:

$ kubectl get service,deploy
NAME                                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
service/kubernetes                     ClusterIP      10.245.0.1      <none>           443/TCP          110m
service/ollama-tunnel-server-control   LoadBalancer   10.245.7.92     157.245.29.186   8123:31581/TCP   76m
service/ollama-tunnel-server-data      ClusterIP      10.245.40.138   <none>           8000/TCP         76m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ollama-tunnel-server   1/1     1            1           88m

5. Create a HTTPS tunnel client using inlets

Now you can create a tunnel client on your local K3s cluster:

export EXTERNAL_IP=""

inlets-pro http client \
    --generate k8s_yaml \
    --generate-name ollama-tunnel-client \
    --url "wss://$EXTERNAL_IP:8123" \
    --token-file inlets-token.txt \
    --upstream=http://ollama-phi3:11434 \
    > ollama-tunnel-client-deploy.yaml

The upstream uses the ClusterIP of the Ollama service within the local cluster, where the port is 11434.

The above will generate three Kubernetes objects:

A Deployment for the inlets client
A Secret for the control-plane token
A Secret for your license key for inlets

Apply the generated manifest to the local cluster:

kubectl apply -f ollama-tunnel-client-deploy.yaml

Check that the tunnel was able to connect:

kubectl logs deploy/ollama-tunnel-client

inlets-pro HTTP client. Version: 0.9.32
Copyright OpenFaaS Ltd 2024.
2024/08/09 11:49:02 Licensed to: alex <alex@openfaas.com>, expires: 47 day(s)
Upstream:  => http://ollama-phi3:11434
time="2024/08/09 11:49:02" level=info msg="Connecting to proxy" url="wss://157.245.29.186:8123/connect"
time="2024/08/09 11:49:03" level=info msg="Connection established" client_id=91612bca9c0f41c4a313424db9b6a0c7

There will be no way to access the tunneled Ollama service from the Internet, this is by design, only the control-plane is made available to the client on port 8123.

Your local Kubernetes cluster should have the following:

$ kubectl get svc,deploy
NAME                  TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
service/kubernetes    ClusterIP   10.96.0.1     <none>        443/TCP     110m
service/ollama-phi3   ClusterIP   10.96.128.9   <none>        11434/TCP   106m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ollama-phi3            1/1     1            1           106m
deployment.apps/ollama-tunnel-client   1/1     1            1           83m

6. Invoke the model from the public cluster

Switch back to the Kubernetes cluster on the public cloud.

Now you can run a Pod in the public cluster and invoke the model served by Ollama over the tunnel:

kubectl run -it --rm --restart=Never --image=alpine:latest ollama-phi3-test -- /bin/sh

# apk add --no-cache curl

# curl -i http://ollama-tunnel-server-data:8000/api/generate -d '{
              "model": "phi3",
              "stream": true,
              "prompt":"How can you combine two networks?"
            }'

Example response:

HTTP/1.1 200 OK
Content-Type: application/x-ndjson
Date: Fri, 09 Aug 2024 11:56:19 GMT
Transfer-Encoding: chunked

{"model":"phi3","created_at":"2024-08-09T11:56:19.486043693Z","response":"Com","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.543130824Z","response":"bin","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.598975314Z","response":"ing","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.654731638Z","response":" two","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.710487681Z","response":" or","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.766891184Z","response":" more","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.822626098Z","response":" neural","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.880506452Z","response":" networks","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.937516988Z","response":" is","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:19.993925621Z","response":" a","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:20.049548473Z","response":" technique","done":false}
{"model":"phi3","created_at":"2024-08-09T11:56:20.106052896Z","response":" commonly","done":false}

Note in this case, the service name for the tunnel is used, along with the default HTTP port for an inlets tunnel which is 8000, instead of the port 11434 that Ollama is listening in the local cluster.

Anything you deploy within your public Kubernetes cluster can access the model served by Ollama, by making HTTP requests to the data plane of the tunnel server with the address: http://ollama-tunnel-server-data:8000.

Conclusion

In this post, you learned how to use a private tunnel to make a local GPU-enabled HTTP service like Ollama available in a remote Kubernetes cluster. This can be useful for serving models from a local GPU, or for exposing a service that is not yet ready for the public Internet.

You can now integrate the model into your applications, or develop your own UI or API to expose to your users using the ClusterIP of the data-plane service.

We exposed the control-plane for the tunnel server over a cloud Load Balancer, however if you have multiple tunnels, you can use a Kubernetes Ingress Controller instead, and direct traffic to the correct tunnel based on the hostname and an Ingress record. If you take this route, just remove the --auto-tls-san flags from the inlets-pro command as they will no longer be needed. You can use cert-manager to terminate TLS instead.

If you’d like to see a live demo, watch the video below:

If you enjoyed this post, you can find similar examples in the inlets docs, or on the inlets blog.

Get Kubernetes Ingress like Magic

2024-06-18T00:00:00+00:00

Learn how to expose Ingress from your Kubernetes cluster like magic without having to setup any additional infrastructure.

This post will start with a bit of background information on inlets, why we think now is the right time to offer hosted tunnels, how it works, and then we’ll give you a walk-through so you can see if it’s for you.

Introduction to inlets

Why did we need inlets in 2019?

I started Inlets in early 2019 as an antidote to the frustrating restrictions of the SaaS-style tunnels of the day like Ngrok and Cloudflare tunnels, and manual port-forwarding that exposed your home address to users.

So rather than having very poor integration with containers, inlets was born at the height of Cloud Native - with a Kubernetes operator, Helm chart, container image, and multi-arch binary for MacOS, Windows and Linux.
Rather than having stringent and impractical rate-limits, inlets was designed to be self-hosted meaning you were free of limits.
Rather than exposing where you lived via your ISP’s IP address, you got to mask it with a public IP address from a cloud provider.

What were the trade-offs?

So what was the trade-off? You would need to set up a VM somewhere and start the tunnel server on it. To make that as easy as possible, two open-source utilities were created, with support for various cloud platforms:

inletsctl creates a cloud VM with a public IP address, and the inlets server pre-installed, the command line for the inlets client is printed out after
inlets-operator runs inside a Kubernetes cluster, and creates a cloud VM for Kubernetes clusters whenever it detects a service of type LoadBalancer

Why are you making a SaaS now?

Making a SaaS for inlets always seemed counter-intuitive, why not use one of the established products? But increasingly, we saw users drawn to the user experience, quality of integration and versatility of inlets. Our team even started to find creative ways to make inlets feel like a SaaS - by using a single VM for multiple different websites, or by setting up inlets tunnel servers on a Kubernetes cluster.

So we’ve built an extension for inlets that makes it into a SaaS, and have already started using it ourselves. Now we’re looking for early users to try it out and provide feedback.

Just gave this a try to get ingress for my OpenFaaS development gateway running in a private k3d cluster. pic.twitter.com/05WMFwiPvP
— Han Verstraete (@welteki) June 18, 2024

How hosted tunnels (the SaaS) works

The inlets client is run inside your Kubernetes cluster as a Deployment just like self-hosted inlets, the magic comes in with how the tunnel server is managed for you.

Conceptual architecture for the SaaS

For the initial version of the SaaS, we’re offering what we’re calling an Ingress Tunnel.

An Ingress Tunnel:

Exposes an Ingress Controller or Istio Gateway to the public internet
Uses ports 80 and 443
Supports ACME HTTP01 and DNS01 challenges for Let’s Encrypt
Supports WebSockets, HTTP/2 and gRPC

Each tenant’s Ingress Tunnel gets its own dedicated deployment of an inlets tunnel server, which at idle takes up about 3MB of RAM. The tunnel server is a single binary written in Go, and is designed to be very efficient.

You’ll need the following:

A domain under your control
The ability to create a DNS CNAME entry to our SaaS cluster’s public IP address
A Kubernetes cluster with an Ingress Controller or Istio Gateway

You can use any Ingress Controller, such as ingress-nginx, Traefik, Kong, or an Istio Gateway.

Once you tell us the domain names you want to expose via your Ingress Tunnel, we’ll provide you with YAML for the tunnel client, which runs inside your cluster.

From there, you can go ahead and use Ingress as if your cluster was being provided by AWS or a similar managed service.

A quick walk-through

Here’s a quick walk-through so you can try out the service. We’ll use OpenFaaS Community Edition (CE), along with ingress-nginx and Let’s Encrypt. arkade.dev will be used to keep the commands simple, but you can use Helm or kubectl if you like to make work for yourself.

We’d suggest going end to end with these instructions before switching over to one of your own services like a Grafana dashboard, or blog, etc.

Create a Kubernetes cluster, some options:
- Start one inside Docker with kind or k3d on your machine
- Setup a VM in your homelab and install k3s with k3sup
- Flash an SD card with Ubuntu 22.04 or Raspberry Pi OS lite and install k3s with k3sup
Install the ingress-nginx via arkade install ingress-nginx
Install cert-manager to obtain certificates via Let’s Encrypt with arkade install cert-manager

Then, provide us with the list of domains you want to expose i.e. openfaas.example.com (replace “example.com” with your own domain).

For each domain name, create a DNS CNAME record to saas.inlets.dev.

Check that nslookup openfaas.example.com resolves to saas.inlets.dev.

We’ll then provide you with YAML for the inlets tunnel client, which creates a Deployment, apply it and then check its logs to see if it’s connected properly:

$ kubectl get deploy/alexellis-inlets-client
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
alexellis-inlets-client   1/1     1            1           21h

$ kubectl logs deploy/alexellis-inlets-client
               ___       __  
  __  ______  / (_)___  / /__
 / / / / __ \/ / / __ \/ //_/
/ /_/ / /_/ / / / / / / ,<   
\__,_/ .___/_/_/_/ /_/_/|_|  
    /_/                      

inlets (tm) uplink client: 0.9.21 - b0c7ed2beeb6f244ecac149e3b72eaeb3fb00d23
All rights reserved OpenFaaS Ltd (2023)
time="2024/06/18 10:18:55" level=info msg="Connecting to proxy"
time="2024/06/18 10:18:56" level=info msg="Connection established" client_id=4458bf47cf7a4022834ad42f67307e0d

To install OpenFaaS CE, you can use the Helm chart with TLS and Ingress enabled by following these instructions.

You should see your ingress entry, along with the domain you provided us with:

kubectl get ingress -n openfaas
NAME               CLASS   HOSTS                 ADDRESS   PORTS     AGE
openfaas-ingress   nginx   openfaas.example.com            80, 443   19h

Then, watch for the certificates to be obtained by cert-manger:

kubectl get certificates -A --watch

NAMESPACE   NAME                    READY   SECRET                  AGE
openfaas    openfaas-gateway-cert   True    openfaas-gateway-cert   19h

You can then access OpenFaaS via its UI or CLI, use arkade info openfaas for more instructions.

Use https://openfaas.example.com for the address for the OpenFaaS gateway, replacing the domain with your own.

Wrapping up

In a matter of seconds, you can start routing traffic to your Ingress Controller or Istio Gateway without having to set up any additional infrastructure, firewall rules, NAT, or cloud VMs. Just provide us with the DNS names for each website you want to host, create a DNS CNAME record to our public IP address, and our SaaS will take care of the rest.

OpenFaaS CE was chosen for a test application because its chart has built-in options for Ingress and TLS, and it’s relatively quick and easy to install. Of course you will have your own applications that you want to expose, and whilst we used one particular option for Ingress, there are many others and they’ll all work.

If you’d like to try out an Ingress Tunnel that’s hosted by the inlets team, then please get in touch with us via the website or reach out to me on X.

Q&A

Q. What does it cost for each Ingress Tunnel?

A. The hosted Ingress Tunnel will be free during our testing period. You can set up your own self-managed tunnel server at any time, either manually, with inletsctl or the inlets-operator.

Q. Who gets priority access to Ingress Tunnels?

A. We’ll try to keep up with demand, but anyone who is an existing inlets subscriber or who sponsors me via GitHub will get priority access.

Q. What if I want to expose more than one domain?

A. Just tell us the names of each, and we’ll configure the Ingress Tunnel for you.

Q. What if I want to expose more than one cluster?

A. That’s not a problem, each cluster will get its own tunnel client connection information.

Q. Can I expose a TCP port for a database or another service?

A. Services like MongoDB, Postgresql and NATS can be exposed via a self-managed inlets TCP tunnel server.

Q. How does this compare to Wireguard?

A. Wireguard is a VPN for connecting hosts privately, not for exposing services to the public Internet. Some inlets users use both - for different things.

Q. Can I expose my Raspberry Pi cluster?

A. Yes.

Q. If I run a cluster with KinD or K3d on my laptop, what happens when I go to a cafe?

A. When you shut down Docker, the tunnel will disconnect. It will reconnect when you restart Docker Desktop, whichever network you happen to be using your laptop on.

Q. If I’m already hosting a tunnel server, should I switch to an Ingress Tunnel?

A. If you’re content with your current setup, feel free to carry on as you are. But you’re welcome to test out the SaaS and see if it’s a better fit for you.