production hostname and proxy settings confusion #12466

XcrigX · 2022-06-10T23:11:42Z

XcrigX
Jun 10, 2022

I'm struggling with how to configure a production instance of Keycloak.

Here is my particular setup:
I'm deploying the docker image quay.io/keycloak/keycloak:18.0.0 (quarkus version) to an Azure Kubernetes (k8s) cluster. Everything in my deployment is in a private intranet except a public facing gateway/waf.

Client traffic is coming in from the internet to a public endpoint of:
https://mydomain.com/keycloak

This traffic is forwarded from a gateway to a (private) Azure API Manager (APIM) instance.
From APIM, I redirect to the actual backend keycloak service running in K8s.

The k8s keycloak service is fronted by an Azure load balancer on a fixed private IP - call it 10.20.3.60.
I created a private DNS record for 10.20.3.60 with a hostname keycloak.internal.private.
I cut a self-signed certificate for 'CN=keycloak.internal.private' and configured keycloak to use it so the traffic to keycloak is encrypted internally.
(I'm also port mapping the internal 8443 port to 443 on the load balancer).

So the external TLS is being terminated at APIM, then forwarded as another TLS request to an internal hostname.

So I have:
(External) https://mydomain.com/keycloak ->
[gateway] -> [APIM] ->
(Internal) https://keycloak.internal.private
-> [k8s pod] - keycloak listening on 0.0.0.0:8443

So far so good, I have it deployed and I can hit the keycloak /health endpoint on the external and internal URLs.
https://mydomain.com/keycloak/health
https://keycloak.internal.private/health

Now it's a little hazy how to complete the configuration.
I (think) I need:

To be able to access the keycloak web console from my intranet - eg) https://keycloak.internal.private/admin/*
For internal services to be able to communicate with keycloak to validate token signatures, etc.
For external clients to request auth tokens, etc. - eg) https://mydomain.com/keycloak/myrealm/auth

I'm confused on how these parameters need to be set up:
KC_HOSTNAME, KC_HOSTNAME_PORT, KC_HOSTNAME_STRICT_BACKCHANNEL, KC_PROXY

I think I'm maybe confused because I have two hostnames, essentially - and some of the terminology used in the guides feels ambiguous in this case.

KC_HOSTNAME - Do I use my internal or external hostname? I assume internal - since I need keycloak to construct correct web urls when using the admin web console. But will this affect external clients in any way? Like If I'm brokering to an external identify provider - will the redirect URLs incorrectly try to use my internal hostname or other similar effects?

KC_PROXY - edge, reencrypt, passthrough - What mode describes my scenario?

If I'm passing /auth requests from the internet through APIM, the scenario looks like reencrypt.
TLS is terminated at APIM, then the request forwarded as another TLS request to my internal hostname.

If I'm hitting the admin console from within my intranet there is no reverse-proxy terminating TLS.
The docs don't actually describe what effects these have - or address different scenarios for internal/external traffic. I don't quite understand why keycloak needs to know if my proxy is terminating TLS and re-encrypting or not- seems like that would be invisible to keycloak.

KC_HOSTNAME_STRICT_BACKCHANNEL - ? I want this false? - but the name is confusing. It seems like I'd only want backend endpoints to be accessible internally, but the docs say to set it to true when all traffic hits the public url.

KC_HOSTNAME_PORT - do I need to mess with this - the actual keycloak service is listing on 8443, but the k8s service is listening on 443.

I appreciate any help/enlightenment anyone can provide.
Thanks!

Answered by XcrigX

Jun 16, 2022

Solved the problem of accessing keycloak from my intranet too. It turns out that when accessing the keycloak portal from my private hostname, the logon dialog still uses the public hostname - and my jumphost in the intranet was not resolving my public hostname. This looks like a likely keycloak bug to me.

In other words - when accessing keycloak portal on my intranet and trying to navigate to here:
https://keycloak.internal.private/admin/master/console/

In the Chrome debug trace, I saw an error trying to hit the public name:
https://mydomain.com/kc/realms/master/protocol/openid-connect/3p-cookies/step1.html?version=uz088

Once I added the hostname/ip to the hosts file - it started working.

View full answer

XcrigX · 2022-06-13T20:19:50Z

XcrigX
Jun 13, 2022
Author

For reference this is the Docker image I'm building and deploying to Kubernetes.

FROM quay.io/keycloak/keycloak:18.0.0 as builder

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=token-exchange

# Install custom providers
RUN curl -sL https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar -o /opt/keycloak/providers/keycloak-metrics-spi-2.5.3.jar

RUN /opt/keycloak/bin/kc.sh build --db=mssql --transaction-xa-enabled=false --health-enabled=true

WORKDIR /opt/keycloak

RUN /opt/keycloak/bin/kc.sh show-config
ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start"]

0 replies

XcrigX · 2022-06-13T20:28:56Z

XcrigX
Jun 13, 2022
Author

I'm trying various things, so far no luck.

I am currently setting:

       - name: KC_HOSTNAME
          value: mydomain.com
        - name: KC_HOSTNAME_PATH
          value: kc
        - name: KC_PROXY
          value: reencrypt

I can now make a call from postman to get an OAuth token and I get the keycloak login dialog at this URL:
https://mydomain.com/kc/realms/myrealm/protocol/openid-connect/auth?response_type=code&client_id=postman-client&scope=launch%2Fpatient%20patient%2F*.read&redirect_uri=https%3A%2F%2Foauth.pstmn.io%2Fv1%2Fcallback&code_challenge=CvRJENmQ2y17_VurJ5N2KL7rLpQJwfZqkOL28LLQ08g&code_challenge_method=S256

However, as soon as I enter credentials, it directs to here and I get a 404:
https://mydomain.com/kc/realms/myrealm/login-actions/authenticate?session_code=H3FmaVYn0cmPtA4PHyrGbmv474J9AuUIdhpm5o76G7g&execution=da3916a7-a45e-4862-90f8-239d50fbb46d&client_id=postman-client&tab_id=JTkrRoqy25w

Also with the above config, I can reach the welcome page from my intranet with:
https://keycloak.internal.private/
But when I click the admin console it goes to a blank page with this URL:
https://keycloak.internal.private/admin/master/console/

I'll keep hammering at it, but would greatly appreciate if anybody happens along with some insight. Thanks.

0 replies

XcrigX · 2022-06-14T16:00:56Z

XcrigX
Jun 14, 2022
Author

Here is a simplified diagram of what I'm trying to achieve - this should be supported, yes?

0 replies

XcrigX · 2022-06-14T19:57:42Z

XcrigX
Jun 14, 2022
Author

Solved the larger part of my problem - my proxy/apim instance was only allowing GET requests through, and the intermediate call made after logging in from the browser is a POST to /login-actions/authenticate?... .

So I can authenticate external clients now.
Now if I can figure out why I can't reach the admin portal from my intranet I'll be off to the races.

0 replies

XcrigX · 2022-06-16T14:21:52Z

XcrigX
Jun 16, 2022
Author

Solved the problem of accessing keycloak from my intranet too. It turns out that when accessing the keycloak portal from my private hostname, the logon dialog still uses the public hostname - and my jumphost in the intranet was not resolving my public hostname. This looks like a likely keycloak bug to me.

In other words - when accessing keycloak portal on my intranet and trying to navigate to here:
https://keycloak.internal.private/admin/master/console/

In the Chrome debug trace, I saw an error trying to hit the public name:
https://mydomain.com/kc/realms/master/protocol/openid-connect/3p-cookies/step1.html?version=uz088

Once I added the hostname/ip to the hosts file - it started working.

8 replies

jaymberry Jun 28, 2022

ok, thank you

XcrigX Jun 28, 2022
Author

@jaymberry - I'm not certain what your problem may be, but if I read it correctly, Grafana is trying to hit Keycloak's token endpoint and failing to reach it. I notice two things there

it's trying to hit localhost - is Grafana running on the same host as Keycloak ? If not then the configured token URL needs to be something Grafana can reach to get to keycloak wherever keycloak is hosted.
it's hitting https://localhost... This doesn't look right - unless have created a TLS certificate for 'localhost' which Keycloak is presenting, and Grafana trusts.

jaymberry Jun 29, 2022

Solved, the localhost was catching on pod container of grafana (unable to connect out).

Changing the host machine's dns to match the keycloak service name (i.e. service/keycloak) and adding that with port 8443 in grafana config (i.e. https://keycloak:8443) solved the problem.

Thank you for your info.

WalkerWalker Jul 2, 2022

Oh my god. This post exactly what I am looking for. My setup is mostly the same, except that I use traefik for the reverse proxy and running keycloak with docker-compose. All the services traefik, keycloak, postgres are within the docker network.

My situation is also mostly the same. Now https://keycloak.mydomain.com/ would be successfully redirect to the keycloak service, which listens to port 8080. But once I click the administrator console, https://keycloak.mydomain.com/admin/master/console/ would show just the blank page.

This one line solution confuses me

Once I added the hostname/ip to the hosts file - it started working.

Could you please elaborate a bit more on how to solve it?

XcrigX Jul 3, 2022
Author

@WalkerWalker - There are a bunch of discussions and stackoverflow posts about blank admin page. There could be several scenarios, I gather.
But I was referring to DNS name resolution. The machine I was running keycloak on could not resolve the public hostname of my keycloak server. The Azure DNS service it was using did not have an entry for "keycloak.mydomain.com" (that's not the actual name I'm using, just for illustrative purposes).
You can edit the "hosts" file, on Linux and Windows systems, and manually enter DNS entries (hostname - IP pairs). DNS requests always check the hosts file first, before going out to an external DNS service for name resolution.
On linux, this file located at /etc/hosts. On windows it's in typically located at C:\Windows\System32\drivers\etc\hosts.

You could probably run a chrome in debug mode (F12) and look at the 'network' tab as you try to get the admin console and see which requests are failing - that may give a clue where your problem lies. It could be that your problem is similar, that keycloak is trying to hit some URL that your machine cannot resolve.

production hostname and proxy settings confusion #12466

Uh oh!

XcrigX Jun 10, 2022

Replies: 5 comments · 8 replies

Uh oh!

XcrigX Jun 13, 2022 Author

Uh oh!

Uh oh!

XcrigX Jun 13, 2022 Author

Uh oh!

XcrigX Jun 14, 2022 Author

Uh oh!

XcrigX Jun 14, 2022 Author

Uh oh!

XcrigX Jun 16, 2022 Author

Uh oh!

jaymberry Jun 28, 2022

Uh oh!

XcrigX Jun 28, 2022 Author

Uh oh!

jaymberry Jun 29, 2022

Uh oh!

WalkerWalker Jul 2, 2022

Uh oh!

XcrigX Jul 3, 2022 Author

XcrigX
Jun 10, 2022

Replies: 5 comments 8 replies

XcrigX
Jun 13, 2022
Author

XcrigX
Jun 13, 2022
Author

XcrigX
Jun 14, 2022
Author

XcrigX
Jun 14, 2022
Author

XcrigX
Jun 16, 2022
Author

XcrigX Jun 28, 2022
Author

XcrigX Jul 3, 2022
Author