If I understand properly what happens in DefaultCacheEmbeddedConfigProviderFactory and CacheConfigurator:

• clusterless feature completely removes all embedded caches for all distributed and replicated caches
• persistent user sessions feature maintains a layer of embedded distributed caches for user, client, offline user and offline client sessions

And if both are activated, clusterless feature takes priority.

@jfgaret-docto - for us to prioritize and shape this feature better, it would be good to know why you would prefer this setup over other setups, and in that scenario you are planning to use the clusterless setup.

As you are using persistent sessions, I would assume that you are not storing a lot of data in the external Infinispan. And for those setups, the embedded Infinispan is usually a good choice. So I am really curious to learn more.

In the meantime, clusterless without persistent sessions (where all user sessions are stored in the external Infinispan) is one possible idea for a Keycloak cluster that is servicing lot of requests (10k-1000k rps) where the Infinispan would allow for horizontal scaling of the storing of sessions, and the Keycloak instances would allow horizontal dynamic scaling of the compute power. As persistent sessions would not be used in this setup, the database would be no longer be a bottleneck. Still, such a setup would come with the downside that ISPN would store the sessions only in memory. While storage can be added to ISPN, this isn't part of Keycloak's tests and documentation today.

@ahus1 The issue we face and are trying to solve is to have zero-downtime rollouts (outside of major version upgrades) of keycloak.
We have some steady traffic (several hundreds rps) and when we restart keycloak pods, we can see an increase in network traffic that we attribute to the embedded cache clustering (rebalancing data owned by the pod being taken down). That increase is also correlated with an increase in p99 response times, leading to temporary degraded service to our users.

As we rely on persistent user sessions and an already setup remote cache, we did not see a strong value in having an additional embedded clustered cache (minor improvement on latency - a few ms per requests). We don't have clear metrics to estimate amount of cache hits within the embedded cache, or the ratio of cache lookups in embedded cache / remote cache, to backup this perceived value.

I hope that clarifies what we were looking into.

Based on the feedback from @pruivo and our code analysis, it seems like multi-site would work well enough for us as well, even if we make use of this feature with a single active site. We will test it out.

Thank you @jfgaret-docto for the feedback on the latency. Do you know which version you were using?

With KC 26.3, jdbc-ping is using the per-destination bundler, which might better effects when one node is slow when warming up or shutting down. Having your data-point would be great. Thanks!

@ahus1 We are on 26.2.5 right now.