Hi all.
In our production environments, we have a keycloak setup running in HA mode (with a couple of kc nodes) and a load balancer (nginx) sitting in front.
In our environments, all the keycloak clusters serve a large federation of Identity Providers, currently counting more than 4k IdPs.
We use custom themes (like this one), to show a different WAYF login page to our users, since the keycloak's default does not cope well with such large numbers of IdPs (no paging, no searching, etc).
We also use a few optimisations on the core side, (like this one), which fetches the IdPs in a paged manner (just like the Clients and the Users), and also removes the Idps and the IdpMappers from the Realm serialization sent to the browser (also mended the realm export function).
Up to the recent days, our keycloak instances (v16.1.0) work like a charm in the production environment.

However, we face a large problem whenever an admin updates anything in the realm (even on a 'simple' IdP mapper property) from the admin UI.
This update triggers a full realm invalidation, which means that the realm needs to be reloaded on all of our keycloak nodes.
Our realm contains apart from the 4k IdPs, a couple of mappers for each one of these IdPs, making the reloading from the database even worse. The problem is, that during the reload, all nodes stop responding, which is quite bad for any end user who tries to use keycloak (i.e. tries to login). With some fine tuning we have made, it takes around 10 seconds to reload the realm after each update.
This is expected to increase, as the number of Identity providers increase.

We have considered different approaches:

1. let infinispan reload the realm in a asynchronous way, but i strongly believe that such a thing could have serious implications
2. use a different store?
3. decouple the IdentityProvider/IdentityProviderMapper from the Realm

Option (3) is consistent with the current Keycloak design for Clients and Users and it scales as the number of IdentityProviders and their mappers increase in terms of memory consumption and queries speed. Would you be interested in a PR implementing this? Is there anything else you would propose to address the issue?