Pagination in Admin Api V2 #41036
Replies: 4 comments 6 replies
-
|
Please also consider the case where new users register while a paginated fetch is taking place. The current API returns results ordered by Id (Guid), so a new user can be inserted into page 1 when I am already fetching a later page.
|
Beta Was this translation helpful? Give feedback.
-
|
@shawkins My main concern here would be how do we identify "resources with an excessive number of entries". Obviously users, clients, but is that it? Users are doing all sorts of weird stuff – e.g. thousands of groups. |
Beta Was this translation helpful? Give feedback.
-
|
A very rough cut off would be to limit a single dataset / response document to 10 MB. Even in a pretty extreme case if you had a composite key and two sorting columns averaging 64 bytes each, you could fetch a keyset of around 30,000. Beyond users (which need even more special consideration due to their federated nature) and groups do you see clients and other resources having that instances? |
Beta Was this translation helpful? Give feedback.
-
|
Here are some other places where pagination has been requested: #17421 |
Beta Was this translation helpful? Give feedback.
-
|
Based upon the recent work for the new java client, another possible implementation path for streaming is to use Multi https://quarkus.io/guides/rest#streaming-support - this may require the usage of resteasy reactive on both the server and the client (https://javadoc.io/static/io.quarkus.resteasy.reactive/resteasy-reactive-client/3.8.0.CR1/org/jboss/resteasy/reactive/client/impl/MultiInvoker.html). The client side usage doesn't seem like a problem. We could use Multi on the server side and consider adding a custom serializer if needed for undertow. Also streaming is directly supported in the next rev of openapi - https://bump.sh/blog/json-streaming-openapi-3-2/ - that article covers a few of the specific media types. |
Beta Was this translation helpful? Give feedback.
-
|
Is switching to resteasy-reactive on the server really an option? I would assume that this is a pretty fundamental change in how Keycloak works? |
Beta Was this translation helpful? Give feedback.
-
|
@sschu we'd try to limit usage as much as possible - just a dependency on mutiny and usage of Multi. |
Beta Was this translation helpful? Give feedback.
-
|
@sschu @robson90 @vmuzikar it looks like we can do client side streaming via an rxinvoker: main...shawkins:keycloak:client-streaming Resteasy reactive is not explicitly needed on the server or the client side. I didn't really see anything that worked better - the next best option seemed to be switching from Stream to a flow Publisher. But we then have to back that with a RestMulti via a factory method and may need to employ some other workarounds as well. It's worth noting that we have seen users sometimes trying to use their own Client or customJacksonProvider in the admin client. That's getting even more problematic - so we may want to think more about how we register providers. |
Beta Was this translation helpful? Give feedback.
-
|
@shawkins Maybe we could draft PR with that so we can discuss the proposal in a wider audience? |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
@sschu just wanted capture some of what I was trying to convey in the meeting and some points that were mentioned in earlier design docs. Please add what you have, and I'll add some more about using a continuation token.
Additional Considerations:
For all resources which will not have an excessive number of entries (possibly everything except for users) we don't need to offer sort and slice features. Instead admin api clients via projection would obtain the indentifier fields and any sorting columns, perform their own client-side sort, then fetch whatever pages they want by identifier. We could add support for bulk fetching by id to cut down on the number of api queries.
For resources with an excessive number of entries, or for which we do want to offer sorting / slicing:
Skipping pages isn't strictly necessary, but is a nice to have feature.
cc @mabartos @vmuzikar
Beta Was this translation helpful? Give feedback.
All reactions