Problem statement

The current pagination for the map storage works by providing as part of the query an offset and a limit. To make this work, this should be combined with a stable sorting mechanism to deliver predictable results.

For JPA, this is achieved by first flushing all changes to the database, and then doing a paged query to the database. JPA supports pages queries with a start index.

For LDAP, this is not possible as flushing of the data to LDAP is done only at the end of the Keycloak transaction as LDAP doesn't allow rollback of transactions. For LDAP, a correct result is achieved by:

• get all results from LDAP matching the query, then eliminating the entries already stored in the transaction as live objects
• get all results from the transaction that match the LDAP query
• combine the results and sort them
• skip the first N entries from the stream, then limit the stream to the expected number of results

While this limits the number of returned entries, all information is fetched from LDAP that matches the query. With the entries already in the transaction it is impossible to determine the start and end offset when querying LDAP.

Assuming a scenario with a tree store combining the results from multiple stores, pagination becomes even more difficult as stores won't find out where to start with the pagination, and could only stop with results retrieved after start offset plus page size is reached.

Approach

• Instead of passing a start offset, the caller passes the ID of the last entry of the previous page.
• Upon retrieving a paged query, the the sorting criteria are amended to make the sorting stable (for example adding the primary key)
• From the last entry of the previous page, the first ordering attribute is retrieved to limit the search to "X >= value"
• From the stream returned, skip all entries until the previously known entries is seen
• Limit the result to the expected page size

In the example above, this can optimize the list retrieved from LDAP. Also entities in the transaction can be retrieved in the restricted way described above. The two result sets are then merge-sorted, and then again limited.

This can be further optimized when the first ordering by value is known to be unique: Then "X > value" can be used, and no entries need to be skipped from the result stream.

Benefits

Pagination is more efficient for LDAP and tree store.

Alternatives

• Keep as is
• Optimize LDAP to apply pagination "the LDAP way" in cases where there is no updated element in the transaction - still any tree store that combines results from different sources can't use a start offset
• When identifying the entities to return, only apply the query to LDAP and don't search for extra entities that have been created/modified in the current transaction as it would hurt pagination despite the result being less correct