Skip to content

Allow importing realms during startup#10754

Merged
stianst merged 1 commit intokeycloak:mainfrom
pedroigor:issue-9261
Mar 24, 2022
Merged

Allow importing realms during startup#10754
stianst merged 1 commit intokeycloak:mainfrom
pedroigor:issue-9261

Conversation

@pedroigor
Copy link
Copy Markdown
Contributor

@pedroigor pedroigor commented Mar 15, 2022

  • Re-enable the ability to import realms at startup similarly as when using the legacy distribution and the keycloak.import system property
  • Adds an import-realm option to both start and start-dev commands to import realm configuration files from the data/import directory. No need to manually specify files, reducing the chance of errors, and reducing the number of steps when importing realms using containers.
  • A new guide to cover the different strategies for importing and exporting realms.

Closes #9261

Copy link
Copy Markdown
Contributor

@DGuhr DGuhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pedroigor Not looked at the code yet, but first review round about the --dir export:
I ran kc.sh export --dir=. after setting up the initial admin user using start-dev.

expectation:
master-realm.json and master-user.json appear in the bin directory. Success msg is shown.

Outcome:

  1. Success-MSG is shown

Suggestion: It states for users org.keycloak.exportimport.dir.DirExportProvider] (main) Users 0-0 exported - can we make this a bit more concise? e.g. "Successfully exported {count} users to file {filename} in dir {dir}"? Users 0-0 sounds a bit weird imo.

  1. files get created, but users-file has a "0" at the end. so called master-users-0.json - I think it would be better when we have only master-users.json as you state below in the docs when there are <50 users to export, and then go on with master-users-1.json for chunks > 1 if any?

General question: This PR allows to use separate commands for import/export. to really allow to "startup and import" in one step, one would now run the container image using...? first import, then start-dev afterwards? Looking at https://www.keycloak.org/server/containers I just wonder how this would fit in the examples for the entrypoint / development mode, and I guess it wouldn't without spinning up the container twice, right? I am wondering if that's really the desired behaviour here. I would've thought it's an env config like KC_IMPORT_DIR=dir/with/my/files and when starting up the container using start/start-dev and this environment var is set, import happens automagically?!

ok forget that last one, just read about the import-realm being a subcommand of start/start-dev :)

Copy link
Copy Markdown
Contributor

@DGuhr DGuhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done with the smoke tests for "export to dir" part for now.

tested:

  • export to dir works in general
  • specifying a file instead of a directory throws error
  • specifying --users-per-file=0 throws an error (division to zero, maybe we want to change the message? I still need types for the CLI ;) )
  • different user export modes are working (see specific comments and my comment about the success msg and file ending for that one, i'd like to remove the -0 to stay consistent)
  • test "non-usual characters and emojis" in username: get exported correctly. see image with list of users i tried:
    Screenshot 2022-03-18 at 11 19 14

Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Comment thread docs/guides/src/main/server/importExport.adoc Outdated
Copy link
Copy Markdown
Contributor

@DGuhr DGuhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall very nice, thanks for that 👍 All of the change requests are minor ones i guess. Will start the actual code review later on

@pedroigor
Copy link
Copy Markdown
Contributor Author

@pedroigor Not looked at the code yet, but first review round about the --dir export: I ran kc.sh export --dir=. after setting up the initial admin user using start-dev.

expectation: master-realm.json and master-user.json appear in the bin directory. Success msg is shown.

Outcome:

  1. Success-MSG is shown

Suggestion: It states for users org.keycloak.exportimport.dir.DirExportProvider] (main) Users 0-0 exported - can we make this a bit more concise? e.g. "Successfully exported {count} users to file {filename} in dir {dir}"? Users 0-0 sounds a bit weird imo.

  1. files get created, but users-file has a "0" at the end. so called master-users-0.json - I think it would be better when we have only master-users.json as you state below in the docs when there are <50 users to export, and then go on with master-users-1.json for chunks > 1 if any?

General question: This PR allows to use separate commands for import/export. to really allow to "startup and import" in one step, one would now run the container image using...? first import, then start-dev afterwards? Looking at https://www.keycloak.org/server/containers I just wonder how this would fit in the examples for the entrypoint / development mode, and I guess it wouldn't without spinning up the container twice, right? I am wondering if that's really the desired behaviour here. I would've thought it's an env config like KC_IMPORT_DIR=dir/with/my/files and when starting up the container using start/start-dev and this environment var is set, import happens automagically?!

ok forget that last one, just read about the import-realm being a subcommand of start/start-dev :)

I'm confused, could you elaborate more, please?

@pedroigor
Copy link
Copy Markdown
Contributor Author

done with the smoke tests for "export to dir" part for now.

tested:

  • export to dir works in general
  • specifying a file instead of a directory throws error
  • specifying --users-per-file=0 throws an error (division to zero, maybe we want to change the message? I still need types for the CLI ;) )
  • different user export modes are working (see specific comments and my comment about the success msg and file ending for that one, i'd like to remove the -0 to stay consistent)
  • test "non-usual characters and emojis" in username: get exported correctly. see image with list of users i tried:
    Screenshot 2022-03-18 at 11 19 14

Could you please make more clear what you think should change in this PR?

@DGuhr
Copy link
Copy Markdown
Contributor

DGuhr commented Mar 18, 2022

@pedroigor Not looked at the code yet, but first review round about the --dir export: I ran kc.sh export --dir=. after setting up the initial admin user using start-dev.
expectation: master-realm.json and master-user.json appear in the bin directory. Success msg is shown.
Outcome:

  1. Success-MSG is shown

Suggestion: It states for users org.keycloak.exportimport.dir.DirExportProvider] (main) Users 0-0 exported - can we make this a bit more concise? e.g. "Successfully exported {count} users to file {filename} in dir {dir}"? Users 0-0 sounds a bit weird imo.

  1. files get created, but users-file has a "0" at the end. so called master-users-0.json - I think it would be better when we have only master-users.json as you state below in the docs when there are <50 users to export, and then go on with master-users-1.json for chunks > 1 if any?

General question: This PR allows to use separate commands for import/export. to really allow to "startup and import" in one step, one would now run the container image using...? first import, then start-dev afterwards? Looking at https://www.keycloak.org/server/containers I just wonder how this would fit in the examples for the entrypoint / development mode, and I guess it wouldn't without spinning up the container twice, right? I am wondering if that's really the desired behaviour here. I would've thought it's an env config like KC_IMPORT_DIR=dir/with/my/files and when starting up the container using start/start-dev and this environment var is set, import happens automagically?!
ok forget that last one, just read about the import-realm being a subcommand of start/start-dev :)

I'm confused, could you elaborate more, please?

  1. Change the message when exporting users to a directory from Users 0-0 exported to a more descriptive one. e.g. foo-users.json exported to my/target/directory.

  2. Change user json file creation, so no -0 is appended to the filename when users < users-per-file value.

Example: kc.sh export --dir=. for a foo realm should return a foo-realm.json and a foo-users.json (as stated in the adoc in this PR), but actually generates foo-realm.json and foo-users**-0**.json. When foo users > users-per-file, it should return foo-realm.json, foo-users-<1-n>.json

For the striked through part: forget about it, as said.

@DGuhr
Copy link
Copy Markdown
Contributor

DGuhr commented Mar 18, 2022

done with the smoke tests for "export to dir" part for now.
tested:

  • export to dir works in general
  • specifying a file instead of a directory throws error
  • specifying --users-per-file=0 throws an error (division to zero, maybe we want to change the message? I still need types for the CLI ;) )
  • different user export modes are working (see specific comments and my comment about the success msg and file ending for that one, i'd like to remove the -0 to stay consistent)
  • test "non-usual characters and emojis" in username: get exported correctly. see image with list of users i tried:
    Screenshot 2022-03-18 at 11 19 14

Could you please make more clear what you think should change in this PR?

for that specific comment, nothing right now. just making clear what I manually tested here for future reference. There are some edge cases not covered by the testsuite, so when I see these I at least document I manually tested them. (e.g. users-per-file=0 leads to an expected error when used, usernames can contain accents like_é_, emojis etc.

@dasniko
Copy link
Copy Markdown
Contributor

dasniko commented Mar 19, 2022

Didn't test the export yet, but auto-import on server startup works for me. 👍
Thanks for the effort! 🙏

@stianst stianst merged commit e177f90 into keycloak:main Mar 24, 2022
@ML-ZoneReaper
Copy link
Copy Markdown
Contributor

@stianst it's a pity that you didn't add this functionality in release 17.0.1 😞 will we get the import only in the 18.0.0 version?

@DGuhr
Copy link
Copy Markdown
Contributor

DGuhr commented Mar 24, 2022

@eabykov yes, it is targeted for 18

@tpischke-bedag
Copy link
Copy Markdown

@stianst it's a pity that you didn't add this functionality in release 17.0.1 disappointed will we get the import only in the 18.0.0 version?

17.0.2?

@ruckc
Copy link
Copy Markdown

ruckc commented Mar 30, 2022

17.1.0?

@ML-ZoneReaper
Copy link
Copy Markdown
Contributor

17.1.0?

@ruckc wait just a bit, it looks like the 18.0.0 version that will be due by March 31, 2022

@tpischke-bedag
Copy link
Copy Markdown

17.1.0?

@ruckc wait just a bit, it looks like the 18.0.0 version that will be due by March 31, 2022

With 45 open issues?

@ML-ZoneReaper
Copy link
Copy Markdown
Contributor

17.1.0?

@ruckc wait just a bit, it looks like the 18.0.0 version that will be due by March 31, 2022

With 45 open issues?

@tpischke-bedag It wasn't me who compiled Milestones 🤣

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import realm at startup

7 participants