Skip to content

Migrate ssl package to new test framework#48407

Open
michalvavrik wants to merge 1 commit intokeycloak:mainfrom
michalvavrik:chore/issues/47812
Open

Migrate ssl package to new test framework#48407
michalvavrik wants to merge 1 commit intokeycloak:mainfrom
michalvavrik:chore/issues/47812

Conversation

@michalvavrik
Copy link
Copy Markdown
Member

@michalvavrik michalvavrik commented Apr 23, 2026 •
edited
Loading

  • Closes: Migrate ssl package to new test framework #47812
  • As far as the email goes, I thought it would be better to avoid adding extra dependency (SubEthaSmtp) to the new test framework for the SMTPS feature that only tests added in this PR are using. So I turned around the paradigm (change truststore on the server side, not keystore on the mail server side). I believe all the original scenarios are tested, you can compare it yourself:
Old test method New test
test01VerifyEmailWithSslWrongCertificate TrustStoreEmailUntrustedCertTest#testVerifyEmailWithUntrustedCert
test02VerifyEmailWithSslWrongCertificateAndAnyHostnamePolicy TrustStoreEmailUntrustedCertAnyHostnameTest#testVerifyEmailWithUntrustedCertAndAnyHostnamePolicy
test03erifyEmailWithSslWrongHostname TrustStoreEmailTest#testVerifyEmailWithSslWrongHostname
test04VerifyEmailWithSslEnabled TrustStoreEmailTest#testVerifyEmailWithSslEnabled
test05VerifyEmailWithSslWrongHostnameButAnyHostnamePolicy TrustStoreEmailAnyHostnameTest#testVerifyEmailWithSslWrongHostnameSucceeds
test06VerifyEmailOpportunisticEncryptionWithAnyHostnamePolicy TrustStoreEmailPlainAuthTest#testVerifyEmailWithPlainSmtpAuth

@michalvavrik michalvavrik requested review from a team as code owners April 23, 2026 12:17
@michalvavrik michalvavrik requested a review from vmuzikar April 23, 2026 12:37
@keycloak-github-bot keycloak-github-bot Bot mentioned this pull request Apr 23, 2026
Open
keycloak-github-bot[bot]
keycloak-github-bot Bot reviewed Apr 23, 2026
View reviewed changes
Copy link
Copy Markdown

@keycloak-github-bot keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link
Copy Markdown

keycloak-github-bot Bot commented Apr 23, 2026

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Keycloak CI - Base IT (3)

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

Report flaky test

@michalvavrik michalvavrik marked this pull request as draft April 23, 2026 14:52
@michalvavrik michalvavrik marked this pull request as ready for review April 23, 2026 16:59
keycloak-github-bot[bot]
keycloak-github-bot Bot reviewed Apr 23, 2026
View reviewed changes
Copy link
Copy Markdown

@keycloak-github-bot keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link
Copy Markdown

keycloak-github-bot Bot commented Apr 23, 2026

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Keycloak CI - Base IT (3)

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

Report flaky test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vmuzikar vmuzikar Awaiting requested review from vmuzikar

1 more reviewer

@keycloak-github-bot keycloak-github-bot[bot] keycloak-github-bot[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

flaky-test team/cloud-native team/core-clients team/core-iam

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Migrate ssl package to new test framework

1 participant

@michalvavrik