feat(internal/librarian/java): appends to versions.txt when new modules is added in generate#5660
feat(internal/librarian/java): appends to versions.txt when new modules is added in generate#5660zhumin8 wants to merge 13 commits intogoogleapis:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request modifies the Java library generation process to track and record newly created artifact versions. It updates the Generate, postProcessLibrary, and syncPOMs functions to return a list of version strings, which are then appended to versions.txt during the PostGenerate phase. The review feedback highlights an empty test function that should be removed or implemented and suggests adding the os.O_CREATE flag when opening versions.txt to ensure the file is created if it does not exist.
| return command.Run(ctx, "unzip", "-q", "-o", src, "-d", dest) | ||
| } | ||
|
|
||
| // AppendLines appends the given lines to an existing file, ensuring that it |
There was a problem hiding this comment.
Is this function for linting? If so, a few questions (again mostly for me to learn):
- Is there no go linting tool that adds a newline automatically, if that's what we're trying to account for?
- Is there a go library that adds a newline?
There was a problem hiding this comment.
- No, this function is used to append new entries to versions.txt file. versions.txt exists in google-cloud-java, and although not encouraged, can be edited by human and I believe does not have a check for trailing empty line.
- I am not aware of. I think we need to read in the file and write.
| // AppendLines appends the given lines to an existing file, ensuring that it | ||
| // ends with a newline character before appending. It returns an error if the | ||
| // file does not exist. | ||
| func AppendLines(path string, lines []string) (err error) { |
There was a problem hiding this comment.
what about simplifying this function to this:
func AppendLines(path string, lines []string) error {
if len(lines) == 0 {
return nil
}
existing, err := os.ReadFile(path)
if err != nil {
return err
}
var buf bytes.Buffer
buf.Write(existing)
if len(existing) > 0 && existing[len(existing)-1] != '\n' {
buf.WriteByte('\n')
}
for _, line := range lines {
buf.WriteString(line)
buf.WriteByte('\n')
}
return os.WriteFile(path, buf.Bytes(), 0644)
}
This will rewrite the whole file each time, but for a small file like versions.txt it seems cleaner
What I don't quite understand is what we need to append the newline at all though?
There was a problem hiding this comment.
why we need to append the newline at all though?
It's more of a safety check. Although not encouraged, this versions.txt can be edited by human and I believe there is no check for trailing empty line in google-cloud-java. Alternatively, we can add a dedicated workflow in google-cloud-java to check for it. But I think it's easier to check when attempting to update.
| } | ||
| } | ||
|
|
||
| releasedVersion, err := deriveLastReleasedVersion(library.Version) |
There was a problem hiding this comment.
what if a library has no versions to parse?
There was a problem hiding this comment.
A Java library should always have version. librarian add sets a default version
librarian/internal/librarian/java/add.go
Line 37 in b92da72
| var allNewVersions []string | ||
| for _, library := range libraries { | ||
| if err := java.Generate(ctx, cfg, library, src); err != nil { | ||
| newVersions, err := java.Generate(ctx, cfg, library, src) |
There was a problem hiding this comment.
we should try to keep the same API interface across languages for Generate. what about moving the post generate logic inside java.Generate?
| // Generate generates a Java client library. | ||
| func Generate(ctx context.Context, cfg *config.Config, library *config.Library, srcs *sources.Sources) error { | ||
| // Generate generates a Java client library and returns a list of newly created | ||
| // artifact version entries to be added to versions.txt. |
There was a problem hiding this comment.
I see we will stop adding multiple lines into version txt for proto- and grpc- artifacts #4650.
There was a problem hiding this comment.
Nope. #4650 is to unify the versions. No changes to versions.txt structure. We can potentially do that, but it does not change that we still need append to versions.txt when new library onboards.
|
|
||
| // Generate generates a Java client library. | ||
| func Generate(ctx context.Context, cfg *config.Config, library *config.Library, srcs *sources.Sources) error { | ||
| // Generate generates a Java client library and returns a list of newly created |
There was a problem hiding this comment.
Can you define the canonical way within Java to detect a library is new or not?
(I believe that the responsibility of detecting the condition is not in Generate function.)
There was a problem hiding this comment.
@julieqiu your comment https://github.com/googleapis/librarian/pull/5660/changes#r3156502889 is related, I'll bring this discussion together:
- Currently, Generate does post processing steps that generates pom.xml files when it is a new library (that is where new library/module is detected and bubbled up).
- I choose to change Generate to return these version entries and let
java.PostGeneratewrite to versions.txt, because versions.txt is located at repo root, and it's natural to update it once for the whole generation. It should be technically possible to move inside java.Generate and workout the parent dir of outdir, but it feels a bit wrong as a per library function updates a repo level file, and may be problematic if we seek to do parallel in the future. - By the time it gets to
java.PostGenerate, pom.xml are already added, I don't see a way to identify which library is new. @suztomo any ideas you have in mind?
Appends version line to versions.txt file for each new module added in generate. The implememtation takes advantage that
syncPOMsalready detects which module is missing, and alters java.Generate to return a list of newly created artifact version entries to be added to versions.txt. Then as a monorepo level post generate task, append these to versions.txt which is at the repo root.Context: versions.txt is a release-please file that tracks the released version and snapshot version for artifacts. Until Java onboards to librarian for releases, we need to update this versions.txt file when new artifacts is added by generate. This happens when new library is introduced or new version API path is added to existing libraries.
Fix #4927