WAS integration: run e2e tests using "main" kubernetes to proactively ensure compatibility

[mimowo]

What would you like to be added:

With the ongoing work in kubernetes on WAS https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4671-gang-scheduling there is some risk for incompatibility.

I imagine a dedicated CI job which runs periodically every day. Our alerting will tell us if the compatibility is broken.

Why is this needed:

In that case we should know early, before k8s is released, so that we can adjust Kueue, or flag early the compatibility problems.

[mimowo]

cc @tenzen-y @gabesaba @mbobrovskyi @sohankunkerkar @kannon92

[kannon92]

I would probably not suggest this.

But we could consider a dedicated job with WAS enabled on kind clusters (1.35 and on).

But honestly we haven't discussed the impact of WAS on Kueue.

[kannon92]

Right now, in 1.36 we may get Job gang scheduling. But there is so much in flight.

[mimowo]

I would probably not suggest this.

Why?

But we could consider a dedicated job with WAS enabled on kind clusters (1.35 and on).

IIUC currently 1.35 WAS is alpha only, so I think it is not that relevant. It becomes onteresting on "main".

We could certainly have two jobs:

• "main" with WAS enabled
• "1.36" with WAS enabled

But it feels "main" is more important to get early feedback about incompatibilities.

But honestly we haven't discussed the impact of WAS on Kueue.

I'm actually part of some internal discussions currently, but the integration is certainly coming.

Discussions are ongoing, but no "dry" discussion can replace actual testing IMO. Even if we discuss and conclude "no impact", the impact might be, and test can help us to spot the issues early.

[kannon92]

Why

What do we do if kubernetes main is breaking? Should we do local-up-cluster or kind builds? I guess we could make the job informing and monitor it but not block. But someone has to sign up to maintain this job and make sure things keep running.

Its a pretty large cost to maintain main branches.

[mimowo]

What do we do if kubernetes main is breaking? Should we do local-up-cluster or kind builds? I guess we could make the job informing and monitor it but not block.

Yes, that was my plan, just make it informative periodic build.

Its a pretty large cost to maintain main branches.

I'm not sure I understand what you mean here. It wouldn't be the cost on Kueue maintainers, but we could open issue in k8s core indicating that some changes are breaking compatibility.

We could also then know early we need to change Kueue.

There is also some cost for Kueue if new kubernetes is released and Kueue is not compatible, and we need 1 month to adapt.

[kannon92]

I'm not sure I understand what you mean here. It wouldn't be the cost on Kueue maintainers, but we could open issue in k8s core indicating that some changes are breaking compatibility.

I was thinking Kueue team needs to maintain a way to build kubernetes main without bringing that in as a go.mod dependency and continue to maintain that code.

Maybe @aojea, @pohly or @BenTheElder have established patterns for this for kubernetes sig projects?

I'd love to have kubernetes-sig dogfood features so I am supportive of the idea. Just trying to understand how much effort we want to maintain.

[tenzen-y]

I think that this is very helpful for both Kueue and kube-scheduler since 2 primary reasons.
The first is, as @mimowo mentioned, that the Kueue side could detect the scheduler-side bugs immediately, and then the scheduler side can fix that before release.

The second is that the Kueue side reports the incompatible and irreversible scheduler-side changes. For example, when scheduler-side introduced API fields incompatible with Kueue accidentally, Kueue side can find that and block releasing the incompatible ones or proposing mitigation ways to kube-scheduler side.

For the actual setup solution, I guess that we can build a KinD node image with https://kind.sigs.k8s.io/docs/user/quick-start/#building-images, but I'm not sure if this way truly works well.

[kannon92]

I'm actually part of some internal discussions currently, but the integration is certainly coming.

I know that @sohankunkerkar and I are very interested in this so I hope this integration could be community led at some point.

My goal now is to review KEPs from k/k and think about impacts to Kueue. And try and think through how workloads will leverage WAS so we can eventually think more about it for Kueue.

[sohankunkerkar]

+1 on having early visibility into WAS compatibility. For the CI approach, I think building KinD images from k/k main is the right path.

[BenTheElder]

I was thinking Kueue team needs to maintain a way to build kubernetes main without bringing that in as a go.mod dependency and continue to maintain that code.

I'm not aware of any sigs projects that import staging modules or similar from main.
(And again, if you speak to sig arch / code organization subproject ... importing k/k aside from staging is NOT supported ...)

Most SIGs subprojects are running with stable releases and relying on the fact that we normally have skew support (e.g. for client-go).
EDIT: The main exception being some tools that bring up clusters. But those are still importing stable packages/types, only using HEAD binaries.

If you just want to run Kubernetes/Kubernetes binaries from main, you can do that with various CI tools including kind, that's a very different problem that shouldn't have any relationship to your go.mod imports.

Keep in mind that you may need the absolute latest version of those tools, which themselves may have breaking changes from time to time. (e.g. kubernetes-sigs/kind#3847)

The second is that the Kueue side reports the incompatible and irreversible scheduler-side changes. For example, when scheduler-side introduced API fields incompatible with Kueue accidentally, Kueue side can find that and block releasing the incompatible ones or proposing mitigation ways to kube-scheduler side.

We usually handle this in k/k with something more like apidiff running in the main repo? There is a job checking this for client-go for example. We also have a linter for the REST API types for similar reasons + human review.

[kannon92]

@BenTheElder woukd there be any issue with having kind build node images on prow?

I know we hit Docker in docker but I guess we already support that.

[BenTheElder]

@BenTheElder woukd there be any issue with having kind build node images on prow?

Running in prow is the main reason kind exists, actually, despite it having some challenges.

I know we hit Docker in docker but I guess we already support that.

Yeah, the dind setup in test-infra is ... dubious (my doing many years ago ...), but it works OK for now and won't be unique to this repo.

We are building node images on every k/k PR.

However, I'd ask for this case that we use something like the ci/latest binaries to avoid unnecessarily re-compiling Kubernetes, as opposed to on a Kubernetes PR.

Not all existing CI jobs have been ported to that, but new efforts should.

There are examples in test-infra, but basically this will work currently and is a lot cheaper than building Kubernetes from source:

kind build node-image --type url https://dl.k8s.io/ci/$(curl -L https://dl.k8s.io/ci/latest.txt)/kubernetes-server-linux-$(go env GOARCH).tar.g

You can do some extra tricks to surface this version to testgrid, it's not super polished at the moment but doable (and not strictly necessary).

[BenTheElder]

dl.k8s.io has various "marker files" for release branches (eg head of release-N, most recently tagged in release-N, ...), there are some docs in k/community or k/release IIRC.

ci/latest will be the most recent cross-compile of master, the layout of the folders is the same as tagged releases.