is this improvement open for contribution?
Yes.
Would it make sense to tackle both together, starting with 318 then plugging the JWKSet caching into it?
That's what I suggested in OIDC-318, yes.