Extending Kubernetes Ingress Authentication with OpenIDConnect

2 min readMay 26, 2020

This article is one I have been meaning to post since last year. In this time lots has changed, including oauth proxy given a fresh focus and maintained by Pusher and potentially moving to become a part of CNCF.

When starting out with Kubernetes and writing a few front-end and micro-services one of the easiest ways to implement any form of integration is at the application or micro-service level. However over time this becomes more and more complex as the inevitable growth and number of front-ends or micro-services increases. As this happens, fragmentation increases, with managing the versions of libraries and ever changing parameters becoming complex, tiresome and frustrating.

This is why there are many organizations turning to service meshes. And is true in the case of authentication. As such:

We wanted to abstract away from implementing in every service and let the kubernetes ingress handle the authentication dance.
Did not have the funds for plus or pro editions of an ingress controller.

Enter oauth2-proxy…

Maintained by the OpenSource community
Integrated with OpenID Connect
Was suggested as the starting position for ingress authentication in the original kubernetes ingress tutorials (albiet a different version of the open source project)
Customizable with a project structure allowing for extensions and new providers

The Problem

The enterprise IDP we were using didn’t implement the introspection / info endpoint which meant we really needed to implement a manual OIDC implementation. At the time this was coupled with the fact that the project wasn’t being maintained.

Thanks to Joel and an active community for maintaining the project, driving development and contributions.

The Solution

We went down the path of forking the project and implementing a new provider for manual OIDC.

Eventually we contributed this back to the community, however do maintain our own fork for some enterprise connectivity changes that we do not want to pollute into the mainstream. As with all open source there should be give and take.

I would recommend to anyone with the same starting out concerns to look into oauth2-proxy.

Lessons Learnt

Utilize the NGINX Ingress Controller auth-signin and auth-url annotations to integrate with oauth2-proxy

annotations:
  ingress.kubernetes.io/auth-signin: "https://$host/sign_in?rd=$request_uri"
  ingress.kubernetes.io/auth-url: "http://<service_name>.<namespace>.svc.cluster.local/auth"

Don’t forget to add the proxy_cache settings for the oauth2-proxy kubernetes ingress

ingress.kubernetes.io/configuration-snippet: |
  proxy_pass_request_body       off;
  proxy_cache                   authentication;
  proxy_cache_key               $cookie_{{ .Release.Name }};
  proxy_cache_valid             202 401 3s;
  proxy_cache_lock              on;
  proxy_ignore_headers          Set-Cookie;

Configure for large headers and proxy buffering for the oauth2-proxy kubernetes ingress

annotations:
  ingress.kubernetes.io/proxy-buffering: "on"
  ingress.kubernetes.io/client-body-buffer-size: "12M"
  ingress.kubernetes.io/client-max-body-size: "12M"
  ingress.kubernetes.io/proxy-buffer-size: "12M"
  ingress.kubernetes.io/proxy-buffers-number: "8"
  ingress.kubernetes.io/proxy-body-size: "8M"

Configure the NGINX Cache key in the NGINX Config Map HTTP-Snippet

proxy_cache_path /tmp/nginx levels=1:2 keys_zone=authentication:10m inactive=3s;

What’s Next

Contributing the helm chart we have created that not only wraps oauth2-proxy but makes switching provider types and basic auth a little bit simpler…we hope!

Thanks go to my colleague Moraru Costel for all his enduring hard work and also Joel Speed for the fantastic job being done as maintainer which can sometimes be thankless.

Extending Kubernetes Ingress Authentication with OpenIDConnect

Written by Tyson Lawrie