Blog

This is the tale of a low-severity incident that took place in our Kubernetes cluster fleet at LinkedIn that taught me a lot about how to think about off the shelf components we bring from the ecosystem into the critical path. Read More →

A quick code search query reveals at least 7,000 Kubernetes Custom Resource Definitions in the open source corpus,¹ most of which are likely generated with controller-gen —a tool that turns Go structs with comments-based markers into Kubernetes CRD manifests, which end up being custom APIs served by the Kubernetes API server.

At LinkedIn, we develop our fair share of custom Kubernetes APIs and controllers to run workloads or manage infrastructure. In doing so, we rely on the custom resource machinery and controller-gen heavily to generate our CRDs. Read More →

I’ve recently done a Twitter poll and only 20% of the participants accurately predicted that it takes Kubernetes 60-90 seconds to propagate changes to Secrets and ConfigMaps on the mounted volumes. So I want to take you on a journey in the codebase on how the mechanics of these volume types work and why it takes so long. Before going on this journey, I would answer the poll “nearly instantly” (like the majority 40% did). Read More →

Files on Kubernetes Secret and ConfigMap volumes work in peculiar and undocumented ways when it comes to watching changes to these files with the inotify(7) syscall. Your typical file watch that works outside Kubernetes might not work as you expect when you run the same progam on Kubernetes.

On a normal filesystem, you start a watch on a file on disk with a library and expect to get an event like IN_MODIFY (file modified) or IN_CLOSE_WRITE (file opened for writing closed) when the file is changed. But these filesystem events never happen for files on Kubernetes Secret/ConfigMap volumes. Read More →

It has been over two years since we announced Knative. As the project and its community is going strong, I think we made some mistakes in the early positioning and messaging of Knative prevented the project from being a go-to addon for Kubernetes that’s adopted widely. Because I have never been a decision-maker for the Knative project and its messaging at Google, I can provide an outsider’s perspective despite having worked on different aspects of Knative during this time. Read More →

In this article I will show you how to code a Go server to serve both gRPC and HTTP/2 endpoints (h2c) from a single service. This is not trivial on Cloud Run so it warrants sample code. Normally, you could use the Go cmux package in your server app and multiplex h2c and grpc requests to the right http.Handlers. However, cmux makes this decision at the connection layer, and pins a connection forever to either h2c or grpc. Read More →

When I first heard of Alex Xu’s System Design Interview – An insider’s guide, my reaction was “finally someone wrote a book with deep discussions around scalable systems design”, so I purchased it.

tl;dr: If you are new to systems design and uninitiated in distributed systems space, this book should be an instant buy for you. If you are seeking answers to how Facebook scales, how APIs like AWS S3 work or want to learn trade-offs around distributed databases and queues, this book might leave you unsatisfied as it seems to be focused on getting the beginners to succeed in the interviews. Read More →

I’m continuing with these articles on Cloud Run REST API that nobody really needs to read. This time, I’m back with a Go code walkthrough that shows how to deploy and manage services in Cloud Run through its Go API client library.

Cloud Run already offers deployment via mainstream method like CLI, web UI, IDEs and Terraform. So this article is dedicated for the <%0.1 of Cloud Run users (a.k.a. the ones keeping it real) out there who needs to use Cloud Run API with Go. Let’s begin. Read More →