In this blog, we’ll discuss how understanding Golang’s context helped us fix certain dangling sub-processes spun up by our Golang service to handle the incoming requests. Also, how a canceled incoming request generates dangling processed in a Golang program.
Now, before we start any further, let’s first briefly talk about concurrency in Golang
Concurrency in Golang
Concurrency is the ability to run functionalities independent of each other (non-interfering parallel execution).
Before Golang, additional code was required to implement concurrent execution and the same was also maintenance heavy. Now, in Golang just by adding a prefix go you would be able to implement parallel execution without writing additional lines of code or maintenance costs.
foo() // A normal function call that executes foo synchronously and
// waits for completing it
go foo() // asynchronous execution, after trigger execution for foo(),
// program will start executing next command
// instead of waiting for
// foo() to complete the execution
Now that you have read about concurrency handling in Golang, before you proceed any further, let’s also discuss two key concepts that you should be familiar with:
goroutinewaitGroupandchannel
Introduction to goroutines
In simple words, you can consider goroutine as a lightweight thread, which is running with its own resources and is independent of the other goroutine functions running within the program. Goroutines also work similarly to threads, but the cost of creating and maintaining a goroutine is much lesser than that of a thread.
Every Golang program has at least one goroutine, which is known as the main goroutine. And, all other goroutines are bound to the lifecycle of this main goroutine, which means if the main routine gets terminated at the very same time all the underlying goroutines of that program will also get terminated.
How to stop main goroutine from terminating in case of nested goroutines
Both waitGroup and channel are two different ways to make the main goroutine wait from termination, till the execution of underlying goroutines is completed.
waitGroup
func main() {
var wg sync.WaitGroup
wg.Add(1)
go func() {
fmt.Println("Hello World")
}()
wg.Wait()
}
channel
func main() {
messages := make(chan string)
go func() { messages <- "Hello World" }()
msg := <-messages
fmt.Println(msg)
}
As you’ve read about goroutines, waitGroup, and channel, now is the time to discuss whether separate goroutines will be created on every API call? Well! the answer is yes.
Since new goroutines will be created on every API call, there will also be new resource allocations to the same goroutine and, there’re chances of your goroutines eating up a lot of your program's memory.
This brings us to the next question if clients of Golang APIs have already killed the request and the request is still being processed by the Golang system, how do we communicate the same to the system?
This is where Context comes to the rescue
What is Context in Golang?
As per the official documentation here
Context carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
Incoming requests to a server should create a Context, and outgoing calls to servers should accept a Context. The chain of function calls between them must propagate the Context, optionally replacing it with a derived Context created using WithCancel, WithDeadline, WithTimeout, or WithValue. When a Context is canceled, all Contexts derived from it are also canceled.
In the aforementioned diagram, when the API request was canceled the corresponding goroutine(main goroutine of the API) was also terminated, and since DB connections were invoked without context, the main goroutine was not aware that there is one DB connection which is still open and working in bringing in some result and hence the main goroutine didn’t wait to close DB connection when cleanup activity was performed as a result of API cancellation from the client.
These dangling DB connections contribute to the sudden increase in the system’s memory utilization and are very difficult to diagnose.
How does context work?
Context is used for passing common scoped context within our application. This commonly passed context helps to transfer data across layers (of application), this way any changes that application wants to convey for all the layers serving HTTP request turns possible.
Diagrams that clearly depicts context usage.
If everything worked perfectly, we’re not dependent on context for closing the inner layers of application serving HTTP request, as inner layers are their time of execution to close and pass data(response) to upper layer(s).
Now, If the client request got canceled and context wasn’t passed to underlying layers, then upper layer (of application) won’t be able to signal inner layer(s) that request has been cancelled and we can close the execution, as client doesn’t need it going forward.
So, If the client request got canceled and same API context was passed to underlying layers, then upper layer (of application) signals inner layer(s) to terminate further execution, as the intended result is not needed by the client anymore.
Listening for cancellation
Context type in Golang provides a Done() method. This method returns an empty struct {} every time it receives a cancellation event.
we wait for the cancellation signal on ← ctx.Done()
Sample code for listening cancellation signal via context.
func foo() {
select {
case <-ctx.Done():
log.Debug("Context closed")
default:
// add bussiness logic
}
}
Emitting a cancellation event
Consider there’re two dependent operations (OP1 and OP2). This means if OP1 fails, it doesn’t make sense for you to run OP2.
Similarly, if we have a client request that gets canceled prior to your DB query completing its execution. We should stop that DB request, as the returned query response w. ll serve nobody.
This is where cancel() of type CancelFunc from the context package comes as a savior for you.
go func() {
err := operation1(ctx)
// If this operation returns an error
// cancel all operations using this context
if err != nil {
cancel()
}
}()
Recap on what we discussed!!
As you know how Golang’s context help pass common scope to internal layers initiated by the Golang program for every incoming request, let us discuss where it helped us for our use case (mentioned here) :
how understanding Golang’s context helped us fix certain dangling sub-processes spun up by our Golang service to handle the incoming requests
We’re observing multiple active DB connections as well as non-zero machine utilization even if all the incoming requests to the Golang service were canceled. After debugging, the active DB connections, we identified that the active connections are for the requests that were canceled already. Based on this understanding we identified the core reason why the request cancellation information was not passed to inner Golang layers.
