If you have ever worked with Ruby, or have maybe maintained a Rails application, I am sure the name Sidekiq will sound familiar. For those unfamiliar with the project, Sidekiq is a job system for Ruby. It is a wildly popular project, and the author has turned it into a successful business.
None of the above would be relevant if Sidekiq’s author Mike Perham, in 2014, did not write a concise and informative post titled “Don’t Daemonize your Daemons!". In it, he covers four guidelines to daemonizing programs correctly:
- Log to
- Shut down on
- Reload config on
- Provide the necessary config file for your favorite init system to control your daemon
(You can also read the whole article on his website.)
So I was thinking, why don’t we explore how to apply these guidelines while daemonizing a Go program?
The program in question is a simple command-line program that can monitor any website by sending periodic HTTP requests to it. If you ever heard of Datadog’s synthetic tests or Pingdom, think of our program as their little sibling.
observer program will read its configuration from flags, environment
variables, or a configuration file. If the configuration is not present as a
flag, it will look into the
ENV vars for it and then in a configuration file
(if present). If nothing is found, it will use the default value or exit with
an error depending on how crucial the configuration is.
To do this, we will use the
package, which is a drop-in replacement for
Go’s flag package, with the addition of parsing files and environment
variables. Being a drop-in replacement means that using the
package is as simple as using the
flag package from the standard library.
observer will have a
config type, which will encapsulate the
configuration for the website that it will observe:
init function will take the command line arguments as input and build a
FlagSet, which represents a set of defined flags. Each of the flags is listed
and parsed; then, their values are assigned to the
flag.DefaultConfigFilename as a flag as well enables our
observer to load the configuration from a
config.conf file. The
file has a
key=value format, with new lines after each key-value pair.
Here’s the main function:
Following Mat Ryer’s
we are going to keep
main very thin while keeping the main logic of the
observer in the
main here just sets up the main context that
will propagate down to the
run method, and it initializes the observer
config. Then it passes all of the relevant arguments to the
run method initializes the
c, using the
method. Then, it loops infinitely until the context
ctx is done. When
is done, it means the
observer process is terminated, so it merely returns a
nil and finishes with its execution.
Alternatively, it will execute the other case every
tick. By using the
time.Tick channel here we run this code by receiving a signal through the
c.tick period. For example, if
c.tick is 30 seconds, we will
receive a signal every 30 seconds, meaning the code will run every 30 seconds.
The code itself is simple – it sends an HTTP
GET request to the URL assigned
c.url. Once the response returns, the
run method compares the relevant
response headers and the status code with the once provided through the
configuration. If any mismatch is detected, it logs the error.
observer is relatively simple. One way is to supply a config file
through the command line:
Alternatively, using flags:
We can do the same using environment variables, or a combination of all three: config file, environment variables, and flags.
Now, how can we apply the four simple rules of daemonization?
While daemons don’t have much to do with web services, one of the 12 factors of
modern web services are treating logs as event streams. While the 12 factors in
this particular case are not applicable, the guiding principle stays: the
daemon itself should not manage log streams, nor it should not concern itself
with writing to or managing log files. Instead, daemons should send their log
stream, unbuffered, to
The service management system will capture each daemon’s stream. The
config file is what we will use to configure logging, such as where the logs
should be stored or streamed.
So, how can we adapt the
observer to log to
First, we will add another argument to the
run function, called
out of type
io.Writer. Then, we will invoke the
log.SetOutput function passing the
out as argument to it.
By doing this, we will have to pass
STDOUT from the
main function, but we
run function more testable. Using a separate
run method means we
can invoke it with any instance that implements the
io.Writer interface. We
basically couple the
run method to a behavior instead of type.
Then, we need to update the
main function to pass the additional argument to
run function when invoking it. And the
io.Writer will be simple
If we run the program again we won’t see a difference:
Why is that? Well, the
log package logs to
there is no visible change of behavior there. Still, we make the dependency on
an output stream explicit to the
run function, which clearly states that
run needs to know where to send its logs when running.
Shut down on
In Go, having errors as values is very helpful to think about what will happen to our program if an error is returned. While this makes our Go programs always have some repetitive error handling, it also gives us confidence that our program will gracefully handle any error.
*nix operating systems (OS) employ a system of signals, which is a mechanism of the OS to ask a process to perform a particular action. There are two general types of signals: those that cause termination of a process and those that do not.
(Refer to the full list of the POSIX-defined signals to learn more.)
Using these system signals, a process that has received one can choose one of the following behaviors to take place: perform the default POSIX-defined action, ignore the signal, or catch the signal with a signal handler and perform some sort of a custom action.
Some signals that just can’t be caught or ignored; it means that the default
action has to happen. For example,
SIGKILL are such signals.
Once a process receives any of these two signals, we just know that it will be
stopped/killed by the OS.
But other signals are more polite. While we cannot ignore them, they give a
chance to our process to clean up and go away with grace. Most of the ones on
are of the polite kind. In this section, we will look into the
SIGINT signals and how we can treat them in our Go programs.
observer’s case, we don’t have to do any cleanup once it receives a
SIGINT. All we have to do is to stop further execution and shut
down gracefully. So, how can we achieve that?
First, we need to create a channel through which we will accept these two signals:
observer process receives a
SIGINT or a
SIGTERM, it will proxy
it through the
signalChan channel. To process the signals, we would need to
create a goroutine that will receive signals through the
signalChan. Once it
gets a signal, it will have to
cancel() the context, which would stop the
further execution of the
So, once the
cancel function is executed, in the
for loop of the
method the execution will stop:
The last thing we need to do in the
main function is to close the
signalChan channel when the programs exits:
Stop function will stop relaying incoming signals to
Stop returns, it is guaranteed that
signalChan will receive no more
Let’s run the
observer program and see the signal handling in action:
Now, having the PID of
observer, we can send any signal using the
command line tool:
By executing the
kill command, we will send a
SIGINT to the
process. This will force
observer to wrap up the execution, log a line to
STDOUT and exit:
We can try the same exercise with
SIGTERM as well:
observer to exit with the same behavior:
Reload config on
Now that we know how to handle signals, we need to add another signal to the
SIGHUP. To do that, we can just add
syscall.SIGHUP to the
Now that we have
SIGHUP covered, in the goroutine that handles the signals,
SIGHUP is received, it should re-run the
config.init method. By
doing that, we will reload the configuration of the
observer, loading any
changes in the configuration:
The change is relatively small. By using a
switch construct, detect the
received signal. If it’s a
SIGHUP, we invoke
cancel() the context and
os.Exit the program.
We can test this using the same trick from before:
Will cause the
observer to reload:
This looks nice. Let’s shut down the server now by sending a
In case you’re following along, you will find out that the
observer is still
running; this is a bug – the goroutine that was receiving signals exited
select construct completed once it received the
To make the goroutine accept signals without exiting, we need to make the
goroutine run infinitely – using a
By wrapping the whole goroutine in a
for loop, we will make sure that it will
not exit, except when a
SIGTERM is received, or if the context is
done. By having this endless goroutine, we also can send multiple
observer, and it will process them correctly.
Let’s send two
SIGHUPs, to perform two reloads, and
SIGTERM to shut down
And that’s it. The
observer now knows how to log to
STDOUT, gracefully exit
when it receives
SIGTERM and reloads the configuration when it
Provide the necessary config file for your favorite init system to control your daemon
Now, given that computer of choice is a MacBook, I will explain here how you
can create a config file for
launchd – macOS’s
service management framework for starting, stopping and managing daemons,
applications, processes, and scripts. In macOS, the system runs daemons, while
the users run programs as agents. So, we will turn our
observer into an
In the past, I have written about creating and managing macOS
agents, so if you would like to
more about this topic, you can head read that as well. Still, let’s see a
launchd configuration for
The configuration is relatively straightforward, here are all of the pieces in order:
Labelidentifies the job and has to be unique for the launchd instance. Think of it as a unique name for the given agent.
launchdwill start the job as soon as it loads it.
KeepAlivetells launchd to keep the agent running no matter what.
ProgramArgumentsprovides command-line options to the agent command. In our case, this will create the following command:
/usr/local/bin/observer -config /etc/observer.conf.
StandardErrorPathare the paths to where
launchdwill write the respective output. In our case, we write these to the
tmpdirectory. An alternative would be to add the log files to
/var/log, but that requires granting write access of the agent to
To make sure we can run the agent, we have to also supply the configuration
observer.conf in the
/etc directory. On my machine, its contents are
After placing the
observer.conf file in
/etc, for the agent to work, we
have to place its
.plist file in
~/Library/LaunchAgents, and load it with:
Now, if we would
tail -f the log files in
/tmp we will see its outputs
Voila! The agent is running and its logging output to
is redirecting that output to a log file.
If we would like to run the
observer on GNU/Linux, we cannot use this
systemd is widespread and popular. If you are interested in a
deeper explanation of
systemd units and unit files, Digital Ocean’s blog has
on “Understanding Systemd Units and Unit Files” by Justin Ellingwood. I
recommend reading it! And keep in mind, the community’s opinion on
There are a bunch of other alternatives, but my knowledge of GNU/Linux init systems is minimal. Therefore, I will stop right here and ask for your help: if you would like to contribute a Linux init system configuration to this article, drop the link to a gist/repo in the comments, and I will include it in this article.
Of course, with proper attribution.
You can see the final implementation of the observer here.