---

In the first part I showed how to run a normal console application in the background, now is the time to build a real Linux daemon with Delphi!

First of all I want to thank my friend and colleague Luca Minuti that first built the daemon .dpr skeleton and showed it at the Delphi Day Italian conference in June.

What is a Linux (UNIX) daemon

There are 3 basic types of processes in Linux:

• Interactive run interactively by a user at the command line
• Batch submitted from a queue of processes and not associated with the command line
• Daemon background process that is designed to run autonomously, with little or no user intervention

Daemons are recognized by the system as any processes whose parent process has a pid = 1 (the process with 1 as a pid, always represents the process init).

init is always the first process that is started when a Linux computer is booted up and it remains on the system until the computer is turned off. init adopts any process whose parent process dies without waiting for the child process's status.

So, the common method for creating a daemon involves forking (twice), and making the parent (and grandparent) processes die, while the grandchild process begins performing its normal function.

Running in background means that a daemon must be detached from the terminal and runs continuously in a non-interactive mode.

A daemon cannot read/write from/to the stdin, stdout and stderr and to be sure that it doesn't crash readind or writing on them they must be redirected to /dev/null

So, basically the operations needed to create a daemon are:

1. fork the parent process and let it terminate

• setsid to create a new session detaching from its controlling terminal
• signals handling to catch/ignore signals
• fork again and let it terminate to ensure that you get rid of the session leading process
• close all open file descriptors inherited from the parent process
• redirect stdin, stdout, stderr to dev/null to ensure that the daemon doesn't crash
• chdir to change the working directory of the daemon
• umask to change the file mode mask according to the needs of the daemon

The Delphi code

I know that you are impatient to see the code so... here it is!

program delphid;

{$APPTYPE CONSOLE}

uses
  System.SysUtils, System.IOUtils,
  Posix.Stdlib, Posix.SysStat, Posix.SysTypes, Posix.Unistd, Posix.Signal, Posix.Fcntl,
  Posix.Syslog in 'Posix.Syslog.pas';

const
  // Missing from linux/StdlibTypes.inc !!! <stdlib.h>
  EXIT_FAILURE = 1;
  EXIT_SUCCESS = 0;

var
  pid: pid_t;
  fid: Integer;
  idx: Integer;
  running: Boolean;

procedure HandleSignals(SigNum: Integer); cdecl;
begin
  case SigNum of
    SIGTERM:
    begin
      running := False;
    end;
    SIGHUP:
    begin
      syslog(LOG_NOTICE, 'daemon: reloading config');
      // Reload configuration
    end;
  end;
end;

begin
  openlog(nil, LOG_PID or LOG_NDELAY, LOG_DAEMON);

  if getppid() > 1 then
  begin
    pid := fork();
    if pid < 0 then
      raise Exception.Create('Error forking the process');
  
    if pid > 0 then
      Halt(EXIT_SUCCESS);
  
    if setsid() < 0 then
      raise Exception.Create('Impossible to create an independent session');
  
    signal(SIGCHLD, TSignalHandler(SIG_IGN));
    signal(SIGHUP, HandleSignals);
    signal(SIGTERM, HandleSignals);
  
    pid := fork();
    if pid < 0 then
      raise Exception.Create('Error forking the process');
  
    if pid > 0 then
      Halt(EXIT_SUCCESS);
  
    for idx := sysconf(_SC_OPEN_MAX) downto 0 do
      __close(idx);
  
    fid := __open('/dev/null', O_RDWR);
    dup(fid);
    dup(fid);
  
    umask(027);
  
    chdir('/');
  end;  

  running := True;
  try
    while running do
    begin
      // deamon actual code
      Sleep(1000);
    end;

    ExitCode := EXIT_SUCCESS;
  except
    on E: Exception do
    begin
      syslog(LOG_ERR, 'Error: ' + E.Message);
      ExitCode := EXIT_FAILURE;
    end;
  end;

  syslog(LOG_NOTICE, 'daemon stopped');
  closelog();
end.

This is the (minimal) Delphi code to build a real Linux daemon, quite simple right? But despite the simplicity of this code I invite you to read the article because I think that to build a robust daemon application you have to understand every step of the process, so please make yourself comfortable because this is a lengthy article! 😀

Program, unit uses and consts

program delphid;

{$APPTYPE CONSOLE}

uses
  System.SysUtils, System.IOUtils,
  Posix.Stdlib, Posix.SysStat, Posix.SysTypes,
  Posix.Unistd, Posix.Signal, Posix.Fcntl,
  Posix.Syslog in 'Posix.Syslog.pas';

const
  // Missing from linux/StdlibTypes.inc !!! <stdlib.h>
  EXIT_FAILURE = 1;
  EXIT_SUCCESS = 0;

As you can see this is a normal console application File -> New -> Other -> Console Application with a bunch of Posix.* uses. These are the POSIX API units (like the Winapi.* counterpart) and you can expect to find const definitions, (external) function declarations, etc... Here I've added the two const EXIT_FAILURE and EXIT_SUCCESS strangely missing from the StdlibTypes.inc file.

The syslog API

The last unit is the Posix.Syslog.pas that contains calls to the syslogd (or compatible) Linux daemon. The syslog API is missing in 10.2 Tokyo (plain or update 1) so remember that this file is required to log to the syslog daemon but it's not required to build a Linux daemon (although IMO a Linux daemon must log via syslogd).

UNIX-like operating systems have an API for programs to send log messages to the system, if an application is considered critical and its state is the system administrator's responsibility then this application must log via syslogd.

Despite the power and flexibility of syslogd, the syslog API is very simple:

• void openlog(char *ident, int option, int facility)
• void syslog (int priority, char *format, ...)
• void closelog(void )

In the code, as you can see, I've called openlog with no custom ident value (the process name will be used), the option value is LOG_PID or LOG_NDELAY to tell the syslogd to log also the pid and to open the connection immediately and, finally, the facility is set to LOG_DAEMON.

openlog(nil, LOG_PID or LOG_NDELAY, LOG_DAEMON);

If you don't call openlog() explicitly, syslog() will call it for you with no arguments.

The call of syslog() is very simple, you have to pass the priority parameter:

• LOG_EMERG: The system is unusable
• LOG_ALERT: Action must be taken immediately
• LOG_CRIT: Critical condition
• LOG_ERR: Generic error condition
• LOG_WARNING: Warning message
• LOG_NOTICE: Normal but important event
• LOG_INFO: Purely informational message
• LOG_DEBUG: Debugging-level message

syslog(LOG_NOTICE, 'daemon: reloading config');

The call of closelog() doesn't need to be explained:

closelog();

The standard location of syslog log is /var/log/syslog:

The syslogd has other features and some are very interesting from a developer's point of view. I recommend to read some documentation about syslogd and its configuration, in addition I'm thinking to write a small article about the logging features found in Linux distros.

The routine HandleSignals

// 1. If SIGTERM is received, shut down the daemon and exit cleanly.
// 2. If SIGHUP is received, reload the configuration files, if applies.
procedure HandleSignals(SigNum: Integer); cdecl;
begin
  case SigNum of
    SIGTERM:
    begin
      running := False;
    end;
    SIGHUP:
    begin
      syslog(LOG_NOTICE, 'daemon: reloading config');
      // Reload configuration
    end;
  end;
end;

I'll explain the meaning of the procedure HandleSignals later, for now please notice that this routine must be marked with the cdecl calling convention. In short, this is a callback from the system when a signal is received.

The fork() system call

The fork() system call is used to create processes and returns a Process ID (pid). The fork() call creates a new process, which becomes the child process of the caller and makes (well, the OS does) an identical copy of the address space for the child, so immediately after a fork() call there are two identical copies of address spaces. Remember that from there the two process are running separately (and concurrently).

A very important thing to know is that after the child is created, both processes will execute the next instruction following the fork() call. To tell the difference between the parent from the child we must test the fork()'s pid returned value:

• pid = 0: its the child
• pid > 0: its the parent
• pid < 0: its an error

so, for example, in the following code:

  // Call fork(), to create a child process.
  pid := fork();

  if pid = 0 then
  begin
    // Only the child execute this section
    DoSomething;
  end
  else if pid > 0 then
  begin
    // Only the parent execute this section
    DoSomethingOther;
  end
  else
    raise Exception.Create('Error forking the process');

different sections are executed by the child or the parent based on the returned pid.
Ok, now that we understand the fork() call, let's see how we can use it to create our daemon.

The first fork()

begin
  // syslog() will call openlog() with no args if the log is not already opened
  openlog(nil, LOG_PID or LOG_NDELAY, LOG_DAEMON);

  syslog(LOG_NOTICE, 'before 1st fork() - original process');

  // Call fork(), to create a background process.
  pid := fork();

  syslog(LOG_NOTICE, 'after 1st fork() - the child is born');

  if pid < 0 then
    raise Exception.Create('Error forking the process');

  // Call exit() in the first child, so that only the second
  // child (the actual daemon process) stays around
  if pid > 0 then
    Halt(EXIT_SUCCESS);

  syslog(LOG_NOTICE, 'the parent is killed!');

The first step to create a Linux daemon is the call of the fork() function. The return value is a Process ID (pid), that you can check to see if the fork has been successful. The other important piece is the call of the Halt(EXIT_SUCCESS) function that has the effect of killing the actual process (in this case, the parent) leaving only the child, spawned by the fork call, alive and well (we hope) 😊

If you remember, the child starts executing the instruction right after the fork() so when we read the syslog logfile we'll see 2 lines almost identical, I said almost because the process ID will be different!

As you can see there are two lines: after 1st fork() - the child is born but the pid in one case is 1654 and in the other 1655.

Creating a Unique Session ID (SID)

  // This call will place the server in a new process group and session and
  // detaches its controlling terminal
  if setsid() < 0 then
    raise Exception.Create('Impossible to create an independent session');

  syslog(LOG_NOTICE, 'session created and process group ID set');

Processes in Linux have several IDs, the most important is the Process ID (pid) that identifies the process in the system.
The Session ID exists because processes are organized into sets of sessions and so the child process must get a unique sid from the kernel otherwise, the child process becomes an orphan for the system.

As you can see in the image, the delphid daemon process has the pid of 932 so I can issue this ps command:
ps -Ao pid,ppid,pgid,sid,command | grep -E "COMMAND|932"
to show more informations (IDs) about the delphid daemon:

• PID: Process Identifier
• PPID: Parent Process Identifier
• PGID: Process Group Identifier
• SID: Session Identifier (Session Leader)

Linux signals

Signals in Linux are software interrupts and they are used everywhere, so to be integrated with the system, your application needs to handle signals!

  // Catch, ignore and handle signals
  signal(SIGCHLD, TSignalHandler(SIG_IGN));
  signal(SIGHUP, HandleSignals);
  signal(SIGTERM, HandleSignals);

In Linux, every signal has a name (that begins with SIG), the most important (to me) are:

• SIGKILL (signal kill) is sent to a process to request its termination immediately
• SIGINT (signal interrupt) is sent to a process by its controlling terminal when a user wants to interrupt the process. Typically initiated with Ctrl+C
• SIGTERM (signal termination) is sent to a process to request its termination. Unlike the SIGKILL signal, it can be caught and interpreted or ignored by the process
• SIGHUP (signal hang up) is sent to a process when its controlling terminal is closed (originally designed to notify the process of a serial line drop)

Calling the function signal() I choose to handle or ignore the specific signal, in the example you can see that the callback function HandleSignals will be called on the SIGHUP and SIGTERM signals.

The second fork()

  // Call fork() again, to be sure daemon can never re-acquire the terminal
  pid := fork();

  syslog(LOG_NOTICE, 'after 2nd fork() - the grandchild is born');

  if pid < 0 then
    raise Exception.Create('Error forking the process');

  // Call exit() in the first child, so that only the second child
  // (the actual daemon process) stays around. This ensures that the daemon
  // process is re-parented to init/PID 1, as all daemons should be.
  if pid > 0 then
    Halt(EXIT_SUCCESS);
  syslog(LOG_NOTICE, 'the 1st child is killed!');

The second fork() it's only there for a technical reason: ensures that the new process is not a session leader, so it won't be able to regain control of a terminal, since daemons are not supposed to have a controlling terminal. As you can see the code is almost identical to the previous fork().

Close stdin, stdout, stderr file descriptors

  // Close all opened file descriptors (stdin, stdout and stderr)
  for idx := sysconf(_SC_OPEN_MAX) downto 0 do
    __close(idx);

  syslog(LOG_NOTICE, 'file descriptors closed');

File descriptors are inherited to child process, this may cause a waste of resources, so file descriptors should be closed before the fork() call or as soon as the child process starts running.

In a UNIX program there are three special descriptors: the standard input stdin, standard output stdout and standard error stderr and they are tipically used by shell programs, but because a daemon separates itself from the shell, it needs to get rid of everything that's keeping it connected to the terminal.

Redirect stdin, stdout, stderr do /dev/null

  // Route I/O connections to > dev/null

  // Open STDIN
  fid := __open('/dev/null', O_RDWR);
  // Dup STDOUT
  dup(fid);
  // Dup STDERR
  dup(fid);

  syslog(LOG_NOTICE, 'stdin, stdout, stderr redirected to /dev/null');

Well, I said that you have to close stdin, stdout, stderr and now I re-open them!??

The reason is because some library that you are using may try to read or write to/from standard I/O but if you attempt to read/write from a closed file descriptor, the operation will fail. The POSIX specification requires that /dev/null has be provided so the daemon can reasonably depend on this device.

As POSIX assigns descriptors sequentially, __open() call will open stdin and dup() calls will provide a copy for stdout and stderr.

Set file mask and working directory

  // Restrict file creation mode to 640 (750)
  umask(027);
  syslog(LOG_NOTICE, 'file permission changed to 640 (750)');

The umask() (user mask) is a command and a function in POSIX environments that sets the file mode creation mask of the current process which limits the permission modes for files and directories created by the process. In practice with umask() you can define permissions of the new files that your process will create. The user mask contains the octal values of the permissions you want to set for all the new files.

So in order to avoid insecure permissions you want to set the umask() to 027 or to 022.

Set the working directory

  // The current working directory should be changed to the root directory (/), in
  // order to avoid that the daemon involuntarily blocks mount points from being unmounted
  chdir('/');
  syslog(LOG_NOTICE, 'changed directory to "/"');

In a Linux daemon you have to change the current working directory to a safe location, for example the root (/), obviously can be changed later if required.

Changing the working directory allows the system administrator to mount and unmount directories without your daemon getting in the way.

You can also change the working directory to the directory containing all your daemons config/data files, the choice is yours.

Main daemon loop

  syslog(LOG_NOTICE, 'daemon started');
  // deamon main loop
  running := True;
  try
    while running do
    begin
      // deamon actual code
      Sleep(1000);
    end;

    ExitCode := EXIT_SUCCESS;
  except
    on E: Exception do
    begin
      syslog(LOG_ERR, 'Error: ' + E.Message);
      ExitCode := EXIT_FAILURE;
    end;
  end;

  syslog(LOG_NOTICE, 'daemon stopped');
  closelog();

Nothing special with the main loop, this is the point where you have to write your actual code.

A real example

This is the log of a real Linux service running (in deployment) in an AWS instance, this service act as a GitHub Webhooks and Repository Hosting Hooks receiver that trigger some actions based on the activity in the code repository.
For example you can use this in a build machine to build and test your code whenever there is a commit on the repository.

You can find the source code of this example in the GitHub repository for this article.

Source code repository

The code is published at my GitHub profile in this repository. In this repo there are 3 projects:

DelphiDaemonBase

This project contains the source code showed and discussed in this article, it's a simple but fully functional Linux daemon implementation in a .dpr project with the syslog.pas unit.

DelphiDaemonWiRL

This second project is built with two (independent) units from the WiRL REST library:

• WiRL.Console.Posix.Daemon.pas
• WiRL.Console.Posix.Syslog.pas

that contains the (same) code to build a Linux daemon but encapsulated in easy-to-use Delphi classes.

GitHubHooksDaemon

The third project is a fully featured REST Linux daemon built with the WiRL REST library and it shows how to encapsulate further the code logic to build a console application that behave as standard console app in debug and a daemon in release.

The demo reacts to valid GitHub WebHooks payloads when a repository is pushed, forked, watched, etc... The interesting thing is that it "restify" GitHub Webhooks calls by converting the X-GitHub-Event extra http header in a REST path in order to manage the calls as standard REST calls.

Conclusions

In this article we learned how to build a proper application that follows the POSIX rules for daemons, we learned how to log in Linux using syslog, a very powerful tool available to Linux developers all built with our beloved development tool: Delphi 😀

If you have a suggestion please contact me or drop a line in the comments below.