Run ps on a busy machine and the neat idea of “a program that is running” immediately gets messy. Some entries are sleeping, some have several threads, some are waiting to be reaped, and some belong to services with no terminal at all. Process management is how Linux keeps that crowd identifiable, scheduled, limited, and controllable.

What is a Process in Linux?

A process is a running instance of a program, with an address space, credentials, open files, signal state, and one or more threads. A process ID (PID) identifies it while it exists, but PIDs are reused after processes exit. Do not treat a PID saved long ago as a permanent identity.

Job-Control and Service Roles

Foreground, background, and daemon describe how a process relates to a terminal or service manager. They are not kernel process states.

Foreground processes

In shell job control, the foreground process group is the one currently allowed to read from and receive terminal-generated signals from that terminal. A graphical application can be visually in front without being a shell foreground job. For example:

  • Running bash myscript.sh normally makes the script part of the foreground job.
  • Pressing Ctrl+C asks the terminal driver to send SIGINT to the foreground process group.

Background processes

Background jobs do not occupy the shell’s foreground job slot. Appending & starts one:

Terminal window
ping google.com > output.txt &

These jobs can be managed with jobs, fg, and bg, but they are still associated with the shell session. They may receive SIGHUP when the terminal closes and may stop if they try to read from the terminal. Use a service manager, nohup, tmux, or screen when work must survive logout.

Daemon processes

A daemon is a long-running service that does not depend on an interactive terminal. On many current distributions, systemd starts and supervises services such as sshd; scheduled work may use cron or systemd timers.

The Lifecycle of a Process

Creation: fork, exec, and spawn

The familiar Unix pattern is fork() followed by an exec function, but the two calls do different jobs. fork() creates a child with a new PID and a logically separate address space. Linux implements the memory side with copy-on-write pages, so it does not eagerly duplicate every byte. The child inherits file descriptors that refer to the same underlying open-file descriptions, including their file offsets.

execve() then replaces the calling process’s program image. It does not create another process: the PID remains the same while the executable, stack, heap, and mappings are replaced. The parent can wait for the child or continue concurrently.

There is an important multithreading edge case. After fork(), only the calling thread exists in the child, although copied mutex state may still reflect threads that vanished. Until execve(), the child should call only async-signal-safe functions. Libraries may instead use posix_spawn(), vfork(), clone(), or clone3() when their semantics fit the job.

Execution and scheduling

Once created, a process is scheduled for execution. The kernel determines when and for how long a process can use the CPU based on:

  • Priority: Scheduling class, real-time priority, nice weight, CPU affinity, and cgroup settings all influence access to CPU time. “Higher priority runs first” is only a rough summary.
  • Policy: Scheduling policies like SCHED_NORMAL and SCHED_RR define execution behavior.

For ordinary tasks, Linux historically used the Completely Fair Scheduler (CFS). Mainline Linux began moving its fair scheduling class to Earliest Eligible Virtual Deadline First (EEVDF) in version 6.6. Both approaches use weighted fairness; the exact implementation depends on the kernel version. Scheduling policies include:

  • SCHED_OTHER / SCHED_NORMAL: Two names for the normal fair-scheduling policy.
  • SCHED_FIFO: Fixed-priority real-time scheduling. A runnable thread continues until it blocks, exits, yields, or is preempted by a higher-priority thread.
  • SCHED_RR: The same fixed-priority model with a time quantum among runnable threads at the same priority.
  • SCHED_BATCH: Uses the fair scheduler but assumes the task is not interactive, reducing unnecessary preemption at the cost of responsiveness.
  • SCHED_IDLE: Gives a task a weight below ordinary nice levels. It is not a promise that the task will run only when the CPU is otherwise idle.

See the kernel’s EEVDF scheduler documentation for the current design.

Exit and signals

At a high level, a process may finish through its own program logic or be terminated by a signal:

  • Ordinary completion: The program returns from its entry point or calls an exit function, which records an exit status and performs the applicable cleanup.
  • Signal termination: A signal may request a graceful shutdown, trigger the program’s handler, stop it immediately, or report a fault. SIGTERM can be caught or ignored; SIGKILL cannot.

Signals are used for inter-process communication (IPC) and process control. Common signals include:

  • SIGKILL: Forcefully terminates a process. It cannot be caught or ignored.
  • SIGTERM: Requests termination and allows the process to clean up. It can be caught or ignored.
  • SIGSTOP: Stops a process. It cannot be caught or ignored.
  • SIGCONT: Continues a stopped process.
  • SIGHUP: Reports a terminal hangup and is also commonly interpreted by daemons as a request to reload configuration.
  • SIGINT: Interrupt signal, typically sent by pressing Ctrl+C.
  • SIGSEGV: Reports an invalid memory access.
  • SIGCHLD: Notifies a parent about a child’s state change.

Signal numbers vary by architecture, so scripts should use names such as TERM instead of hard-coded numbers.

Process States

A process state is a snapshot of what a task is doing when the kernel reports it:

  • R — running or runnable: executing on a CPU or waiting in a run queue;
  • S — interruptible sleep: waiting for an event and able to wake for a signal;
  • D — uninterruptible sleep: usually waiting in a kernel I/O path that does not currently permit ordinary signal handling;
  • T/t — stopped or traced: paused by job control, a signal, or a debugger;
  • Z — zombie: exited, with status still waiting to be collected by its parent;
  • X — dead: a transient removal state, not something normally observed long enough to manage.

An orphaned child is a relationship, not a state. It is reparented to the nearest configured child subreaper or, if none exists, the system’s PID 1 process.

The state letter is a clue, not a diagnosis. A server with many S tasks may be healthy. A task stuck in D may be waiting on a failed disk or network filesystem; sending SIGKILL does not remove a task until the kernel wait completes. A zombie consumes almost no runtime memory or CPU, but a growing population shows that a parent is failing to reap children.

Terminal window
ps -eo pid,ppid,user,state,wchan:24,etimes,cmd --sort=state

wchan names the kernel wait channel when available. Permissions, kernel configuration, and symbol visibility can limit what it shows.

/proc Is the Evidence Behind Many Tools

Utilities such as ps and top obtain much of their information from procfs. Looking at /proc directly is useful when a formatted view hides the field needed for an incident:

Terminal window
pid=1234


readlink -f "/proc/$pid/exe"
tr '\0' ' ' < "/proc/$pid/cmdline"
sed -n '1,25p' "/proc/$pid/status"
cat "/proc/$pid/cgroup"
ls -l "/proc/$pid/fd" | head

Check ownership and the executable path before acting. A PID may have been reused between the alert and the investigation. For automation that must hold a stable process reference, Linux pidfds avoid signalling a different process after PID reuse; shell scripts should at least revalidate identity immediately before a destructive action.

Do not parse /proc/<pid>/stat by splitting blindly on spaces. The command name is enclosed in parentheses and may itself contain spaces or parentheses. Prefer a library, ps, or a parser that follows the procfs format.

Tools for Process Management

ps: a snapshot

ps provides a static snapshot of running processes:

Terminal window
ps aux

This combines BSD-style selection and output options: it normally shows processes for all users, includes processes without a controlling terminal, and uses a user-oriented display. The individual letters do not behave exactly like three independent switches in every ps personality, so use explicit fields in scripts:

Terminal window
ps -eo pid,ppid,user,state,etimes,%cpu,%mem,cmd --sort=-%mem

top and htop: a changing view

top continuously refreshes its process view:

Terminal window
top

In top, P sorts by CPU, M by memory, N by PID, H toggles threads, and k sends a signal. htop offers a more navigable view, but neither tool explains causality by itself. A high CPU percentage tells you where to look, not why the work exists.

kill, pgrep, and pkill

kill sends a signal to a PID. Two signals commonly used during incident response are:

  • SIGTERM (15): Requests a graceful termination.
  • SIGKILL (9): Forcefully terminates a process.

Examples:

Terminal window
kill 1234      # Sends SIGTERM
kill -9 1234   # Sends SIGKILL

kill -SIGSTOP <PID> stops a process; kill -SIGCONT <PID> allows it to continue.

pgrep selects processes by name or attributes; pkill sends a signal to the same kind of selection:

Terminal window
pgrep -a firefox
pkill firefox

pkill may match more than one process, and -f matches the complete command line rather than the short process name. Preview the same criteria with pgrep -a or pgrep -af before sending a signal. Prefer TERM before KILL unless the process cannot be recovered another way.

nice and renice

nice starts a command with an adjusted niceness; without -n, it adds 10 to the inherited value. renice changes an existing process. Niceness values normally range from -20 (more CPU weight) to 19 (less CPU weight), with 0 as the usual login default.

Example:

Terminal window
nice -n 10 long_task.sh
renice -n -5 -p 1234

Lowering a niceness value normally requires the appropriate privilege. Niceness changes the weight of ordinary fair-scheduled tasks; it is not a hard CPU reservation.

Shell job control

jobs lists the shell’s jobs, fg %1 brings job 1 to the foreground, and bg %1 resumes it in the background. These are shell job IDs, not process IDs.

Advanced Process Management Techniques

Trace system calls with strace

strace records system calls and their results. A short trace can expose repeated failures or a blocking call:

Terminal window
strace -p 1234

Attaching changes timing and may reveal sensitive arguments. Bound the capture and secure its output on a production system.

Limit resources with rlimits and cgroups

Shell ulimit configures inherited per-process resource limits such as open files or address-space size. Cgroups account for and control resources for a group of processes, which is usually the right boundary for a service.

Cgroup v2

Control groups organize processes hierarchically and let controllers account for or distribute resources such as CPU, memory, and I/O. Most current distributions use cgroup v2. Its interface includes files such as cpu.weight, cpu.max, memory.current, and memory.max; older examples using cpu.shares or memory.usage_in_bytes describe cgroup v1.

When systemd manages the hierarchy, it is safer to use systemd properties than to create directories under /sys/fs/cgroup by hand:

Terminal window
systemd-run --user --scope -p CPUWeight=50 -p MemoryMax=1G ./long_task.sh

A cgroup controls resources; it does not hide processes, users, mounts, or networks. That kind of isolation comes from Linux namespaces and other security controls. Container managers commonly combine namespaces with cgroups.

The kernel’s cgroup v2 documentation is the reference for controller semantics.

Change scheduling policy carefully

chrt can inspect or change real-time scheduling attributes:

Terminal window
chrt -r -p 10 1234

Real-time policies can starve ordinary work when misconfigured. They need explicit latency requirements, bounded CPU use, and the privileges and resource limits appropriate to the service.

Work through the service manager

On a systemd machine, inspect the unit rather than treating its main PID as an isolated program:

Terminal window
systemctl status sshd
systemctl show sshd -p MainPID -p ControlGroup -p MemoryCurrent
journalctl -u sshd --since '-15 min'

systemctl start, stop, restart, and reload have different operational consequences. enable controls whether a unit is wired into boot targets; it does not necessarily start the unit immediately. Unit dependencies and ordering are also separate ideas: Requires= expresses a requirement, while After= controls ordering.

Incident Walkthrough: A Service Is Saturating a Host

Assume an alert says checkout.service is consuming CPU and requests are timing out. Restarting it immediately may restore service, but it also destroys evidence. Start by establishing scope:

Terminal window
systemctl status checkout.service
systemctl show checkout.service \
  -p MainPID -p ControlGroup -p ActiveState -p SubState \
  -p MemoryCurrent -p CPUUsageNSec
journalctl -u checkout.service --since '-15 min' --no-pager

Use the unit’s main PID rather than copying the first process name that looks familiar:

Terminal window
pid=$(systemctl show --property MainPID --value checkout.service)
ps -p "$pid" -o pid,ppid,user,state,etimes,%cpu,%mem,nlwp,cmd
top -H -p "$pid"

top -H separates threads. One hot thread suggests a different investigation from hundreds of runnable workers. Next, decide whether the service is CPU-bound, blocked on I/O, or allocating memory:

Terminal window
cat "/proc/$pid/io"
sed -n '/VmRSS/p;/VmSwap/p;/Threads/p' "/proc/$pid/status"
printf 'open_fds='; ls -1U "/proc/$pid/fd" | wc -l

If system-call behavior matters, a short trace can reveal repeated failures, lock waits, or unexpected network activity:

Terminal window
timeout 10s strace -ff -tt -T -p "$pid" -o /tmp/checkout.strace

Tracing adds overhead and can expose sensitive arguments or data. Use a bounded duration, secure the output, and avoid attaching casually to a latency-critical process.

The unit’s cgroup explains whether the problem belongs to one process or the whole service tree:

Terminal window
cg=$(systemctl show --property ControlGroup --value checkout.service)
cat "/sys/fs/cgroup$cg/cpu.stat"
cat "/sys/fs/cgroup$cg/memory.current"
cat "/sys/fs/cgroup$cg/memory.events"

memory.events can distinguish approaching a configured limit from an ordinary high resident set. cpu.stat can show throttling when a quota is active. A process-level view alone misses both.

Only after collecting enough evidence should the response move to control. Prefer the service manager so the signal reaches the intended unit and its state remains coherent:

Terminal window
systemctl kill --signal=TERM checkout.service
systemctl stop checkout.service

Escalate to KILL only when graceful termination has failed and losing in-process state is acceptable. If a task is stuck in D, escalation may not help; investigate the underlying I/O path. If the process is a zombie, signal its parent or fix the parent’s wait logic rather than repeatedly trying to kill an already exited child.

This sequence—identify, observe, classify, then control—is the core of process management. The commands are secondary to preserving that order.

Best Practices for Linux Process Management

  1. Observe before acting: Use ps, top, service status, metrics, and logs to establish what the process is doing.
  2. Prefer the service manager: Let systemd supervise long-running services, restart policies, dependencies, credentials, and resource limits.
  3. Send the least forceful signal first: Give a process a chance to shut down cleanly before considering SIGKILL.
  4. Use resource controls deliberately: A nice value affects CPU scheduling, while cgroups can govern CPU, memory, and I/O. Neither substitutes for finding a leak or a bad query.
  5. Automate repeatable work: Use cron or systemd timers for scheduled jobs, with useful logs, timeouts, and failure notifications.
  6. Check identity and scope: PIDs are reused. Confirm the command line, owner, unit, container, and host before signalling or tracing a process.

Conclusion

Process management is mostly a cycle of identification, observation, and controlled action. Find the correct process, understand its state and owner, inspect its service and logs, then choose the narrowest intervention that solves the problem. The dangerous mistakes usually happen when that order is reversed.

References