Blog | zealcharm

A method for maintaining locally patched Arch Linux packages

2025-03-30T00:00:00+00:00

Due to my ~~stubbornness~~ fondness for testing the latest and greatest software releases, I often run into problems, and then I need to apply patches on top of Arch Linux's packages to fix them.

When the number of patched packages grows large enough, this may devolve into chaos, so this is a rough sketch of the method I have used over the years to keep this under control.

Identifying patched packages

The first challenge is marking the packages that I have patched, so both pacman and I can understand what is installed on my system.

I do this by simply appending a snippet like this at the end of each of my patched PKGBUILDs:

custompkgrel=1
pkgrel="${custompkgrel}00${pkgrel}"

This will turn a package version like helloworld-1.23-4 into helloworld-1.23-1004.

This both makes it identifiable (usually, packages have a small pkgrel), and makes pacman detect the patched packages as updates of the unpatched ones.

When I patch the package further, I bump custompkgrel, so that each patched package has a unique version.

Applying .patch files

The straightforward way to apply .patch files is to put them inline with the rest of the PKGBUILD, like this:

 # Maintainer: John Smith <john.smith@example.com>
 pkgname=helloworld
 pkgver=1.23
 pkgrel=4
 pkgdesc="A simple Hello World program"
 url='https://www.example.com'
 arch=('any')
 license=('MIT')
 source=("hello.tar.gz"
+        "helloworld-i18n-support.patch"
         "helloworld-fix-typo.patch")
 sha256sums=('d267ac47beecb944222bc2e8f7d6dd5e3ec9602cf4a4f3ed70b1e424c8213400'
+            '87d2f5b9365fbd0b2342f6632fc202f56d7237097596aa30e3aaf92aa2d19b86'
             '5635261bac29f1c0ed8cfbb4044e56bda9e85e9f50898fc0e6d1ca00f205f891')

 prepare() {
+  patch -Np1 -i helloworld-i18n-support.patch
   patch -Np1 -i helloworld-fix-typo.patch
 }

 build() {
   gcc -o hello hello.c
 }

 package() {
   install -Dm755 "$srcdir/hello" "$pkgdir/usr/bin/hello"
 }
+
+custompkgrel=1
+pkgrel="${custompkgrel}00${pkgrel}"

However, my experience is that it often takes a long time for a patch to be polished, submitted, accepted and released upstream; and during that time, keeping the package synchronized with the Arch Linux repositories or the AUR (via git pull) turns into a chore of resolving frequent spurious merge conflicts.

Instead, I have settled on a hack to keep all my patches at the end of the PKGBUILD. Behold:

# [...the original, unchanged PKGBUILD...]

custompkgrel=1
pkgrel="${custompkgrel}00${pkgrel}"

# This monkey-patches "prepare" to append a line of code at the beginning
# Note that this can be called multiple times, each appending a further line of code
before_prepare() {
  if [ -z "${before_prepare_script+x}" ]; then
    before_prepare_script=""
    eval "$(echo 'prepare() { eval "$before_prepare_script";'; declare -f prepare | head -n -1 | tail +3; echo '}')"
  fi
  before_prepare_script="$before_prepare_script$(printf "%q " "$@");"$'\n'
}

source+=("helloworld-i18n-support.patch")
sha256sums+=('87d2f5b9365fbd0b2342f6632fc202f56d7237097596aa30e3aaf92aa2d19b86')
before_prepare patch -Np1 -i helloworld-i18n-support.patch

With this method, all my changes are an appendage to the PKGBUILD, so they don't conflict. You might object that this is contrary to version control best practices, where a merge conflict is a strong hint that you must look at whatever has changed and confirm that it is correct.

However, my experience is that for this very particular case of a personal patch repository, this greatly improves the UX, while not sacrificing much: The patch command will fail when the code targeted by the .patch file changes, so there is still a conflict detection method in place.

Putting it all together

Usually, one of my patched PKGBUILD files looks like this:

# [...the original, unchanged PKGBUILD...]

custompkgrel=1
source "../custompkg.sh" || exit 1
# ZEALCHARM START Add i18n support
# See https://example.com/helloworld-org/helloworld/pull/1234
source+=("helloworld-i18n-support.patch")
sha256sums+=('87d2f5b9365fbd0b2342f6632fc202f56d7237097596aa30e3aaf92aa2d19b86')
before_prepare patch -Np1 -i helloworld-i18n-support.patch
# ZEALCHARM END Add i18n support

Here, I have refactored the previous utility into a custompkg.sh script, so I can share the common parts across all my patched packages.

The overall flow

When I decide to patch a PKGBUILD, what I do is:

Clone the PKGBUILD git repository, e.g. git clone https://gitlab.archlinux.org/archlinux/packaging/packages/helloworld.git
Apply and commit my changes to the PKGBUILD, as explained above.
Build the package and add it to a custom local repository.
Run pacman -Syu. Pacman will detect the package as an update, since the pkgrel has changed (e.g. 1.23-4 -> 1.23-1004).

To keep the package up to date, the process is similar:

Pull the PKGBUILD git repository, i.e. cd helloworld && git pull.
Rebuild the package and add it to the custom local repository.
Run pacman -Syu. Pacman will detect the package as an update, since the pkgver or pkgrel will have changed (e.g. 1.23-1004 -> 1.23-1005).

I have some scripts that automate most of this process, so the overall amount of work to keep the patched packages up to date is pretty small.

A pitfall of Python's subprocess Ctrl+C handling (compared to bash)

2024-08-10T00:00:00+00:00

Usually, I start writing my scripts in bash, and then translate them to Python if they get too complex. When doing so, I assume that invoking a simple command from a bash script is equivalent to a subprocess.run call. However, this is not true when it comes to Ctrl+C (aka. SIGINT) handling.

A reliable bash script

This bash script shows a common pattern where a script will first set up some state, does something over it, then cleans the state using a shell trap:

#!/usr/bin/env bash
set -eu

# Set up some temporary stuff and ensure that it is cleaned up on exit. The `sleep`s simulate non-trivial clean-up.
trap 'rm -f a && sleep 0.2 && rm -f b && sleep 0.2 && rm -f c' EXIT
touch a b c

# Simulate a long running process
tail -q -f a b c

For example, we may want to perform a backup using rsync. Before doing it, we unlock and mount the target volume using cryptsetup open and mount. Whether things go right or wrong, we clean up the setup steps with umount and cryptsetup close.

Granted, this pattern is not 100% bulletproof. If our script gets killed [or the power goes off, or the system crashes...] the cleanup step will never run, which may leave some junk behind. But since this is very uncommon, it's often acceptable to deal with it manually.

But something that may happen more often is that you forgot something, or you ran out of time, and you want to stop the process. In this case, you press Ctrl+C, which sends a SIGINT to the shell, which triggers the trap and cleans everything up properly:

$ ./shell_script_with_traps
^C
$ ls a b c
ls: cannot access 'a': No such file or directory
ls: cannot access 'b': No such file or directory
ls: cannot access 'c': No such file or directory

...maybe it wasn't so reliable after all

Thus, our script becomes part of our collection of reliable, battle-hardened scripts. We move to building greater things, which involve invoking that script using Python's subprocess.run. Can you guess what happens when we press Ctrl+C this time?

$ python -c "import subprocess; subprocess.run(['./shell_script_with_traps'])"
^C
[...the traceback for the KeyboardInterrupt exception...]
$ ls a b c
ls: cannot access 'a': No such file or directory
ls: cannot access 'b': No such file or directory
c

Only half of our cleanup trap seems to have run. Let's try using asyncio.create_subprocess_exec instead:

$ python -c "import asyncio
async def run():
    proc = await asyncio.create_subprocess_exec('./shell_script_with_traps')
    await proc.communicate()
asyncio.run(run())"
^C
[...the traceback for the KeyboardInterrupt exception...]
$ ls a b c
a  b  c

Now the trap does not appear to run at all. What's going on?

`SIGINT` handling in Python vs bash

When we press Ctrl+C, all processes in the foreground process group receive the SIGINT signal. So, both Python and bash get a chance to handle the signal.

As expected, bash will run the trap upon receiving SIGINT. But, on the other hand, Python's subprocess.run will give the bash process a grace period of 250 milliseconds to exit, and then forcefully kill the process. This is why the example's trap got interrupted halfway through.

When using asyncio.create_subprocess_exec, we don't even get this grace period, so the bash process gets killed immediately, often before the trap can run at all.

By contrast, in a bash-only scenario (if the script were called from within another bash script), the exit trap would not get interrupted. See this excerpt from Greg's Wiki:

bash is among a few shells that implement a wait and cooperative exit approach at handling SIGINT/SIGQUIT delivery. When interpreting a script, upon receiving a SIGINT, it doesn't exit straight away but instead waits for the currently running command to return and only exits (by killing itself with SIGINT) if that command was also killed by that SIGINT. The idea is that if your script calls vi for instance, and you press Ctrl+C within vi to cancel an action, that should not be considered as a request to abort the script.

Finally, note that bash and shell traps are irrelevant to the problem at large. For example, a C program that runs some expensive cleanup logic on SIGINT would also get prematurely killed by Python.

What can we do about it?

Part of the problem is that our bash script's cleanup logic was never truly reliable after all. We implemented something that was apparently good enough, but as it often happens, it turns out that our assumption was too optimistic and it's not so uncommon to get our shell script killed after all.

Unfortunately, it is not always simple or possible to implement bulletproof cleanup logic, which generally involves passing the buck to the operating system. For scenarios involving regular files, there are quite a few tricks, such as using pipes (including bash's process substitution), and various Linux extras such as open(..., O_TMPFILE), memfd_create, namespaces (see also: bubblewrap), and so on. But more complex scenarios involving cryptsetup, filesystem mounts, swap files, and so on, are often impossible or very awkward to handle this way.

We can also try to work around it from the Python side, though unfortunately there doesn't seem to be any simple way to do it without adding a bunch of complexity (switching to lower level interfaces such as subprocess.Popen and handling errors manually). This may be something to improve in the standard library in the future.

Another possibility could be to spawn the exit logic as a background process that waits for the foreground script to exit and then cleans up, though this feels like pushing in the direction of building a Rube Goldberg machine that will break in even more subtle ways.

Ultimately, I don't have any simple, universal and reliable fix for this at the moment. If you have any idea, don't hesitate to contact me!

Faster logins on dockerized SFTP

2021-06-02T00:00:00+00:00

I recently dockerized an SFTP file server setup on an old Raspberry Pi 2 server, and I was surprised by how slow logging in to the server was: about 4 seconds per connection¹.

My first intuition was that the time was lost to network latency or encryption, but the previous setup didn't have this problem, so that couldn't be it.

In fact the problem easily reproduces on a cheap VPS, though to a lesser degree:

$ docker run --name sftptest -d -p 127.0.0.1:1234:22 --entrypoint ash atmoz/sftp:alpine -c "/entrypoint foo:pass:::upload"
$ time sshpass -p pass ssh -p1234 -o PreferredAuthentications=password -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null foo@127.0.0.1
[...]
real    0m1.290s
user    0m0.014s
sys     0m0.008s
$ docker rm -f sftptest

At first, I tried playing with various OpenSSH switches and debugging options, but that didn't help much, as the time seemed to be lost "doing nothing". But after some more tinkering, I tried strace, and quickly found that it was trying to close several million file descriptors:

$ docker run --name sftptest -d -p 127.0.0.1:1234:22 --entrypoint ash atmoz/sftp:alpine -c "apk add strace && strace -f /entrypoint foo:pass:::upload"
$ timeout 10s sshpass -p pass ssh -p1234 -o PreferredAuthentications=password -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null foo@127.0.0.1
$ docker logs sftptest
[...a log of logs...]
[pid    39] close(119676)               = -1 EBADF (Bad file descriptor)
[pid    39] close(119677)               = -1 EBADF (Bad file descriptor)
[pid    39] close(119678)               = -1 EBADF (Bad file descriptor)
[pid    39] close(119679)               = -1 EBADF (Bad file descriptor)
[pid    39] close(119680)               = -1 EBADF (Bad file descriptor)
[pid    39] close(119681)               = -1 EBADF (Bad file descriptor)
[pid    39] close(119682)               = -1 EBADF (Bad file descriptor)
[...a lot of logs...]
^C
$ docker rm -f sftptest

After some more analysis, I found that all those close() calls seemed to come from a function named closefrom_fallback in OpenSSH, which will close all file descriptors up to OPEN_MAX in an attempt to ensure most file descriptors are closed².

So why does this only happen when running in a Docker container? The problem is that Docker containers will have by default a very generous limit of 1048576 open files, instead of the typical default of 1024 on typical setups:

$ getconf OPEN_MAX
1024
$ docker run --rm alpine:3.13 getconf OPEN_MAX
1048576

This high OPEN_MAX setting, when combined with the brute-force 'close all file descriptors' approach, has already caused various issues in various applications, but I haven't found anyone reporting this for OpenSSH, so I guess there are a lot of annoyingly slow SFTP logins out there.

Thankfully working around the problem is pretty easy once we know the cause, by specifying a smaller OPEN_MAX limit by passing the --ulimit nofile=1024:1024 flag to docker run to get about 14x faster logins:

$ docker run --name sftptest -d -p 127.0.0.1:1234:22 --ulimit nofile=1024:1024 --entrypoint ash atmoz/sftp:alpine -c "/entrypoint foo:pass:::upload"
$ time sshpass -p pass ssh -p1234 -o PreferredAuthentications=password -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null foo@127.0.0.1
[...]
real    0m0.091s
user    0m0.015s
sys     0m0.000s
$ docker rm -f sftptest

I was using nPlayer as the client, which seems to open a new connection on every folder change, so those 4 seconds quickly added up to a minute. ↩
I haven't looked in detail why this closefrom_fallback code path is hit, but it appears that OpenSSH will try to enumerate open files by looking at /proc/$$/fd, but it can't do this after chroot is called to jail the SFTP to a directory. ↩

Insanely easy to inadvertently expose services in development machines with Docker

2021-01-12T00:00:00+00:00

As you probably know, Docker is a popular way to deploy and develop software. If you use it for this second purpose, it's also insanely easy to inadvertently make your services available to more people than you, and too hard to ensure it doesn't.

(There's nothing really new is in this post, it's just a quick rant and summary)

The first day

So, you install Docker on your development machine and you run something like this:

$ docker run -p 80:80 myproject

And now everyone on your network (or even the entire Internet) can access your project by connecting to your machine on port 80, because the default binding address of Docker is 0.0.0.0 (listen to all interfaces).

I can understand this, because you may want to use Docker on an actual server. But this is your development machine, so you only want your service accessible on loopback. So you decide to solve the problem for good, and set the ip option on the /etc/docker/daemon.json configuration file (the "default IP when binding container ports") to 127.0.0.1 and call it a day, right?

The next day

With time your project grows, and you switch to docker-compose, so you run something like this:

$ cat docker-compose.yml
version: '3.3'
services:
    myproject:
        ports:
            - '80:80'
        image: myproject
$ docker-compose up

And now your project is again accessible to everyone on your network. This is docker-compose issue #2999. The problem is that the ip option you configured before actually only applies to the default network, and docker-compose generally creates a new network for your project, which will bind again to 0.0.0.0. And as far as I know, there's no way to configure the default binding address for new networks.

Use a firewall!

So you decide to stop trying to configure Docker and use a firewall instead. You install ufw which is very easy to install and configure. Problem solved, right?

Well, it turns out that Docker by default sets up iptables in a way that bypasses UFW, so you are going to have to apply some extra configuration to make it work.

Non-solutions

Always put the loopback IP on your docker commands and docker-compose.yml files: This is too error prone and you are eventually going to make an human mistake and expose your services to the outside world. And it's painful to have to change every port binding between development and production machines.
Don't you have an home router which also acts as a firewall? Yes, but it doesn't help if you have guests, or some other machine on your local network is compromised, or if there's some bug that allows the firewall to be bypassed (your random home router is probably not the best firewall on the market). And my philosophy is that you shouldn't rely on a firewall as the first and only security measure.
Use something else: Even with those problems, Docker is often the best option overall. And you don't always make those decisions.

Actual solutions

As far as I know, if you don't want to rely on some workaround, your best chance is to use firewalld (which Docker integrated with), a hardware/external firewall, or an isolated virtual machine.

UPDATE (2021-10-15): Nowadays, in addition to trying to avoid binding to all interfaces as much as I can, my go-to solution for development machines is using Docker in rootless mode instead, launching the daemon as an unprivileged user instead of root. Then, by design, Docker can no longer bypass the firewall. Be warned though, that rootless mode can be somewhat painful to set up if you aren't on a rolling/fast-moving Linux distro.

Final note

I am not an expert in Docker networking. There may be some simpler way to fix this. But even if a simple solution actually exists, the fact that there are multiple options that at first sight ought to work but don't shows that this is a problem. How many developers are one (or zero) steps away from exposing their services?

Headless setup of Alpine Linux on the Raspberry Pi

2020-01-08T00:00:00+00:00

Alpine Linux is a lightweight Linux distribution which has gained popularity mainly as a base image for Docker containers. It is also a good candidate for a small system like the Raspberry Pi. However, I couldn't find out how to do a headless setup, i.e. one without a monitor or keyboard, but rather just a network connection and SSH, unlike what's easily possible with Raspbian.

This post explains how to do it. The process is a bit more convoluted, which is specially inconvenient since you can't get feedback from a monitor, but it can be done. Additionally, an important limitation is that you will need an Ethernet connection for the initial setup, not a wireless connection.

To start, you will need to generate an SSH public/private key pair using ssh-keygen, which you will use to authenticate through SSH once the system is ready.

Afterwards, follow the Raspberry Pi "Preparation" section in the Alpine Linux Wiki, up to the point where you have extracted the root filesystem on the SD card.

Now, this is where the magic comes: It turns out that Alpine Linux's initramfs accepts some additional parameters in the kernel command line that allow us to initialize the network and add an SSH public key to the .ssh/authorized_keys file. Those are a bit obscure, yet more or less documented and perfect for our headless setup use case. We can add those parameters in the cmdline.txt file in the root of the SD card in order to get a SSH connection after the system boots for the first time.

The ip parameter theoretically accepts the value ip=dhcp for automatic IP assignment. However, this resulted in a boot failure in my experience, so we will use the static IP format instead. For a typical home network, this should look like the following: ip=192.168.1.222::192.168.1.1:255.255.255.0::eth0::192.168.1.1: (where 192.168.1.222 is the IP that the Raspberry Pi will have after booting). Note: You will be able to switch to DHCP after the initial setup.
The ssh_key parameter accepts an URL to an SSH public key. You can decide to upload your SSH public key somewhere, but if you have a computer on your same network as the Raspberry, an easy alternative is to serve the key yourself using e.g. Python's built-in HTTP server:

[~/temp_folder]$ ls
mykey.pub
[~/temp_folder]$ cat mykey.pub
ssh-rsa blablablablabla root@mykey
[~/temp_folder]$ python -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...

For this setup, the parameter should look like the following: ssh_key=http://192.168.1.111:8080/mykey.pub (where 192.168.1.111 is the IP of the computer serving the key).

Add those two parameters together in the cmdline.txt file in the root of the SD card, so the entire file should look like the following:

modules=loop,squashfs,sd-mod,usb-storage quiet console=tty1 ip=192.168.1.222::192.168.1.1:255.255.255.0::eth0::192.168.1.1: ssh_key=http://192.168.1.111:8080/mykey.pub

Finally, unmount the SD card and try to boot your Raspberry with it. After a bit (the first boot will take longer than usual due to SSH key generation), it should boot correctly and make an SSH server available. Otherwise, make sure you haven't missed any step from the regular Alpine setup, and check if the system responds to ping or if it fetched the SSH key to diagnose the problem.

If everything went well, you should be able to log as the root user through SSH:

[~]$ ssh root@192.168.1.222 -i mykey
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

localhost:~#

After this, follow the steps in the Alpine Linux Wiki and run setup-alpine to install the system. Afterwards, make sure to persist the SSH key with lbu, in order to keep SSH access across reboots:

localhost:~# lbu include /root/.ssh/authorized_keys 
localhost:~# lbu commit -d

Finally, shut down the Raspberry, plug the SD card back into your computer, and remove the ip and ssh_key parameters from cmdline.txt. You should now have a clean, SSH-accessible Alpine Linux install on your Raspberry Pi.