3. Charliecloud command reference¶

3.1.1. Synopsis¶

$ ch-build [-b BUILDER] [--builder-info] -t TAG [ARGS ...] CONTEXT

3.1.2. Description¶

Build an image named TAG described by a Dockerfile. Place the result into the builder’s back-end storage.

Using this script is not required for a working Charliecloud image. You can also use any builder that can produce a Linux filesystem tree directly, whether or not it is in the list below. However, this script hides the vagaries of making the supported builders work smoothly with Charliecloud and adds some conveniences (e.g., pass HTTP proxy environment variables to the build environment if the builder doesn’t do this by default).

Supported builders, unprivileged:

• ch-image: Our internal builder.

Supported builders, privileged:

• docker: Docker.

Experimental builders (i.e., the code is there but not tested much):

• buildah: Buildah in “rootless” mode with no setuid helpers, using ch-run (via ch-run-oci) for RUN instructions. This mode is fully unprivileged.

• buildah-runc: Buildah in “rootless” mode with setuid helpers, using the default runc for RUN instructions.

• buildah-setuid: Buildah in “rootless” mode with setuid helpers, using ch-run (via ch-run-oci) for RUN instructions.

Specifying the builder, in descending order of priority:

-b, --builder BUILDER

Command line option.

$CH_BUILDER

Environment variable

Default

docker if Docker is installed; otherwise, ch-image.

Other arguments:

--builder-info

Print the builder to be used and its version, then exit.

-f, --file DOCKERFILE

Dockerfile to use (default: $CONTEXT/Dockerfile)

-t TAG

Name (tag) of Docker image to build.

--help

Print help and exit.

--version

Print version and exit.

Additional arguments are accepted and passed unchanged to the underlying builder.

3.1.3. Bugs¶

The tag suffix :latest is somewhat misleading, as by default neither ch-build nor bare builders will notice if the base FROM image has been updated. Use --pull to make sure you have the latest base image.

3.1.4. Examples¶

Create an image tagged foo and specified by the file Dockerfile located in the context directory. Use /bar as the Docker context directory. Use the default builder.

$ ch-build -t foo /bar

Equivalent to above:

$ ch-build -t foo --file=/bar/Dockerfile /bar

Instead, use /bar/Dockerfile.baz:

$ ch-build -t foo --file=/bar/Dockerfile.baz /bar

Equivalent to the first example, but use ch-image even if Docker is installed:

$ ch-build -b ch-image -t foo /bar

Equivalent to above:

$ export CH_BUILDER=ch-image
$ ch-build -t foo /bar

3.2.1. Synopsis¶

$ ch-build2dir -t TAG [ARGS ...] CONTEXT OUTDIR

3.2.2. Description¶

Build a Docker image named TAG described by a Dockerfile (default $CONTEXT/Dockerfile) and unpack it into OUTDIR/TAG. This is a wrapper for ch-build, ch-builder2tar, and ch-tar2dir; see also those man pages.

Arguments:

ARGS

additional arguments passed to ch-build

CONTEXT

Docker context directory

OUTDIR

directory in which to place image directory (named TAG) and temporary tarball

-t TAG

name (tag) of Docker image to build

--help

print help and exit

--version

print version and exit

3.2.3. Examples¶

To build using ./Dockerfile and create image directory /var/tmp/foo:

$ ch-build2dir -t foo . /var/tmp

Same as above, but build with a different Dockerfile:

$ ch-build2dir -t foo -f ./Dockerfile.foo . /var/tmp

3.3.1. Synopsis¶

$ ch-builder2tar [-b BUILDER] [--nocompress] IMAGE OUTDIR

3.3.2. Description¶

Flatten the builder image tagged IMAGE into a Charliecloud tarball in directory OUTDIR.

The builder-specified environment (e.g., ENV statements) is placed in a file in the tarball at $IMAGE/ch/environment, in a form suitable for ch-run --set-env.

See ch-build(1) for details on specifying the builder.

Additional arguments:

-b, --builder BUILDER

Use specified builder; if not given, use $CH_BUILDER or default.

--nocompress

Do not compress tarball.

--help

Print help and exit.

--version

Print version and exit.

3.3.3. Example¶

$ ch-builder2tar hello /var/tmp
57M /var/tmp/hello.tar.gz
$ ls -lh /var/tmp
-rw-r-----  1 reidpr reidpr  57M Feb 13 16:14 hello.tar.gz

3.4.1. Synopsis¶

$ ch-checkns

3.4.2. Description¶

Check ch-run prerequisites, e.g., namespaces and pivot_root(2).

3.4.3. Example¶

$ ch-checkns
ok

3.5.1. Synopsis¶

$ ch-dir2squash IMGDIR OUTDIR [ARGS ...]

3.5.2. Description¶

Create Charliecloud SquashFS file from image directory IMGDIR under directory OUTDIR, named as last component of IMGDIR plus suffix .sqfs.

Optional ARGS will passed to mksquashfs unchanged.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.5.3. Example¶

$ ch-dir2squash /var/tmp/debian /var/tmp
Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on /var/tmp/debian.sqfs, block size 131072.
[...]
-rw-r--r--  1 charlie charlie 41M Apr 23 14:41 /var/tmp/debian.sqfs

3.6.1. Synopsis¶

$ ch-builder2squash [-b BUILDER] IMAGE OUTDIR [ARGS ...]

3.6.2. Description¶

Flattens the builder image tagged IMAGE into a SquashFS file in OUTDIR.

Wrapper for ch-builder2tar --nocompress and ch-tar2sqfs. Intermediate files and directories are removed.

Sudo privileges are required to run docker export.

Optional ARGS passed to mksquashfs unchanged.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.6.3. Example¶

$ docker image list | fgrep debian
REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
debian       stretch   2d337f242f07   3 weeks ago  101MB
$ ch-builder2squash debian /var/tmp
Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on /var/tmp/debian.sqfs, block size 131072.
[...]
squashed /var/tmp/debian.sqfs OK
$ ls -lh /var/tmp/debian*
-rw-r--r-- 1 charlie charlie 41M Apr 23 14:37 debian.sqfs

3.7.1. Synopsis¶

$ ch-fromhost [OPTION ...] [FILE_OPTION ...] IMGDIR

3.7.2. Description¶

Inject files from the host into the Charliecloud image directory IMGDIR.

The purpose of this command is to inject files into a container image that are necessary to run the container on a specific host; e.g., GPU libraries that are tied to a specific kernel version. It is not a general copy-to-image tool; see further discussion on use cases below. It should be run after ch-tar2dir and before ch-run. After invocation, the image is no longer portable to other hosts.

Injection is not atomic; if an error occurs partway through injection, the image is left in an undefined state. Injection is currently implemented using a simple file copy, but that may change in the future.

By default, file paths that contain the strings /bin or /sbin are assumed to be executables and placed in /usr/bin within the container. File paths that contain the strings /lib or .so are assumed to be shared libraries and are placed in the first-priority directory reported by ldconfig (see --lib-path below). Other files are placed in the directory specified by --dest.

If any shared libraries are injected, run ldconfig inside the container (using ch-run -w) after injection.

3.7.3. Options¶

3.7.4. When to use ch-fromhost¶

This command does a lot of heuristic magic; while it can copy arbitrary files into an image, this usage is discouraged and prone to error. Here are some use cases and the recommended approach:

1. I have some files on my build host that I want to include in the image. Use the COPY instruction within your Dockerfile. Note that it’s OK to build an image that meets your specific needs but isn’t generally portable, e.g., only runs on specific micro-architectures you’re using.

2. I have an already built image and want to install a program I compiled separately into the image. Consider whether a building a new derived image with a Dockerfile is appropriate. Another good option is to bind-mount the directory containing your program at run time. A less good option is to cp(1) the program into your image, because this permanently alters the image in a non-reproducible way.

3. I have some shared libraries that I need in the image for functionality or performance, and they aren’t available in a place where I can use COPY. This is the intended use case of ch-fromhost. You can use --cmd, --file, and/or --path to put together a custom solution. But, please consider filing an issue so we can package your functionality with a tidy option like --cray-mpi or --nvidia.

3.7.5. --cray-mpi dependencies and quirks¶

The implementation of --cray-mpi is messy, foul smelling, and brittle. It replaces or overrides the MPICH or OpenMPI libraries installed in the container. Users should be aware of the following.

1. Containers must have the following software installed:

   1. Corresponding open source MPI implementation. (MPICH and OpenMPI.)

   2. PatchELF with our patches. Use the shrink-soname branch. (MPICH only.)

   3. libgfortran.so.3, because Cray’s libmpi.so.12 links to it. (MPICH only.)

2. Applications must be dynamically linked to libmpi.so.12 (not e.g. libmpich.so.12).

   1. How to configure MPICH to accomplish this is not yet clear to us; test/Dockerfile.mpich does it, while the Debian packages do not. (MPICH only.)

3. An ABI compatible module for the given MPI implementation must be loaded when ch-fromhost is invoked.

   1. Load the cray-mpich-abi module. (MPICH only.)

   2. We recommend loading the module of a version as close to what is installed in the image as possible. This OpenMPI install needs to be built such that libmpi contains all needed plugins (as opposed to them being standalone shared libraries). See OpenMPI’s documentation for how to do this. (OpenMPI only.)

4. Tested only for C programs compiled with GCC, and it probably won’t work otherwise. If you’d like to use another compiler or another programming language, please get in touch so we can implement the necessary support.

Please file a bug if we missed anything above or if you know how to make the code better.

3.7.6. Notes¶

Symbolic links are dereferenced, i.e., the files pointed to are injected, not the links themselves.

As a corollary, do not include symlinks to shared libraries. These will be re-created by ldconfig.

There are two alternate approaches for nVidia GPU libraries:

1. Link libnvidia-containers into ch-run and call the library functions directly. However, this would mean that Charliecloud would either (a) need to be compiled differently on machines with and without nVidia GPUs or (b) have libnvidia-containers available even on machines without nVidia GPUs. Neither of these is consistent with Charliecloud’s philosophies of simplicity and minimal dependencies.

2. Use nvidia-container-cli configure to do the injecting. This would require that containers have a half-started state, where the namespaces are active and everything is mounted but pivot_root(2) has not been performed. This is not feasible because Charliecloud has no notion of a half-started container.

Further, while these alternate approaches would simplify or eliminate this script for nVidia GPUs, they would not solve the problem for other situations.

3.7.7. Bugs¶

File paths may not contain colons or newlines.

3.7.8. Examples¶

Place shared library /usr/lib64/libfoo.so at path /usr/lib/libfoo.so (assuming /usr/lib is the first directory searched by the dynamic loader in the image), within the image /var/tmp/baz and executable /bin/bar at path /usr/bin/bar. Then, create appropriate symlinks to libfoo and update the ld.so cache.

$ cat qux.txt
/bin/bar
/usr/lib64/libfoo.so
$ ch-fromhost --file qux.txt /var/tmp/baz

Same as above:

$ ch-fromhost --cmd 'cat qux.txt' /var/tmp/baz

Same as above:

$ ch-fromhost --path /bin/bar --path /usr/lib64/libfoo.so /var/tmp/baz

Same as above, but place the files into /corge instead (and the shared library will not be found by ldconfig):

$ ch-fromhost --dest /corge --file qux.txt /var/tmp/baz

Same as above, and also place file /etc/quux at /etc/quux within the container:

$ ch-fromhost --file qux.txt --dest /etc --path /etc/quux /var/tmp/baz

Inject the executables and libraries recommended by nVidia into the image, and then run ldconfig:

$ ch-fromhost --nvidia /var/tmp/baz

Inject the Cray-enabled MPI libraries into the image, and then run ldconfig:

$ ch-fromhost --cray-mpi /var/tmp/baz

3.7.9. Acknowledgements¶

This command was inspired by the similar Shifter feature that allows Shifter containers to use the Cray Aries network. We particularly appreciate the help provided by Shane Canon and Doug Jacobsen during our implementation of --cray-mpi.

We appreciate the advice of Ryan Olson at nVidia on implementing --nvidia.

3.8.1. Synopsis¶

$ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT
$ ch-image [...] delete IMAGE_REF
$ ch-image [...] list
$ ch-image [...] pull [...] IMAGE_REF [IMAGE_DIR]
$ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF]
$ ch-image [...] reset
$ ch-image [...] storage-path
$ ch-image { --help | --version | --dependencies }

3.8.2. Description¶

ch-image is a tool for building and manipulating container images, but not running them (for that you want ch-run). It is completely unprivileged, with no setuid/setgid/setcap helpers.

Options that print brief information and then exit:

-h, --help

Print help and exit successfully.

--dependencies

Report dependency problems on standard output, if any, and exit. If all is well, there is no output and the exit is successful; in case of problems, the exit is unsuccessful.

--version

Print version number and exit successfully.

Common options placed before the sub-command:

--no-cache

Download everything needed, ignoring the cache.

-s, --storage DIR

Set the storage directory (see below for important details).

--tls-no-verify

Don’t verify TLS certificates of the repository. (Do not use this option unless you understand the risks.)

-v, --verbose

Print extra chatter; can be repeated.

3.8.3. Storage directory¶

ch-image maintains state using normal files and directories, including unpacked container images, located in its storage directory. There is no notion of storage drivers, graph drivers, etc., to select and/or configure. In descending order of priority, this directory is located at:

-s, --storage DIR

Command line option.

$CH_IMAGE_STORAGE

Environment variable.

/var/tmp/$USER/ch-image

Default.

The storage directory can reside on any filesystem. However, it contains lots of small files and metadata traffic can be intense. For example, the Charliecloud test suite uses approximately 400,000 files and directories in the storage directory as of this writing. Place it on a filesystem appropriate for this; tmpfs’es such as /var/tmp are a good choice if you have enough RAM (/tmp is not recommended because ch-run bind-mounts it into containers by default).

While you can currently poke around in the storage directory and find unpacked images runnable with ch-run, this is not a supported use case. The supported workflow uses ch-builder2tar or ch-builder2squash to obtain a packed image; see the tutorial for details.

3.8.4. Subcommands¶

  Updating   : 1:dbus-1.10.24-13.el7_6.x86_64                            2/4
Error unpacking rpm package 1:dbus-1.10.24-13.el7_6.x86_64
error: unpacking of archive failed on file /usr/libexec/dbus-1/dbus-daemon-launch-helper;5cffd726: cpio: chown
  Cleanup    : 1:dbus-libs-1.10.24-12.el7.x86_64                         3/4
error: dbus-1:1.10.24-13.el7_6.x86_64: install failed

E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 100 failed - seteuid (22: Invalid argument)
E: setgroups 0 failed - setgroups (1: Operation not permitted)

3.8.5. Compatibility with other Dockerfile interpreters¶

ch-image is an independent implementation and shares no code with other Dockerfile interpreters. It uses a formal Dockerfile parsing grammar developed from the Dockerfile reference documentation and miscellaneous other sources, which you can examine in the source code.

We believe this independence is valuable for several reasons. First, it helps the community examine Dockerfile syntax and semantics critically, think rigorously about what is really needed, and build a more robust standard. Second, it yields disjoint sets of bugs (note that Podman, Buildah, and Docker all share the same Dockerfile parser). Third, because it is a much smaller code base, it illustrates how Dockerfiles work more clearly. Finally, it allows straightforward extensions if needed to support scientific computing.

ch-image tries hard to be compatible with Docker and other interpreters, though as an independent implementation, it is not bug-compatible.

This section describes differences from the Dockerfile reference that we expect to be approximately permanent. For an overview of features we have not yet implemented and our plans, see our road map on GitHub. Plain old bugs are in our GitHub issues.

None of these are set in stone. We are very interested in feedback on our assessments and open questions. This helps us prioritize new features and revise our thinking about what is needed for HPC containers.

HTTP_PROXY
http_proxy
HTTPS_PROXY
https_proxy
FTP_PROXY
ftp_proxy
NO_PROXY
no_proxy

PATH=/ch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TAR_OPTIONS=--no-same-owner

3.8.6. Environment variables¶

CH_LOG_FILE

If set, append log chatter to this file, rather than standard error. This is useful for debugging situations where standard error is consumed or lost.

Also sets verbose mode if not already set (equivalent to --verbose).

CH_LOG_FESTOON

If set, prepend PID and timestamp to logged chatter.

3.8.7. Examples¶

$ ch-image build -t bar -f ./foo/bar/Dockerfile ./foo/bar
[...]
grown in 4 instructions: bar

$ ch-image build ./foo/bar
[...]
grown in 4 instructions: bar

$ ch-image build --bind /opt/bigvendor:/opt .
$ cat Dockerfile
FROM centos:7

RUN /opt/bin/cc hello.c
#COPY /opt/lib/*.so /usr/local/lib   # fail: COPY doesn't bind mount
RUN cp /opt/lib/*.so /usr/local/lib  # possible workaround
RUN ldconfig

$ ch-image pull debian:buster
pulling image:   debian:buster

manifest: downloading
layer 1/1: d6ff36c: downloading
layer 1/1: d6ff36c: listing
validating tarball members
resolving whiteouts
flattening image
layer 1/1: d6ff36c: extracting
done

$ ch-image pull debian:buster /tmp/buster
[...]
$ ls /tmp/buster
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

$ ch-image push example.com:5000/foo/bar:latest
pushing image:   example.com:5000/foo/bar:latest
layer 1/1: gathering
layer 1/1: preparing
preparing metadata
starting upload
layer 1/1: a1664c4: checking if already in repository
layer 1/1: a1664c4: not present, uploading
config: 89315a2: checking if already in repository
config: 89315a2: not present, uploading
manifest: uploading
cleaning up
done

$ ch-image push alpine:3.9 example.com:5000/foo/bar:latest
pushing image:   alpine:3.9
destination:     example.com:5000/foo/bar:latest
layer 1/1: gathering
layer 1/1: preparing
preparing metadata
starting upload
layer 1/1: a1664c4: checking if already in repository
layer 1/1: a1664c4: not present, uploading
config: 89315a2: checking if already in repository
config: 89315a2: not present, uploading
manifest: uploading
cleaning up
done

$ ch-image push --image /var/tmp/image example.com:5000/foo/bar:latest
pushing image:   example.com:5000/foo/bar:latest
image path:      /var/tmp/image
layer 1/1: gathering
layer 1/1: preparing
preparing metadata
starting upload
layer 1/1: 892e38d: checking if already in repository
layer 1/1: 892e38d: already present
config: 546f447: checking if already in repository
config: 546f447: not present, uploading
manifest: uploading
cleaning up
done

3.9.1. Synopsis¶

$ ch-mount SQFS PARENTDIR

3.9.2. Description¶

Create new empty directory named SQFS with suffix (e.g., .sqfs) removed, then mount SQFS on this new directory. This new directory must not already exist.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.9.3. Example¶

$ ch-mount /var/tmp/debian.sqfs /var/tmp
$ ls /var/tmp/debian
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr  WEIRD_AL_YANKOVIC

3.10.1. Synopsis¶

$ ch-pull2dir IMAGE[:TAG] DIR

3.10.2. Description¶

Pull Docker image named IMAGE[:TAG] from Docker Hub and extract it into a subdirectory of DIR. A temporary tarball is stored in DIR.

Sudo privileges are required to run the docker pull command.

This runs the following command sequence: ch-pull2tar, ch-tar2dir. See warning in the documentation for ch-tar2dir.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.10.3. Examples¶

$ ch-pull2dir alpine /var/tmp
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Image is up to date for alpine:latest
-rw-r--r--. 1 charlie charlie 2.1M Oct  5 19:52 /var/tmp/alpine.tar.gz
creating new image /var/tmp/alpine
/var/tmp/alpine unpacked ok
removed '/var/tmp/alpine.tar.gz'

Same as above, except optional TAG is specified:

$ ch-pull2dir alpine:3.6 /var/tmp
3.6: Pulling from library/alpine
Digest: sha256:cc24af836d1377e092ecb4e8f0a4324c3b1aa2b5295c2239edcc7bbc86a9cbc6
Status: Image is up to date for alpine:3.6
-rw-r--r--. 1 charlie charlie 2.1M Oct  5 19:54 /var/tmp/alpine:3.6.tar.gz
creating new image /var/tmp/alpine:3.6
/var/tmp/alpine:3.6 unpacked ok
removed '/var/tmp/alpine:3.6.tar.gz'

3.11.1. Synopsis¶

$ ch-pull2tar IMAGE[:TAG] OUTDIR

3.11.2. Description¶

Pull a Docker image named IMAGE[:TAG] from Docker Hub and flatten it into a Charliecloud tarball in directory OUTDIR.

This runs the following command sequence: docker pull, ch-builder2tar but provides less flexibility than the individual commands.

Sudo privileges are required for docker pull.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.11.3. Examples¶

$ ch-pull2tar alpine /var/tmp
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Image is up to date for alpine:latest
-rw-r--r--. 1 charlie charlie 2.1M Oct  5 19:52 /var/tmp/alpine.tar.gz

Same as above, except optional TAG is specified:

$ ch-pull2tar alpine:3.6
3.6: Pulling from library/alpine
Digest: sha256:cc24af836d1377e092ecb4e8f0a4324c3b1aa2b5295c2239edcc7bbc86a9cbc6
Status: Image is up to date for alpine:3.6
-rw-r--r--. 1 charlie charlie 2.1M Oct  5 19:54 /var/tmp/alpine:3.6.tar.gz

3.12.1. Synopsis¶

$ ch-run [OPTION...] NEWROOT CMD [ARG...]

3.12.2. Description¶

Run command CMD in a Charliecloud container using the flattened and unpacked image directory located at NEWROOT.

-b, --bind=SRC[:DST]

mount SRC at guest DST (default /mnt/0, /mnt/1, etc.)

-c, --cd=DIR

initial working directory in container

--ch-ssh

bind ch-ssh(1) into container at /usr/bin/ch-ssh

-g, --gid=GID

run as group GID within container

-j, --join

use the same container (namespaces) as peer ch-run invocations

--join-pid=PID

join the namespaces of an existing process

--join-ct=N

number of ch-run peers (implies --join; default: see below)

--join-tag=TAG

label for ch-run peer group (implies --join; default: see below)

--no-home

do not bind-mount your home directory (by default, your home directory is mounted at /home/$USER in the container)

-t, --private-tmp

use container-private /tmp (by default, /tmp is shared with the host)

--set-env=FILE

set environment variables as specified in host path FILE

-u, --uid=UID

run as user UID within container

--unset-env=GLOB

unset environment variables whose names match GLOB

-v, --verbose

be more verbose (debug if repeated)

-w, --write

mount image read-write (by default, the image is mounted read-only)

-?, --help

print help and exit

--usage

print a short usage message and exit

-V, --version

print version and exit

3.12.3. Host files and directories available in container via bind mounts¶

In addition to any directories specified by the user with --bind, ch-run has standard host files and directories that are bind-mounted in as well.

The following host files and directories are bind-mounted at the same location in the container. These cannot be disabled.

• /dev

• /etc/passwd

• /etc/group

• /etc/hosts

• /etc/resolv.conf

• /proc

• /sys

Three additional bind mounts can be disabled by the user:

• Your home directory (i.e., $HOME) is mounted at guest /home/$USER by default. This is accomplished by mounting a new tmpfs at /home, which hides any image content under that path. If --no-home is specified, neither of these things happens and the image’s /home is exposed unaltered.

• /tmp is shared with the host by default. If --private-tmp is specified, a new tmpfs is mounted on the guest’s /tmp instead.

• If file /usr/bin/ch-ssh is present in the image, it is over-mounted with the ch-ssh binary in the same directory as ch-run.

3.12.4. Multiple processes in the same container with --join¶

By default, different ch-run invocations use different user and mount namespaces (i.e., different containers). While this has no impact on sharing most resources between invocations, there are a few important exceptions. These include:

1. ptrace(2), used by debuggers and related tools. One can attach a debugger to processes in descendant namespaces, but not sibling namespaces. The practical effect of this is that (without --join), you can’t run a command with ch-run and then attach to it with a debugger also run with ch-run.

2. Cross-memory attach (CMA) is used by cooperating processes to communicate by simply reading and writing one another’s memory. This is also not permitted between sibling namespaces. This affects various MPI implementations that use CMA to pass messages between ranks on the same node, because it’s faster than traditional shared memory.

--join is designed to address this by placing related ch-run commands (the “peer group”) in the same container. This is done by one of the peers creating the namespaces with unshare(2) and the others joining with setns(2).

To do so, we need to know the number of peers and a name for the group. These are specified by additional arguments that can (hopefully) be left at default values in most cases:

• --join-ct sets the number of peers. The default is the value of the first of the following environment variables that is defined: OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE, SLURM_CPUS_ON_NODE.

• --join-tag sets the tag that names the peer group. The default is environment variable SLURM_STEP_ID, if defined; otherwise, the PID of ch-run’s parent. Tags can be re-used for peer groups that start at different times, i.e., once all peer ch-run have replaced themselves with the user command, the tag can be re-used.

Caveats:

• One cannot currently add peers after the fact, for example, if one decides to start a debugger after the fact. (This is only required for code with bugs and is thus an unusual use case.)

• ch-run instances race. The winner of this race sets up the namespaces, and the other peers use the winner to find the namespaces to join. Therefore, if the user command of the winner exits, any remaining peers will not be able to join the namespaces, even if they are still active. There is currently no general way to specify which ch-run should be the winner.

• If --join-ct is too high, the winning ch-run’s user command exits before all peers join, or ch-run itself crashes, IPC resources such as semaphores and shared memory segments will be leaked. These appear as files in /dev/shm/ and can be removed with rm(1).

• Many of the arguments given to the race losers, such as the image path and --bind, will be ignored in favor of what was given to the winner.

3.12.5. Environment variables¶

ch-run leaves environment variables unchanged, i.e. the host environment is passed through unaltered, except:

• limited tweaks to avoid significant guest breakage;

• user-set variables via --set-env;

• user-unset variables via --unset-env; and

• set CH_RUNNING.

This section describes these features.

The default tweaks happen first, and then --set-env and --unset-env in the order specified on the command line. The latter two can be repeated arbitrarily many times, e.g. to add/remove multiple variable sets or add only some variables in a file.

$ export FOO=bar
$ env | fgrep FOO
FOO=bar
$ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
$

$ salloc -N1
$ env | egrep '^SLURM' | wc
   44      44    1092
$ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
[... long error message ...]
$ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
0: MPI version:
Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
0: init ok cn001.localdomain, 1 ranks, userns 4026532530
0: send/receive ok
0: finalize ok

$ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
$

$ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env
SHLVL=1
PWD=/
_=/usr/bin/env
$

3.12.6. Examples¶

Run the command echo hello inside a Charliecloud container using the unpacked image at /data/foo:

$ ch-run /data/foo -- echo hello
hello

Run an MPI job that can use CMA to communicate:

$ srun ch-run --join /data/foo -- bar

3.13.1. Synopsis¶

$ ch-run-oci OPERATION [ARG ...]

3.13.2. Description¶

Open Containers Initiative (OCI) wrapper for ch-run(1). You probably don’t want to run this command directly; it is intended to interface with other software that expects an OCI runtime. The current goal is to support completely unprivileged image building (e.g. buildah --runtime=ch-run-oci) rather than general OCI container running.

Support of the OCI runtime specification is only partial. This is for two reasons. First, it’s an experimental and incomplete feature. More importantly, the philosophy and goals of OCI differ significantly from those of Charliecloud. Key differences include:

• OCI is designed to run services, while Charliecloud is designed to run scientific applications.

• OCI containers are persistent things with a complex lifecycle, while Charliecloud containers are simply UNIX processes.

• OCI expects support for a variety of namespaces, while Charliecloud supports user and mount, no more and no less.

• OCI expects runtimes to maintain a supervisor process in addition to user processes; Charliecloud has no need for this.

• OCI expects runtimes to maintain state throughout the container lifecycle in a location independent from the caller.

For these reasons, ch-run-oci is a bit of a kludge, and much of what it does is provide scaffolding to satisfy OCI requirements.

Which OCI features are and are not supported is provided in the rest of this man page, and technical analysis and discussion are in the Contributor’s Guide.

This command supports OCI version 1.0.0 only and fails with an error if other versions are offered.

3.13.3. Operations¶

All OCI operations are accepted, but some are no-ops or merely scaffolding to satisfy the caller. For comparison, see also:

• OCI runtime and lifecycle spec

• The runc man pages

$ ch-run-oci create --bundle DIR --pid-file FILE [--no-new-keyring] CONTAINER_ID

$ ch-run-oci delete CONTAINER_ID

$ ch-run-oci kill CONTAINER_ID

$ ch-run-oci start CONTAINER_ID

$ ch-run-oci state CONTAINER_ID

3.13.4. Unsupported OCI features¶

As noted above, various OCI features are not supported by Charliecloud. We have tried to guess which features would be essential to callers; ch-run-oci fails with an error if these are requested. Otherwise, the request is simply ignored.

We are interested in hearing about scientific-computing use cases for unsupported features, so we can add support for things that are needed.

Our goal is for this man page to be comprehensive: every OCI runtime feature should either work or be listed as unsupported.

Unsupported features that are an error:

• Pseudoterminals

• Hooks (prestart, poststart, and prestop)

• Annotations

• Joining existing namespaces

• Intel Resource Director Technology (RDT)

Unsupported features that are ignored:

• Mounts other than the root filesystem (we do use --no-home)

• User/group mappings beyond one user mapped to EUID and one group mapped to EGID

• Disabling prctl(PR_SET_NO_NEW_PRIVS)

• Root filesystem propagation mode

• sysctl directives

• masked and read-only paths (remaining unprivileged protects you)

• Capabilities

• rlimits

• Devices (all devices are inherited from the host)

• cgroups

• seccomp

• SELinux

• AppArmor

• Container hostname setting

3.13.5. Environment variables¶

CH_LOG_FILE

If set, append log chatter to this file, rather than standard error. This is useful for debugging situations where standard error is consumed or lost.

Also sets verbose mode if not already set (equivalent to --verbose).

CH_LOG_FESTOON

If set, prepend PID and timestamp to logged chatter.

CH_RUN_OCI_HANG

If set to the name of a command (e.g., create), sleep indefinitely when that command is invoked. The purpose here is to halt a build so it can be examined and debugged.

3.14.1. Synopsis¶

$ CH_RUN_ARGS="NEWROOT [ARG...]"
$ ch-ssh [OPTION...] HOST CMD [ARG...]

3.14.2. Description¶

Runs command CMD in a Charliecloud container on remote host HOST. Use the content of environment variable CH_RUN_ARGS as the arguments to ch-run on the remote host.

3.14.3. Example¶

On host bar.example.com, run the command echo hello inside a Charliecloud container using the unpacked image at /data/foo with starting directory /baz:

$ hostname
foo
$ export CH_RUN_ARGS='--cd /baz /data/foo'
$ ch-ssh bar.example.com -- hostname
bar

3.15.1. Synopsis¶

$ ch-tar2dir TARBALL DIR

3.15.2. Description¶

Extract the tarball TARBALL into a subdirectory of DIR. TARBALL must contain a Linux filesystem image, e.g. as created by ch-builder2tar, and be compressed with gzip or xz. If TARBALL has no extension, try appending .tar.gz and .tar.xz.

Inside DIR, a subdirectory will be created whose name corresponds to the name of the tarball with .tar.gz or other suffix removed. If such a directory exists already and appears to be a Charliecloud container image, it is removed and replaced. If the existing directory doesn’t appear to be a container image, the script aborts with an error.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.15.3. Example¶

$ ls -lh /var/tmp
total 57M
-rw-r-----  1 reidpr reidpr  57M Feb 13 16:14 hello.tar.gz
$ ch-tar2dir /var/tmp/hello.tar.gz /var/tmp
creating new image /var/tmp/hello
/var/tmp/hello unpacked ok
$ ls -lh /var/tmp
total 57M
drwxr-x--- 22 reidpr reidpr 4.0K Feb 13 16:29 hello
-rw-r-----  1 reidpr reidpr  57M Feb 13 16:14 hello.tar.gz

3.16.1. Synopsis¶

$ ch-test [PHASE] [--scope SCOPE] [ARGS]

3.16.2. Description¶

Charliecloud comes with a comprehensive test suite that exercises the container workflow itself as well as a few example applications. ch-test coordinates running the test suite.

While the CLI has lots of options, the defaults are reasonable, and bare ch-test will give useful results in a few minutes on single-node, internet-connected systems with a few GB available in /var/tmp.

The test suite requires a few GB (standard scope) or tens of GB (full scope) of storage for test fixtures:

• Builder storage (e.g., layer cache). This goes wherever the builder puts it.

• Packed images directory: image tarballs or SquashFS files.

• Unpacked images directory. Images are unpacked into and then run from here.

• Filesystem permissions directories. These are used to test that the kernel is enforcing permissions correctly. Note that this exercises the kernel, not Charliecloud, and can be omitted from routine Charliecloud testing.

The first three are created when needed if they don’t exist, while the filesystem permissions fixtures must be created manually, in order to accommodate configurations where sudo is not available via the same login path used for running tests.

The packed and unpacked image directories specified for testing are volatile. The contents of these directories are deleted before the build and run phases, respectively.

In all four cases, when creating directories, only the final path component is created. Parent directories must already exist, i.e., ch-test uses the behavior of mkdir rather than mkdir -p.

Some of the tests exercise parallel functionality. If ch-test is run on a single node, multiple cores will be used; if in a Slurm allocation, multiple nodes too.

The subset of tests to run mostly splits along two key dimensions. The phase is which parts of the workflow to run. Different parts of the workflow can be tested on different systems by copying the necessary artifacts between them, e.g. by building images on one system and running them on another. The scope allows trading off thoroughness versus time.

PHASE must be one of the following:

build

Image building and associated functionality, with the selected builder.

run

Running containers and associated functionality. This requires a packed images directory produced by a successful build phase, which can be copied from the build system if it’s not also the run system.

examples

Example applications. Requires an unpacked images directory produced by a successful run phase.

all

Execute phases build, run, and examples, in that order.

mk-perm-dirs

Create the filesystem permissions directories. Requires --perm-dirs.

clean

Delete automatically-generated test files, and packed and unpacked image directories.

rm-perm-dirs

Remove the filesystem permissions directories. Requires --perm-dirs.

-f, --file FILE[:TEST]

Run the tests in the given file only, which can be an arbitrary .bats file, except for test.bats under examples, where you must specify the corresponding Dockerfile or Build file instead. This is somewhat brittle and typically used for development or debugging. For example, it does not check whether the pre-requisites of whatever is in the file are satisfied. Often running build and run first is sufficient, but this varies.

If TEST is also given, then run only the test with that name, skipping the others. The separator is a literal colon. Most test names contain spaces, so you’ll usually need to quote the argument to protect it from the shell.

Scope is specified with:

-s, --scope SCOPE

SCOPE must be one of the following; the default is standard.

• quick: Most important subset of workflow. Handy for development. Completion time: 1–2 minutes.

• standard: All tested workflow functionality and a selection of more important examples. Completion time: 5–10 minutes.

• full: All available tests, including all examples. Completion time, hot cache: 7–15 minutes; cold cache: 1–2 hours.

Additional arguments:

-b, --builder BUILDER

Image builder to use. See ch-build(1) for how the default is selected.

--dry-run

Print summary of what would be tested and then exit.

-h, --help

Print usage and then exit.

--img-dir DIR

Set unpacked images directory to DIR. In a multi-node allocation, this directory may not be shared between nodes. Default: $CH_TEST_IMGDIR if set; otherwise /var/tmp/img.

--pack-dir DIR

Set packed images directory to DIR. Default: $CH_TEST_TARDIR if set; otherwise /var/tmp/pack.

--pedantic (yes|no)

Some tests require configurations that are very specific (e.g., being a member of at least two groups) or unusual (e.g., sudo to a non-root group). If yes, then fail if the requirement is not met; if no, then skip. The default is yes for CI environments or people listed in README.md, no otherwise.

If yes and sudo seems to be available, implies --sudo.

--perm-dir DIR

Add DIR to filesystem permission fixture directories; can be specified multiple times. We recommend one such directory per mounted filesystem type whose kernel module you do not trust; e.g., you probably don’t need to test your tmpfses, but out-of-tree filesystems very likely need this.

Implies --sudo. Default: CH_TEST_PERMDIRS if set; otherwise skip the filesystem permissions tests.

--pack-fmt FMT

Use packed image format FMT (squash or tar).

--sudo

Enable things that require sudo, such as certain privilege escalation tests and creating/removing the filesystem permissions fixtures. Requires generic sudo capabilities. Note that the Docker builder uses sudo docker even without this option.

--lustre DIR

Use DIR for run-phase Lustre tests. Default: CH_TEST_LUSTREDIR if set; otherwise skip them.

The tests will create, populate, and delete a new subdirectory under DIR, leaving everything else in DIR untouched.

3.16.3. Exit status¶

Zero if all tests passed; non-zero if any failed. For setup and teardown phases, zero if everything was created or deleted correctly, non-zero otherwise.

3.16.4. Bugs¶

Bats will wait until all descendant processes finish before exiting, so if you get into a failure mode where a test sequence doesn’t clean up all its processes, ch-test will hang.

3.16.5. Examples¶

Many systems can simply use the defaults. To run the build, run, and examples phases on a single system, without the filesystem permissions tests:

$ ch-test
ch-test version 0.12

ch-run: 0.12 /usr/local/bin/ch-run
bats:   0.4.0 /usr/bin/bats
tests:  /usr/local/libexec/charliecloud/test

phase:                build run examples
scope:                standard (default)
builder:              docker (default)
use generic sudo:     no (default)
unpacked images dir:  /var/tmp/img (default)
packed images dir:    /var/tmp/tar (default)
fs permissions dirs:  skip (default)

checking namespaces ...
ok

checking builder ...
found: /usr/bin/docker 19.03.2

bats build.bats build_auto.bats build_post.bats
 ✓ documentation seems sane
 ✓ version number seems sane
[...]
All tests passed.

The next example is for a more complex setup like you might find in HPC centers:

• Non-default fixture directories.

• Non-default scope.

• Different build and run systems.

• Run the filesystem permissions tests.

Output has been omitted.

(mybox)$ ssh hpc-admin
(hpc-admin)$ ch-test mk-perm-dirs --perm-dir /scratch/$USER/perms \
                                  --perm-dir /home/$USER/perms
(hpc-admin)$ exit
(mybox)$ ch-test build --scope full
(mybox)$ scp -r /var/tmp/pack hpc:/scratch/$USER/pack
(mybox)$ ssh hpc
(hpc)$ salloc -N2
(cn001)$ export CH_TEST_TARDIR=/scratch/$USER/pack
(cn001)$ export CH_TEST_IMGDIR=/local/tmp
(cn001)$ export CH_TEST_PERMDIRS="/scratch/$USER/perms /home/$USER/perms"
(cn001)$ export CH_TEST_SCOPE=full
(cn001)$ ch-test run
(cn001)$ ch-test examples

3.17.1. Synopsis¶

$ ch-umount MOUNTDIR

3.17.2. Description¶

Unmount Charliecloud SquashFS file at target directory MOUNTDIR. Remove empty MOUNTDIR after successful unmounting.

Additional arguments:

--help

print help and exit

--version

print version and exit

3.17.3. Example¶

$ ls /var/tmp/debian
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr  WEIRD_AL_YANKOVIC
$ ch-umount /var/tmp/debian
unmounted and removed /var/tmp/debian
$ ls /var/tmp/debian
ls: cannot access /var/tmp/debian: No such file or directory