Advanced Implementation

Here is an explanation of the advanced implementation of CI/CD, using more detailed treatments and tooling. main acts the same as in root, build and test treatments are from advanced.mel file and contains CI/CD implementation.

CI/CD implementation

The build and test treatments are implemented the same way: they require to setup a runner, on which steps are distributed. Their basic implementation is given as naive working example.

Both takes the CicdDispatchEngine model, and an input trigger signal. CicdDispatchEngine and input trigger are used to require a new distributed runner CicdRunnerEngine through setupRunner as soon as it is useful to get one ready.


treatment build[cicd: CicdDispatchEngine](repository_clone_url: string, repository_clone_ref: string)
  model runner: CicdRunnerEngine()
  input trigger: Block<void>
  output data:   Stream<byte>
{
    setupRunner[
        dispatcher=cicd,
        runner=runner
    ](
        name="build",
        cpu=100, // mCPU
        memory=500, // MB
        storage=1000, // MB
        volumes=[
            // Volumes made available
            |volume("build-result", 30 /* MB */)
        ],
        containers=[
            |container("ubuntu", // Container name
                1000, // mCPU
                200, // MB
                8000, // MB
                |amd64(),
                [ // Mount points
                    |mount("build-result", "/mounted/result")
                ],
                "ubuntu:noble", // Image name
                _)
        ]
    )
    gitClone: stepOn[runner=runner](
        executor_name="ubuntu", // Use the “ubuntu” container
        environment=|wrap<Environment>(|environment(|map([/* No env variables */]), "/root", false, false)),
        commands=[
            |command("apt-get", ["update"]),
            |command("apt-get", ["install", "-y", "git"]),
            |command("git", ["config", "--global", "url.https://.insteadOf", "git://"]),
            |command("git", ["clone", "--branch", repository_clone_ref, "--depth", "1", repository_clone_url, "project"])
        ]
    )
    makeList: stepOn[runner=runner](
        executor_name="ubuntu", // Use the “ubuntu” container
        environment=|wrap<Environment>(|environment(|map([]), "/root/project", false, false)),
        commands=|raw_commands([
            "sh -c \"ls -l --almost-all | tee files-list.txt\"",
            "mv files-list.txt /mounted/result/"
        ]),
        out_filesystem="build-result",
        out_file="files-list.txt"
    )
    stopRunner[runner=runner]()
 
    Self.trigger -> setupRunner.trigger,ready -> gitClone.trigger,completed -> makeList.trigger,finished -> stopRunner.trigger
                                                                               makeList.data -------------> Self.data
}

Setup runner

setupRunner takes different parameters, the main ones are:

name, a string with which all logs will be marked;
memory, memory (in mebibytes) dedicated to the Mélodium engine;
cpu, amount of CPUs (in millicores) the Mélodium engine must have;
storage, disk space (in mebibytes) the Mélodium engine have at disposal;
volumes, array of volumes the Mélodium engine and containers can share on the runner;
containers, list of containers that are available to run steps on the runner.

Volumes are basically labels associated with size (in mebibytes), and can be mounted inside containers to share data with them. Containers are defined by the following:


|container(name, memory, cpu, storage, arch, mounts, image, pull_secret)

name being the designated name with which the container can be adressed as executor;
memory, cpu, and storage having the same meaning than for the Mélodium engine;
arch designating the hardware architecture on which the container must run;
mounts being an array of mounting points for defined volumes within the container filesystem;
image the container image to use as base for the container;
pull_secret the (optional) secret to use to pull the container image.

The Mélodium engine itself can be quite frugal in resources as work in CI/CD is usually done within containers, leaving only the orchestration and data streaming work, as such, 0.1 CPU and 100 MiB are good enough as default values.

Currently the AMD64/x86_64 |amd64() and ARM64/aarch64 |arm64() architectures are supported, and all the containers in the same runner must use the same architecture.

All mounted volumes inside containers must fit within the container storage itself. While CPU and memory values are cumulative among the main engine and containers, requiring the runner to have the sum of all the required capacities, the storage space corresponding to volumes are shared and not duplicated between containers.

Run steps

On every runner an arbitrary number of steps can be applied using the stepOn and stepOnWithInput treatments. These treatments takes a CicdRunnerEngine, on which work will be done, according to the following logic:

inputting data file (for stepOnWithInput),
processing commands,
outputting data file.

Inputting data

stepOnWithInput have two parameters setting up what data file is wrote: in_filesystem and in_file respectively tells what volume and what file name are filled with data coming from data input.

Here the file "list.txt" is wrote on filesystem volume "tested-data" and accessible on the mounting point "/mounted/data/" of executor "alpine":


makeTest: stepOnWithInput[runner=runner](
    executor_name="alpine",
    in_filesystem="tested-data",
    in_file="list.txt",
    commands=[
        |command("test", ["-s", "/mounted/data/list.txt"]),
        |command("cat", ["/mounted/data/list.txt"])
    ]
)

Processing commands

The commands parameter is a list of commands run on executor designated by executor_name within optionally-defined environment.

Commands can be specified in multiple ways:

|command() function, for a finely-detailed command and list of arguments;
|raw_command() function, for a raw string command;
|raw_commands() function, for full list of string commands.

environment can also be given, to specify additionnal environment variables or working directory. The environemnt is applied on all commands of a given step.


gitClone: stepOn[runner=runner](
    executor_name="ubuntu",
    environment=|wrap<Environment>(|environment(|map([]), "/root", false, false)),
    commands=[
        |command("apt-get", ["update"]),
        |command("apt-get", ["install", "-y", "git"]),
        |command("git", ["config", "--global", "url.https://.insteadOf", "git://"]),
        |command("git", ["clone", "--branch", repository_clone_ref, "--depth", "1", repository_clone_url, "project"])
    ]
)

Outputting data

stepOn and stepOnWithInput treatments both allows to output any data. out_filesystem and out_file specify what volume and file is streamed after step commands.

Here the file "files-list.txt" is streamed from volume "build-result" after commands are run:


makeList: stepOn[runner=runner](
    executor_name="ubuntu",
    environment=|wrap<Environment>(|environment(|map([]), "/root/project", false, false)),
    commands=|raw_commands([
        "sh -c \"ls -l --almost-all | tee files-list.txt\"",
        "mv files-list.txt /mounted/result/"
    ]),
    out_filesystem="build-result",
    out_file="files-list.txt"
)

Execute CI/CD Locally

In order to run the CI/CD program, invoke Mélodium with program parameters.


melodium run .melodium-ci/Compo.toml advanced --api_token <YOUR_RUN_API_TOKEN> --output_directory /your/output/directory --repository_clone_url <FULL_CLONE_URL>.git --repository_clone_ref <BRANCH_OR_TAG>

Where YOUR_RUN_API_TOKEN is a Cadence.CI Run Token. This is executing the CI/CD “main engine” from local computer and distributing the work on distant workers.

All the CI/CD logs are shown in terminal, and data wrote under /your/output/directory.

CI/CD Template Install the Mélodium Skill