diff --git a/docs/caching.md b/docs/caching.md new file mode 100644 index 0000000..0a84110 --- /dev/null +++ b/docs/caching.md @@ -0,0 +1,50 @@ +# Caching + +Nix GitLab CI supports several caching mechanisms to speed up your pipelines. + +## GitLab Runner Cache + +The runner cache strategy copies the new store paths into a directory `.nix-cache`, +which is then saved in the regular GitLab cache (technically runner cache). +It's also configured as a substituter automatically. + +To enable, set the cache strategy to `runner`. + +Configure it using these environment variables: + +- `RUNNER_CACHE`: path to the runner cache (default `.nix-cache`) + +!!! warning + + This is very inefficient and should probably only be used for very very small + dependency counts. Otherwise it takes an eternity to save to cache. + +## Cachix + +Cachix is a hosted binary cache service that can significantly speed up Nix +builds by sharing build results. + +To enable, set the cache strategy to `attic`. + +Configure it using these environment variables: + +- `CACHIX_CACHE`: name of the cache to use +- (`CACHIX_AUTH_TOKEN`): cachix client itself uses this for authentication + +!!! warning + + Cachix has not been tested. Feedback is appreciated :) + +## Attic (Self-Hosted Cache) + +Attic is a self-hosted, deduplicating binary cache. It's a great option if you +want more control over your caching infrastructure and to have the cache closer +to your runners. + +To enable, set the cache strategy to `attic`. + +Configure it using these environment variables: + +- `ATTIC_SERVER`: URL of the server +- `ATTIC_CACHE`: name of the cache to use +- `ATTIC_TOKEN`: auth token from the attic server diff --git a/docs/cicd_component.md b/docs/cicd_component.md new file mode 100644 index 0000000..2fe2ae2 --- /dev/null +++ b/docs/cicd_component.md @@ -0,0 +1,41 @@ +# CI/CD Component + +The CI/CD Component has some inputs which configure defaults for Nix GitLab CI. + +## `version` + +- Type: `string` + +Which version of the Nix CI image to use. Using a tag/version is recommended. +Will not do anything if a custom image is specified using `NIX_CI_IMAGE`. + +## `cache_strategy` + +- Type: `string` +- Default: `"auto"` +- Options: `auto` | `none` | `runner` | `cachix` | `attic` + +Sets the default caching strategy. + +- `auto`: dynamically selects the best strategy for every job based on env variables +- `none`: disables caching +- `runner`, `cachix` & `attic`: forces every job to use this strategy + +Can be overridden by `NIX_CI_CACHE_STRATEGY`, see [Environment Variables](./environment_variables.md). + +## `cache_files` + +- Type: `array` (of strings) +- Default: `["flake.nix", "flake.lock"]` + +Files to use as the cache key for the generated pipeline yaml. +If you use a file like `ci.nix` to define CI, add that here for example. +This makes sure that changes to your Nix CI configuration will invalidate the cache, +otherwise an old pipeline yaml might be used. + +!!! warning + + The value of this is used in `cache:key:files`, which currently only supports + a max of 2 entries. So use something like `["flake.*", "ci.nix"]` to match + `flake.lock`, `flake.nix` and `ci.nix`. + See [gitlab-org/gitlab#301161](https://gitlab.com/gitlab-org/gitlab/-/issues/301161) diff --git a/docs/environment_variables.md b/docs/environment_variables.md new file mode 100644 index 0000000..56c0ba5 --- /dev/null +++ b/docs/environment_variables.md @@ -0,0 +1,107 @@ +# Environment Variables + +Nix GitLab CI is mostly controlled using environment variables. +This page outlines all the variables and their use case. + +## `NIX_CI_IMAGE` + +| | | +| ----------- | -------------------------------------------------------------------------- | +| Default | `registry.gitlab.com/technofab/nix-gitlab-ci/nix-ci@$[[ inputs.version ]]` | +| Description | Image to use for the jobs | + +## `NIX_CI_PIPELINE_NAME` + +| | | +| ----------- | ------------------------------------------------- | +| Default | N/A | +| Description | Explicitly request a pipeline to be built and ran | +| See also | [Multi Pipeline](./multi_pipeline.md) | + +## `NIX_CI_DEFAULT_SOURCES` + +| | | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------- | +| Default | `.*` | +| Description | Regex to match `$CI_PIPELINE_SOURCE` against. If it matches, the `default` pipeline will be ran, otherwise `$CI_PIPELINE_SOURCE` | +| See also | [Multi Pipeline](./multi_pipeline.md) | + +## `NIX_CI_FORCE_BUILD` + +| | | +| ----------- | -------------------------------------------------------------------------------------- | +| Default | N/A | +| Description | Set to any non-empty value to force the `nix-ci:build` job to freshly build the config | +| See also | [Caching](./caching.md) | + +## `NIX_CI_DISABLE_CACHE` + +| | | +| ----------- | ------------------------------------------------------ | +| Default | N/A | +| Description | Set to any non-empty value to disable caching for jobs | +| See also | [Caching](./caching.md) | + +## `NIX_CI_CACHE_STRATEGY` + +| | | +| ----------- | --------------------------------------------------------------------------------- | +| Default | `$[[ inputs.cache_strategy ]]` -> defaults to `auto` | +| Description | Caching strategy to use. `auto` will select the strategy based on runner settings | +| See also | [Caching](./caching.md) | + +## `NIX_CI_RUNNER_CACHE_STRATEGY` + +| | | +| ----------- | ---------------------------------------------------------------- | +| Default | N/A | +| Description | Every runner can set it's own preferred cache strategy with this | +| See also | [Caching](./caching.md) | + +## `NIX_CI_DEFAULT_CACHE_STRATEGY` + +| | | +| ----------- | ------------------------------------------------------------------------------------------------- | +| Default | `none` | +| Description | If no runner cache strategy is set and the main strategy is set to auto, this will be the default | +| See also | [Caching](./caching.md) | + +## `RUNNER_CACHE` + +| | | +| ----------- | -------------------------------------- | +| Default | `.nix-cache` | +| Description | Path to directory for the runner cache | +| See also | [Caching](./caching.md) | + +## `CACHIX_CACHE` + +| | | +| ----------- | ------------------------------- | +| Default | N/A | +| Description | Name of the cachix cache to use | +| See also | [Caching](./caching.md) | + +## `ATTIC_CACHE` + +| | | +| ----------- | ------------------------------ | +| Default | N/A | +| Description | Name of the attic cache to use | +| See also | [Caching](./caching.md) | + +## `ATTIC_SERVER` + +| | | +| ----------- | ----------------------- | +| Default | N/A | +| Description | URL of the attic server | +| See also | [Caching](./caching.md) | + +## `ATTIC_TOKEN` + +| | | +| ----------- | ------------------------------- | +| Default | N/A | +| Description | API token from the attic server | +| See also | [Caching](./caching.md) | diff --git a/docs/examples.md b/docs/examples.md new file mode 100644 index 0000000..336a55a --- /dev/null +++ b/docs/examples.md @@ -0,0 +1,13 @@ +# Example Configs + +- [TECHNOFAB/nix-gitlab-ci](https://gitlab.com/TECHNOFAB/nix-gitlab-ci) +- [TECHNOFAB/nixlets](https://gitlab.com/TECHNOFAB/nixlets) +- [TECHNOFAB/nixible](https://gitlab.com/TECHNOFAB/nixible) +- [TECHNOFAB/nixmkdocs](https://gitlab.com/TECHNOFAB/nixmkdocs) +- [TECHNOFAB/tofunix](https://gitlab.com/TECHNOFAB/tofunix) +- [TECHNOFAB/nixtest](https://gitlab.com/TECHNOFAB/nixtest) + +!!! note + + Feel free to edit this page and add your project if you're using + Nix GitLab CI :) diff --git a/docs/index.md b/docs/index.md index 07a2ade..214fbf3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,2 +1,11 @@ -# Nix-GitLab-CI +# Nix GitLab CI +This project provides a Nix flake module that allows you to generate your `.gitlab-ci.yml` file directly from your Nix configuration. + +## Features + +- **Reproducibility:** Leverage Nix's strength in creating reproducible environments for your CI jobs. +- **Easy Dependency Management:** Easily include any package available in Nixpkgs or your own defined packages within your CI jobs using Nix. +- **Modularity:** Define and manage your CI configurations in a structured and modular way using Nix modules, making it easier to share and reuse CI logic across multiple projects. + +This documentation will guide you through setting up and using Nix GitLab CI for your projects. diff --git a/docs/kubernetes_runner.md b/docs/kubernetes_runner.md new file mode 100644 index 0000000..4c20421 --- /dev/null +++ b/docs/kubernetes_runner.md @@ -0,0 +1,38 @@ +# Kubernetes Runner Setup + +Using the GitLab Kubernetes runner allows your CI jobs to run as pods in a Kubernetes cluster. +Nix GitLab CI can be integrated with this setup, and using advanced configuration options like +`pod_spec` makes it easy to add runner specific caching. + +Using this Runner configuration ... + +```toml +[[runners.kubernetes.pod_spec]] +name = "nix-ci-cache-secrets" +patch = ''' + containers: + - name: build + envFrom: + - secretRef: + name: nix-ci-cache-env +''' +``` + +... and a secret containing ... + +```yaml +NIX_CI_RUNNER_CACHE_STRATEGY: attic +ATTIC_SERVER: # example: http://atticd..svc.cluster.local:8080 +ATTIC_CACHE: ci # name however you want, just needs to exist +ATTIC_TOKEN: +``` + +... makes your jobs automatically cache their Nix store paths to the in-cluster +attic when running with this runner. + +Other runners could use cachix or no cache, you get the idea ;P + +!!! note + + This of course works with any executor where you can set environment + variables. This is just an example how to do it in Kubernetes easily. diff --git a/docs/multi_pipeline.md b/docs/multi_pipeline.md new file mode 100644 index 0000000..d8dbfe6 --- /dev/null +++ b/docs/multi_pipeline.md @@ -0,0 +1,31 @@ +# Multiple Pipelines + +With V2, Nix GitLab CI can generate different pipelines, depending on the +pipeline source (`$CI_PIPELINE_SOURCE`). + +By default, no matter which source, the `default` pipeline is built and ran. +`$NIX_CI_PIPELINE` can override that, eg. when manually triggering a run. +To configure which source should be 1-to-1 translated to a pipeline with the +same name, set `$NIX_CI_DEFAULT_SOURCES` to a regex which explicitly does not +match these sources. Or set it to an impossible to match regex, then it will +always run the pipeline named after `$CI_PIPELINE_SOURCE`. + +## Example 1: always run default + +If you only have a single pipeline, you just have to call it `default`. +Everything else works out of the box. + +## Example 2: default and merge_request_event + +If you want the source `merge_request_event` to trigger a different pipeline, +name it like that and set `$NIX_CI_DEFAULT_SOURCES` to `^(merge_request_event)$`. +Now a merge request will run this pipeline, while everything else runs `default`. + +## Example 3: default, push and web + +Set `$NIX_CI_DEFAULT_SOURCES` to `^(push|web)$`. + +## Example 4: always run the specific pipelines, never default + +Set `$NIX_CI_DEFAULT_SOURCES` to any regex that never matches the sources, +like `a\A` or `nothing`. diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 0000000..c1f85df --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,81 @@ +# Setup + +To integrate Nix GitLab CI into your project, you need to make two main changes: + +1. Add the `nix-gitlab-ci` flake module to your `flake.nix`. +1. Include the necessary component in your `.gitlab-ci.yml`. + +## Adding to `flake.nix` + +In your project's `flake.nix`, add `nix-gitlab-ci` as an input and import its +flake module within your `flake-parts` configuration. + +```nix title="flake.nix" +{ + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; # Or your preferred nixpkgs branch/version + flake-parts.url = "github:hercules-ci/flake-parts"; + + # Add nix-gitlab-ci as an input + # recommendation: pin to a specific release/version + nix-gitlab-ci.url = "gitlab:TECHNOFAB/nix-gitlab-ci/?dir=lib"; + }; + + outputs = { nixpkgs, flake-parts, ... }@inputs: + flake-parts.lib.mkFlake { inherit inputs; } { + imports = [ + # Import the nix-gitlab-ci flake module + inputs.nix-gitlab-ci.flakeModule + ]; + + systems = [ + "x86_64-linux" + "aarch64-linux" + # Add other systems you need + ]; + + perSystem = { pkgs, ... }: { + # define your CI pipelines + # ci.pipelines."merge_request_event" = { ... }; + }; + }; +} +``` + +Replace `` with the specific version or commit hash of `nix-gitlab-ci` +you wish to use. Pinning to a specific version is highly recommended for +reproducibility and compatibility. + +!!! warning + + While the flake input is locked through `flake.lock`, the CI/CD component + will always use the latest commit of the reference. This means that by using + a branch like `main` as version for both, the CI/CD component will always use + the latest commit while your flake uses a fixed one. + This could result in drift between both, potentially breaking stuff. + +## Including in `.gitlab-ci.yml` + +Your `.gitlab-ci.yml` file will be minimal. Its primary role is to include the +`nix-gitlab-ci` component, which will then generate the full CI configuration +based on your Nix code. + +```yaml title=".gitlab-ci.yml" +include: + - component: gitlab.com/TECHNOFAB/nix-gitlab-ci/nix-gitlab-ci@ + inputs: + # This input sets the Docker image tag used for the CI jobs. + # Use the same version as you pinned in your flake.nix for consistency. + version: +``` + +Again, ensure `` matches the version used in your `flake.nix`. +This component includes a job (`build:nix-ci`) that will evaluate your Nix +configuration and generate the `.gitlab-ci.yml` used for the pipeline run. + +!!! note + + Since V3 [Soonix](https://soonix.projects.tf) is supported, this can + automatically generate the `.gitlab-ci.yml` for you, with the version + automatically following the flake. + See [Soonix Integration](./soonix.md) for more. diff --git a/docs/soonix.md b/docs/soonix.md new file mode 100644 index 0000000..b333ab6 --- /dev/null +++ b/docs/soonix.md @@ -0,0 +1,20 @@ +# Soonix Integration + +[Soonix](https://soonix.projects.tf) can be used to automatically generate the +`.gitlab-ci.yml` for you. +This will by default include the CI/CD component with the same version as the +flake (using the `VERSION` file in this repo). + +You can specify some options to configure this, like changing the component URL +or adding extra data to your `.gitlab-ci.yml` (like this repo does to bootstrap +Nix-GitLab-CI). See [Options](./options.md#configsoonix) for all the config options. + +You can use it like this: + +```nix +let + ci = cilib.mkCI { ... }; +in { + soonix.hooks."ci" = ci.soonix; +} +``` diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..44c0bf7 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,73 @@ +# Usage + +## Usage (with flake-parts) + +```nix +# flake.nix +{ + ... + inputs.nix-gitlab-ci.url = "gitlab:TECHNOFAB/nix-gitlab-ci/?dir=lib"; # recommendation: pin to the latest release/version + + outputs = {...}: flake-parts.lib.mkFlake {...} { + imports = [ + inputs.nix-gitlab-ci.flakeModule + ]; + ... + + perSystem = {pkgs, ...}: { + ci = { + config = { + # configure Nix-GitLab-CI here, see docs for options + }; + pipelines."default" = { + stages = ["test"]; + jobs = { + "test" = { + stage = "test"; + nix.deps = [pkgs.unixtools.ping]; + script = [ + "ping -c 5 8.8.8.8" + ]; + }; + }; + }; + # runs on a merge request for example + pipelines."merge_request_event" = { + stages = ["some_stage"]; + jobs = { ... }; + }; + }; + ... + } + } +} +``` + +Now either use this in your .gitlab-ci.yml or setup Soonix to auto generate this +file for you with the right version (see the [docs][docs-soonix] for more). + +```yaml +# .gitlab-ci.yml +include: + - component: gitlab.com/TECHNOFAB/nix-gitlab-ci/nix-gitlab-ci@ # recommendation: pin to the latest release/version (don't use "main" etc.) + inputs: + version: # docker image tag, use the same version as a above +``` + +## Usage (directly) + +```nix +let + cilib = inputs.nix-gitlab-ci.lib {inherit pkgs;}; +in + cilib.mkCI { + config = ...; + pipelines."default" = ...; + }; + # exposes `soonix` for the soonix hook and `packages` which contain the configs, jobs etc. +``` + +______________________________________________________________________ + +Since V2 multiple pipelines are supported. +See [Multiple Pipelines](./multi_pipeline.md) for more. diff --git a/docs/utilities.md b/docs/utilities.md new file mode 100644 index 0000000..b82091d --- /dev/null +++ b/docs/utilities.md @@ -0,0 +1,64 @@ +# Utilities + +Nix GitLab CI provides a couple of utilities to help with development and +debugging. + +## Disabling caching temporarily + +Nix GitLab CI often utilizes caching mechanisms to speed up your pipelines +(see [Caching](./caching.md)). +However, there might be situations where you need to temporarily disable these +caches for a specific pipeline run, for example, to debug a caching issue or +ensure a clean build. + +To disable most of the provided caches for a pipeline, set the environment +variable `NIX_CI_DISABLE_CACHE` to any non-empty value (e.g., `yes`, `true`, `1`) +when triggering the pipeline in the GitLab UI or via the API. + +## Forcing a rebuild of the CI pipeline definition + +The job responsible for generating the `.gitlab-ci.yml` from your Nix code +(`build:nix-ci`) might itself be cached. If you've made changes to your Nix CI +configuration and the pipeline doesn't seem to pick them up, the cached job +definition might be the reason. + +You should first double check if all the Nix files you defined the CI config in +are specified in the `cache_files` CI/CD-component input +(see [CI/CD Component](./cicd_component.md) for more). + +To force this specific job to rebuild and re-evaluate your Nix configuration, +set the environment variable `NIX_CI_FORCE_BUILD` when triggering the pipeline. + +## Running jobs locally + +One of the benefits of defining your CI jobs with Nix is the ability to run them +locally in an environment that closely mirrors the CI environment. This can +significantly speed up debugging and development. + +You can run the script of any defined job locally using the `nix run` command. +The syntax is: + +```sh +nix run .#gitlab-ci:pipeline::job: +``` + +Replace `` with the name of the pipeline the job belongs to +(e.g., `default` for jobs defined under the `ci` attribute) and `` +with the name of the job you want to run. + +This command will set up the environment with the specified `nix.deps` and +execute the job's `script`. + +There is also an attribute `.#gitlab-ci:pipeline::job-deps:`. +Building this derivation will generate a shell script which exports the required +environment variables for the job, such as the `PATH` including all dependencies +and any custom environment variables that contain store paths (ensuring they are +correctly resolved across different architectures). + +You can use this to inspect the environment that would be set up for a job without +running the full script. + +## Viewing generated config in YAML + +Since the GitLab CI config is generated simply using JSON, it's hard to read and +debug. For debugging V3 now adds another package `gitlab-ci:pipeline::pretty`. diff --git a/nix/repo/docs.nix b/nix/repo/docs.nix index dad03e1..bb05f74 100644 --- a/nix/repo/docs.nix +++ b/nix/repo/docs.nix @@ -47,6 +47,16 @@ in }; nav = [ {"Introduction" = "index.md";} + {"Setup" = "setup.md";} + {"Usage" = "usage.md";} + {"CI/CD Component" = "cicd_component.md";} + {"Environment Variables" = "environment_variables.md";} + {"Caching" = "caching.md";} + {"Multiple Pipelines" = "multi_pipeline.md";} + {"Soonix Integration" = "soonix.md";} + {"Utilities" = "utilities.md";} + {"Kubernetes Runner Example" = "kubernetes_runner.md";} + {"Example Configs" = "examples.md";} {"Options" = "options.md";} ]; markdown_extensions = [