blog::weyl.io

Task and Taskfiles

chris@weyl.io (Chris Weyl) — Sun, 28 Dec 2025 00:00:00 +0000

If you’ve been around a project for any length of time, you’re familiar with the need for automation – specifically, something to keep track of all the little tasks that need to be done over and over again. These tasks are typically at least somewhat deterministic, don’t vary from run to run, and are run by more than one person.

The classic solution here is to use make and a Makefile. For older codebases where you’re only worried about building C or C++ code, this is very acceptable – it’s what make was designed for, after all. make is nearly universally available on Unix-like systems, and it’s relatively simple to use for basic tasks. The challenge comes in when you need to execute non-deterministic tasks; things like running tests, linting code, bringing test environments up and down, and so on. make can handle these tasks but it’s not always the best tool for the job. ¹

`task`, a modern job-runner

Note

This is likely to be the first in an irregular series on task.

Roughly 8 months ago, a colleague and friend introduced me to the task program and the Taskfiles it uses. task is a general purpose task runner; a system designed to trivially enable writing, documenting, and running tasks. More YAML, yes, but that’s hardly unusual these days, but more importantly – no arcane incantations, no mystical recipes. Just a well-designed YAML schema and straight-forward embedded bash.

jobs, targets and dependencies

Both make and task define each job as a “target” (though task prefers to call them… “tasks”). Both allow for the chaining of tasks, either implicitly or explicitly.

I’m going to set aside how make dependencies work for the moment – there being more books, papers, and articles on that topic than I’m willing to even hazard a guess at enumerating – and take a look at how task handles it.

Generally speaking, task dependencies can be broken out into:

Does this task need to be run?
- Did the source change; or
- Does this generic test fail?
This other task needs to be run before/after/during.

The former of which can be thought of as “self dependency”, while the latter is more typically what we think of as a dependency.

I’m not going to regurgitate the documentation here, as, well, if you’re reading this you’re not likely to appreciate that very much, but I do want to give a couple examples that illustrate how powerful this approach is.

Both of these forms may be combined or used in dependent tasks to support useful workflows. For example, let’s say you have a golang project, one that requires certain services to be spun up beforehand before you can attempt to run/test the built binary. Note that we have a couple different dependency types here: does the binary need to be (re)built? Is the test service up?

---
version: "3"

tasks:

  # executes "cmds" only if task determines that "thingie" needs to be
  # rebuilt from the sources listed
  build:
    desc: 'Build it!'
    sources:
      - '**/*.go'
      - go.mod
      - go.sum
    generates:
      - thingie
    cmds:
      - cmd: go build -o thingie

  service-up:
    desc: Bring up a service we need for... reasons
    status:
      - systemctl is-active ollama.service
    cmds:
      - cmd: systemctl start ollama.service
      - task: service-pull

  service-pull:
    desc: Something we need to do and may want to do independently
    deps:
      - service-up
    vars:
      MODEL: 'hf.co/ibm-granite/granite-4.0-h-micro-gguf:latest'
    status:
      - 'ollama list | grep -q {{.MODEL}}'
    cmds:
      - cmd: 'ollama pull {{.MODEL}}'

  run:
    desc: Run the test/app/whatever
    deps:
      - build
      - service-up
    cmds:
      - cmd: ./thingie ...

In the above, note how:

build will only execute if any of the sources change (much as make does).
service-up only attempts to start the service if it isn’t already started. (Agreed, with systemd this isn’t a huge savings, but imagine if the service is running inside a container launched by podman, etc)
service-up declares a “run-time” dependency on service-pull.
service-pull only attempts to pull the model if it is not already present on the system.

The net result of this is a set of tasks flexible enough that we’re able to run each step individually (if we so chose), but with expressive enough dependency information that we’re not going to have tasks failing due to a dependency not being built / started / fetched / etc.

Enjoy!

For an excellent example of both the power and the pitfalls of using make for non-build tasks, see the Makefile generated by the kubebuilder project. ↩︎

Linting Thoughts

chris@weyl.io (Chris Weyl) — Tue, 29 Oct 2024 00:00:00 +0000

Linting is an one of those things that can be weirdly controversial. I say weirdly as I cannot quite understand the objections to it – it’s not like we’re all perfect typists, after all. Generally the objections range from “My editor does it for me” to “I forget to run them before pushing!” to my favorite, “They always fail in CI!”

None of those are legitimate reasons to avoid linting. We call it “linting” because it’s like the lint on your clothes: small, annoying, seems to spontaneously self-incarnate, and is trivial to remove. Linting is one of the things computers excel at.

What is Linting?

Simply put: Linting is any automated process that enforces a deterministic, agreed-on set of coding, format, style, or other technical standards.

Tip

The content you’re linting can have a great impact on the tools you use to lint, and what you can lint. For example, a statically typed language like Go can be linted in ways that a dynamically typed language like Perl cannot.

Linting is:

Spell-checking
Validating data file formats
Validating end-of-file and other platform issues
Static code checkers / analysis
Code style checks
Ensuring license headers are applied everywhere
Other fully automated checks for formatting, style, etc, that can be run without human intervention

Linting is not:

Code reviews
Manual testing
User acceptance testing
Anything requiring human intervention

Linting may be:

Running unit tests
Building binaries (to validate it actually builds, not deployment)

…that’s really up to you.

Linters may also suggest fixes that can then be evaluated and applied, if appropriate. This is not necessary, but is very convenient.

How do we lint?

All effective linting implementations I’ve seen have the following characteristics:

They use a linting framework
They can be run locally
They are run and enforced in CI/CD

Linting Frameworks

It’s certainly possible to “roll your own” approach here, but why? While there aren’t a huge number of linting frameworks, all you really need is one good, general-purpose one. (Which is why there aren’t a huge number of them.)

Pre-commit is an excellent example of a linting framework. It is Open Source, is widely used, has a large number of pre-built hooks, and is trivially extensible. Additionally, it is well-documented, well-supported, and actively maintained. It’s name implies a certain workflow: it is designed to be run before each commit. However, this is not the case. It is a general-purpose linting framework with built-in support for Git hooks, but it can be run at any time.

pre-commit/pre-commit Public

A framework for managing and maintaining multi-language pre-commit hooks.

Python 15k 927

pre-commit/pre-commit-hooks Public

Some out-of-the-box hooks for pre-commit

Python 6.4k 777

Pre-commit linters (“hooks”)

Note

Pre-commit “hooks” are the individual linters that are run by the pre-commit framework; they should not be confused with Git hooks.

I’ve included the base, general pre-commit-hooks repository above. Note that this is not the only repository of hooks available – there are many others, including some that are specific to a particular language or tool.

Here’s a few examples of pre-commit hooks that I’ve found useful:

TekWizely/pre-commit-golang Public

Pre-commit hooks for Golang with support for monorepos, the ability to pass arguments and environment variables to all hooks, and the ability to invoke custom go tools.

Shell 357 41

adrienverge/yamllint Public

A linter for YAML files.

Python 3.3k 307

syntaqx/git-hooks Public

A collection of git hooks for use with pre-commit

Shell 34 16

hadolint/hadolint Public

Dockerfile linter, validate inline bash, written in Haskell

Haskell 12k 487

python-jsonschema/check-jsonschema Public

A CLI and set of pre-commit hooks for jsonschema validation with built-in support for GitHub Workflows, Renovate, Azure Pipelines, and more!

Python 299 57

IBM/detect-secrets Public

Fork from Yelp/detect-secrets

An enterprise friendly way of detecting and preventing secrets in code.

Python 84 53

Lucas-C/pre-commit-hooks Public

git pre-commit hooks

Python 151 53

terraform-docs/terraform-docs Public

Generate documentation from Terraform modules in various output formats

Go 4.7k 587

Pre-commit hooks are also surprisingly easy to create.

When do we lint?

Tip

Generally speaking, at a minimum one should lint before submitting a changeset for review (that is, creating a pull request). This is a minimum level of courtesy to your peers.

I always lint before pushing – setting up a pre-push hook tends to make this impossible to forget. Others have other preferences, and that’s fine: running the linters in CI/CD ensures that they cannot be forgotten. (…and that the reviewer does not need to enforce them.)

A successful linting workflow tends to run like this:

Local
- Lint at some point before pushing
CI
- Lint as the first pipeline job on every push
- Fail the entire pipeline early if linting fails

Linting Locally

Running locally is both critical and not essential. It’s critical in that if you push a change that fails linting, your changeset will fail CI. Similarly, it is not essential as you can always push a change, wait for CI to fail and then fix it. Your call.

As Pre-commit’s name implies, it was originally designed to be run before you make a commit – e.g. as a pre-commit hook. However, with modern Git workflows this can be quite time-consuming. (Imagine having to wait for your linters to run every time you create a fixup! commit – it gets old fast.) A happy balance between “every time” and “never” is to run it before you push, typically as a pre-push hook.

pre-commit install --install-hooks --hook-type pre-push

Regardless of the tool used to lint, leveraging a pre-push hook helps ensure that you catch any linting errors before anyone else sees them.

Lint in CI

Warning

I cannot emphasize this enough: Enforce your lint rules in CI. People didn’t stop littering because it was the right thing to do. They stopped because they were fined for it.

It is imperative that you lint in CI. Not linting in CI defeats the whole purpose of linting. We lint to catch the (typically) trivial mistakes we may make and we run CI to automate testing for mistakes. If we believed every changeset to be perfect, we wouldn’t have CI in the first place.

If you find your CI/CD pipelines failing frequently due to linting errors, you may want to examine some of your own practices. (Like, say, setting up a pre-push hook?)

If you do not enforce your linting rules in CI, you’re effectively forcing other people to do it for you. This is not only rude, but it’s also a waste of time and resources.

What do we lint?

Tip

Linting is not just for code. It can be used to validate data files, check your spelling, and even ensure that your documentation is up-to-date.

Simple answer? We lint everything we can.

If you can lint it, you should. Linting is much like brushing your teeth or washing your hands – it’s a simple, effective way to prevent problems. Also, it’s really gross, obvious to even the most casual of observers, and just downright lazy¹ if you don’t.

If you’re using pre-commit, there are a huge number of pre-built hooks you can use as well as a large number of third-party hooks. It is also easy to write your own, should the need arise.

Important things to lint include:

platform-specific line and file endings are consistent
data files (e.g. JSON, YAML, TOML) are not malformed
code is legal and styled consistently
the project is buildable (e.g. it is “complete” and coherent)
secrets have not been committed

You have to start somewhere

Tip

Take the Boy Scout approach to linting: Every time you find lint, leave the project a little cleaner (and the linter a little smarter) than you found it.

It’s not often to possible to know all the things you need to lint from the start. Much as with testing, you’ll find additional things to lint for as your project moves forward – especially as additional people, perhaps using different operating systems, editors, or tooling, begin to contribute. I’ve found it useful to treat linting like other testing processes:

Lint (test) all the obvious things right off the bat; and
Add additional linting as you find issues.

For example, it’s fairly obvious in a Go project that running gofmt is a good idea. It is perhaps less obvious that you should lint your Markdown README.md to ensure it renders correctly – until someone pushes one that does not. Similarly, it’s fairly obvious that one should lint YAML to ensure it’s valid, but less obvious that you should enforce a consistent style of indenting lists.

Finally…

Linting is a simple, effective way to ensure that your project is clean and consistent. We created computers to help us with automated, repetitive, deterministic tasks like this – tasks that humans have a tendency to forget or overlook. Automated, enforced linting is a simple and effective way to instantly raise and ensure the overall hygiene and quality of any project.

Just do it! Linting is only ever a big deal when it is not done.

I mean, come on. This isn’t the good type of lazy – this is the bad type that imperils your Hubris. ↩︎

terraform-provider-gitlabci: Register GitLab CI Runners

chris@weyl.io (Chris Weyl) — Fri, 14 Jan 2022 00:00:00 +0000

I’m a huge fan of terraform, so when I needed to build out cloud infrastructure for GitLab CI/CD it was the first thing I reached for. The native terraform-provider-gitlab was very useful, but left out one critical detail: it was not possible to register a runner.

Ouch. 💢

This left a rather annoyingly awkward gap in my terraform configurations, as I’d need to provision a runner token outside of terraform. I messed around with this, coming up with a couple… interesting approaches, but ultimately I realized that the only proper solution to this (read: that wasn’t just a giant hack) would require writing a provider.

`terraform-provider-gitlabci`

Recently I’ve had some free time, so I cleaned it up a bit and published it.

A quick example

Documentation and the like can be found over at the terraform registry, but here’s a quick example with only a minimum of hand-wavey:

terraform {
  required_providers {
    gitlabci = {
      source = "registry.terraform.io/rsrchboy/gitlabci"
    }
    gitlab = {
      source = "registry.terraform.io/gitlabhq/gitlab"
    }
  }
}

provider "gitlabci" { }
provider "gitlab"   { }

data "gitlab_project" "this" {
  id = "rsrchboy/terraform-provider-gitlabci"
}

resource "gitlabci_runner_token" "this" {
  registration_token = data.gitlab_project.this.runners_token
  locked             = true
  tags = [
    "jinx",
    "powder",
    "cupcake",
  ]
}

output "token" {
  sensitive = true
  value     = gitlabci_runner_token.this.token
}

Note how, using both the gitlab and gitlabci providers we can now register GitLab runners. The example shows us using a registration token obtained from a project data source, but terraform-provider-gitlabci doesn’t care if it’s a project, group, or even instance registration token. Additionally, while the gitlab provider does require API access, the gitlabci provider only requires a valid registration token.

Enjoy!

Docker container network interface ordering

chris@weyl.io (Chris Weyl) — Sat, 08 Jan 2022 00:00:00 +0000

Lately I’ve found myself routinely attaching Docker-based containers to multiple networks. This has led to a couple… interesting surprises.

It doesn’t seem to be well documented (AFAIK), so here’s what I’ve learned. This isn’t a big serious how-to post, just something that irked me. 😄

Tooling (a little background)

Observed behavior

As this is observed behavior rather than documented (AFAICT, at any rate), it may change without warning.

Oh well.

For the purposes of this article, let’s say we’re using the following tools:

Network name determines ordering

What’s the actual network name?

When using Compose, there are really two network “names” for each network.

The name specified in the Compose configuration; and
The name Compose uses when creating the Docker network.

In a compose configuration, we always use #1 to refer to a network; however behind the scenes this will be different from the name of the network as known to Docker.

Networks are attached in alphabetical order. Internal networks are attached last, though still in alphabetical order.

Networks appear to be attached in alphabetical order, further segregated by the internal status. Not-internal networks are attached to the container before internal networks.

Let’s say we have a container with three networks attached:

version: "3.9"

services:
  container1:
  image: docker.io/containous/whoami
    networks:
      backend: {}
      frontend: {}
      apples: {}
      oranges: {}

networks:
  backend: {}
  frontend: {}
  apples:
    internal: true
  oranges:
    name: lemons

Let’s say this has a project name of services. When container1 is launched, it will see a number of interfaces:

Interface	Network (Compose)	Network (Docker)
`eth0`	`oranges`	`lemons`
`eth1`	`backend`	`services_backend`
`eth2`	`frontend`	`services_frontend`
`eth3`	`apples`	`services_apples`

Why? apples is internal, so it ends up at the end of the list. oranges has an actual name of lemons, so it ends up before the other networks with a services_ prefix.

Default route is always through `eth0`

If you want a specific network to be the default route, you’re going to need to make sure it’s first – alphabetically.

Routing and macvlan interfaces

This is more of a corner case. If you’re attaching containers directly to a vlan/network using, say, macvlan, and that network ends up being your default route (app_vlan, anyone?), then all traffic for networks you’re not directly connected to will be routed over this vlan.

This may or may not matter, but if you have the (reasonable) expectation that non-local traffic will route through and be masqueraded by the host, this can be surprising.

Using a Metadata Proxy to Limit AWS/IAM Access with GitLab CI

chris@weyl.io (Chris Weyl) — Mon, 05 Oct 2020 00:00:00 +0000

The gitlab-runner agent is very flexible, with multiple executors to handle most situations. Similarly, AWS IAM allows one to use “instance profiles” with EC2 instances, obviating the need for static, long-lived credentials. In the situation where one is running gitlab-runner on an EC2 instance, this presents us with a couple interesting challenges – and opportunities.

How does one prevent CI jobs from being able to obtain credentials against the instance’s profile role?
How does one allow certain CI jobs to assume credentials through the metadata service without allowing all CI jobs to assume those credentials?

Criteria

Relevant Configuration

This isn’t going to be a comprehensive, step-by-step guide that can be followed without any external knowledge or resources. Rather, we’re going to focus on what one needs to know in order to implement this solution, however you’re currently provisioning CI agents.

For our purposes, we want:

The gitlab-runner agent to run on an EC2 instance, with one or more runners configured.¹
All configured runners should be using the Docker executor.
Jobs to run, by default, without access to the EC2 instance’s profile credentials.
Certain jobs to assume a specific role transparently through the EC2 metadata service by virtue of what runner picks them up.
Reasonable security:
- Jobs can’t just specify an arbitrary role to assume
- No hardcoded, static, or long-lived credentials

Only short-term, transient credentials

It’s worth emphasizing this: no hardcoded, static, or long-lived credentials. Sure, it’s easy to generate an IAM user and plunk its keys in (hopefully) protected environment variables, but then you have to worry about key rotation, audits, etc, in the way one doesn’t with transient credentials.

Executor implies methodology

For our purposes, we’re going to solution this using the agent’s docker executor. Other executors will have different solutions (e.g. kubernetes has tools like kiam).

However, for fun let’s cheat a bit and do a quick-and-fuzzy run-through of a couple of the other executors.

docker+machine executor

This is largely like the plain docker executor, except that as EC2 instances will be spun up to handle jobs you can take a detour around anything complex by simply telling the agent to associate specific instance profiles with those new instances, e.g.:

[[runners]]
  [runners.machine]
    MachineOptions = [
        "amazonEC2-iam-instance-profile=everything-except-the-thing",
        ...,
      ]

The instance running the gitlab-runner agent does not need to be associated with the same profile – but the agent does need to be able to EC2:AssociateIamInstanceProfile and iam:PassRole the relevant resources.

The downside is that you’ll have to have multiple runners configured if you want to be able to allow different jobs to assume different roles.

kubernetes executor

The kubernetes executor is going to be a bit trickier, and, as ever, TMTOWTDI[^tmtowtdi]. Depending on what you’re doing, any of the following might work for you:

Launch nodes with the different profiles and use constraints to pick and choose which job pods end up running on them.
Use a solution like kiam.
…

Brute force

Ever a popular option, you can just brute-force block container (job) access to the EC2 metadata service by firewalling it off, e.g.:

iptables -t nat -I PREROUTING \
    --destination 169.254.169.254 --protocol tcp --dport 80 \
    -i docker+ -j REJECT

If you just want to block all access from jobs, this is a good way to do it.

This approach is contraindicated if you want to be able to allow some containers to access the metadata service, or to allow them to retrieve credentials of some (semi) arbitrary role.

EC2 metadata proxy

A more flexible solution can be found by using a metadata proxy. This sort of service should be a benevolent man-in-the-middle: able to access the actual EC2 metadata service for its own credentials, able to inspect containers making requests to determine what role (if any) they should be assuming, and able to assume those roles and pass tokens back to jobs without those jobs being any the wiser about it.

For our purposes, we will use go-metadataproxy², which will handle:

EC2 metadata requests made by processes in containers (e.g. CI jobs);
Sourcing its own credentials from the actual EC2 metadata service;
Inspecting containers for the IAM role that should be assumed (via the IAM_ROLE environment variable);
Blocking direct access to the EC2 metadata service; and
Assuming the correct role and providing STS tokens transparently to the contained process.

The authentication flow will look something like this:

sequenceDiagram
    autonumber
    participant mdp as metadataproxy
    participant docker
    participant job as CI job
    job->>mdp: client attempts to request credentials from EC2
    mdp-->>docker: inspect job container
    docker-->>mdp: "IAM_ROLE" is "foobar"
    mdp-->>mdp: STS tokens for role "foobar"
    mdp->>job: STS tokens for assumed role "foobar" returned

This also means that the instance profile role must be able to assume the individual roles we want to allow jobs to assume, and the trust policy of the individual roles must allow the instance profile role to assume them.

In short:

The instance profile’s IAM role policy should only permit certain roles to be assumed, either by ARN or some sensible condition (tagged in a certain way, etc).
Roles in the account, in general, should not blindly trust any principal in the account to assume them.³

Configuring the CI agent correctly

Take care when registering the runner

We’re not going to cover it here, but take care when registering the runner. Under this approach, judiciously restricting access to the runner is a critical part of controlling what jobs may run with elevated IAM authority.

Keep a couple things in mind:

Registering runners is cheap; better to have more runners for more granular security than allow projects / pipelines with no need for access to use them.
Runners can be registered at the project, group, or (unless you’re on gitlab.com) the instance level; register them as precisely as your requirements allow.
Runner access can be further restricted and combined with project/group access by allowing them to run against protected refs only, and then restricting who can push/merge to protected branches (including protected tags) to trusted individuals.

Always set IAM_ROLE in the runner configuration

Anything that allows a pipeline author to control what role the proxy assumes is a security… concern. In this context, IAM_ROLE can be set on the container in one of several ways (in order of precedence):

Through the runner configuration;
By the pipeline author; or
By the creator of the image.

Unless you intend to allow the pipeline author to specify the role to assume, it is recommended that IAM_ROLE always be set in the runner configuration file, config.toml. If you don’t want any role to be assumed, great, set the variable to a blank value.

go-metadataproxy discovers the role to assume by interrogating the docker daemon, inspecting the container of the process seeking credentials from the EC2 metadata service. It does this by looking for the value of the IAM_ROLE environment set on the container.

IAM_ROLE must be set on the container itself. While whitelisting the list of allowed images isn’t a terrible idea, the safest and most reliable way of controlling this as the administrator of the runner is to simply set the environment variable as part of the runner configuration.

[[runners]]
  environment = [
    "IAM_ROLE=some-role-name-or-arn",
    ...,
  ]

This also means that we’re going to want a runner configuration per IAM role. (Not terribly surprising, I would hope.)

Running the metadata proxy

This is reasonably straight-forward, in two parts. There are a number of ways to run it, but as we’re doing this in a docker environment anyways, why not let it handle all the messy bits for us?

$ git clone https://github.com/jippi/go-metadataproxy.git
$ cd go-metadataproxy
$ docker build -t local/go-metadataproxy:latest .
$ docker run \
    --detach \
    --restart=always \
    --net=host \
    --name=metadataproxy \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e AWS_REGION=us-west-2 \
    -e ENABLE_PROMETHEUS=1 \
    local/go-metadataproxy:latest

Using the metadata proxy

To use the proxy, the containers must be able to reach it in the same way they would reach the actual EC2 metadata endpoint. We need to prevent requests to the metadata endpoint from reaching the actual endpoint, and instead be transparently redirected to the proxy. (That is, we’re going to play Faythe⁴ here)

To “hijack” container requests to the EC2 metadata service, a little iptables magic is in order. This is well described in the project’s README. I’m including it here as well for completeness’ sake, and with one small change: instead of redirecting connections off of docker0, we reconnect any off of docker+. (If you’re using the runner’s network per build functionality, you may need to tweak this.)

As we’re exposing the metadataproxy on port 8000, you’ll want to make sure that port is firewalled off from the outside; either via iptables or a security group.

# this makes an excellent addition to /etc/rc.local
LOCAL_IPV4=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)

/sbin/iptables \
  --append PREROUTING \
  --destination 169.254.169.254 \
  --protocol tcp \
  --dport 80 \
  --in-interface docker+ \
  --jump DNAT \
  --table nat \
  --to-destination $LOCAL_IPV4:8000 \
  --wait

/sbin/iptables \
  --wait \
  --insert INPUT 1 \
  --protocol tcp \
  --dport 80 \
  \! \
  --in-interface docker0 \
  --jump DROP

IAM role requirements

EC2 Instance Profile

The role belonging to the instance profile associated with the instance our agent lives on should be able to assume the roles we want to allow CI jobs to assume. Specifically, the trust policy must permit iam:GetRole and sts:AssumeRole on these roles.

If you’re using S3 for shared runner caches, you may wish to permit this access through the instance profile role as well. (Implemented properly, the proxy will not permit direct CI jobs to use this role.)

Container / Job IAM roles for assumption

As before, only containers with IAM_ROLE set at the container level will have tokens returned to them by the metadata proxy⁵, and then only if the proxy can successfully assume and convince STS to issue tokens for them. For this to happen, the container/job role’s trust policy must alllows the role of the instance profile associated with the EC2 instance to assume them. Specifically, the trust policy must permit iam:GetRole and sts:AssumeRole.

Profit!

Alright! You should now have a good idea as to how create and run CI jobs that:

CANNOT request tokens directly from the EC2 metadata service
CANNOT implicitly assume the EC2 instance profile’s role
CANNOT leak static or long-lived credentials
CAN transparently assume certain specific roles

Enjoy :)

The nomenclature gets a bit tricky here.

gitlab-runner

The agent responsible for running one or more runner configurations.

A “runner”

A single runner configuration being handled by the gitlab-runner agent.

An entity that can run CI jobs, from the perspective of the CI server (e.g. gitlab.com proper).

↩︎
Lyft also has an excellent tool at https://github.com/lyft/metadataproxy. I’ve used it with success, but go-metadataproxy provides at least rudimentary metrics for scraping. ↩︎
Not that anyone would ever create a trust policy like that, or that it would be one of the defaults offered by the AWS web console. Nope. That would never happen. ↩︎
https://en.wikipedia.org/wiki/Alice_and_Bob ↩︎
Unless, of course, the metadata proxy is configured with a default role – but we’re not going to do that here. ↩︎

Additional GitLab Metrics using `pg_exporter`

chris@weyl.io (Chris Weyl) — Thu, 27 Aug 2020 00:00:00 +0000

GitLab makes a great deal of information available through Prometheus metrics. But not everything.

sql_exporter

Since writing this article, it has come to my attention that there are two more generic “sql_exporter” options. This is less important with GitLab, as you’re basically going to be running Pg, but these are the generic SQL exporter we’re turning postgres_exporter into, below.

The other day I was looking for how many CI jobs and pipelines had been created, total. I figured that would be somewhere in the collection of existing metrics, but the closest I could find was a metric that gave the totals relative to the last time the exporter was restarted.

I use a couple GitLab-specific exporters in this environment already, and thought about creating another one to handle this. As it turns out, this information isn’t exposed through the API, either. It looked like the only way to get this information was to query the database directly.

`--extend.query-path`

While poking around, I noticed that postgres_exporter has an interesting flag, --extend.query-path.

–extend.query-path

Path to a YAML file containing custom queries to run. Check out queries.yaml for examples of the format.

I did as suggested and checked out the queries.yml. Turns out it’s surprisingly easy to create new metrics out of database queries, e.g.:

ci_builds:
  query: "SELECT MAX(id) as total from ci_builds"
  metrics:
    - total:
        usage: "COUNTER"
        description: "Total builds created"

ci_pipelines:
  query: "SELECT MAX(id) as total from ci_pipelines"
  metrics:
    - total:
        usage: "COUNTER"
        description: "Total pipelines created"

The above causes two additional metrics to be generated by the exporter: ci_builds_total and ci_pipelines_total. Neat. To get the information I want, all I need to do is ask postgres_exporter nicely for it.

Configuring the exporter

The GitLab Omnibus package sets up a number of exporters, including postgres_exporter with --extend.query-path already set. However, messing around with a configuration file the omnibus package is responsible for did not sound like fun, and neither did I want to cause the same Pg metrics to be exported twice. Examining the exporter’s flags again, I see two that may help.

postgresql_exporter documentation

disable-default-metrics Use only metrics supplied from queries.yaml via --extend.query-path.
disable-settings-metrics Use the flag if you don’t want to scrape pg_settings.

Looking at those two flags, it appears that I should be able to disable the “standard” metrics and only run the ones I provide in the query file.

Running the exporter

To me, it seems easiest to run our custom postgresql_exporter in parallel with the GitLab supplied one. Running it in a container also allows us to ensure it keeps running (--restart) and runs as the correct user/group for access.

You get to keep both parts

This involves configuring a tool for direct access to your GitLab instance’s database. While the postgres_exporter is a widely-used and reliable tool, if you break your server you get to keep both parts.

A small scriptie to start the exporter, disable standard metrics, and run ours is below. Note that it also ensures it is run as the correct user/group for database access, and the Pg socket + configuration is bind-mounted inside the container for access.

PG_USER="${PG_USER:-gitlab-psql}"
PG_UID="$(id -u $PG_USER)"
PG_GID="$(id -g $PG_USER)"

docker run -d \
    --name gitlab-custom-metrics \
    --restart unless-stopped \
    --user $PG_UID:$PG_GID \
    --publish 19187:9187 \
    -v /var/opt/gitlab/postgresql:/var/opt/gitlab/postgresql \
        -v `pwd`/queries.yml:/queries.yml:ro \
    -e DATA_SOURCE_NAME="user=$PG_USER host=/var/opt/gitlab/postgresql database=gitlabhq_production" \
    wrouesnel/postgres_exporter \
        --disable-default-metrics \
        --disable-settings-metrics \
        --extend.query-path /queries.yml

With that, the metrics exporter is exposed and ready to be scraped at localhost:19187/metrics.

# HELP ci_builds_total Total builds created
# TYPE ci_builds_total counter
ci_builds_total{server="/var/opt/gitlab/postgresql:5432"} 437695
# HELP ci_pipelines_total Total pipelines created
# TYPE ci_pipelines_total counter
ci_pipelines_total{server="/var/opt/gitlab/postgresql:5432"} 67665

Conclusion

With this in place, we can collect and display or alert on these custom metrics. And, of course, everyone loves a good dashboard graph:

This might seem like a lot for two small metrics, but compared to writing a custom exporter it’s nothing. If you’re like me, you’ll also discover your queries.yml will quickly grow with additional metrics definitions.

KMS key context, IAM conditions, and s3

chris@weyl.io (Chris Weyl) — Sun, 23 Aug 2020 00:00:00 +0000

At $work, I’ve been using KMS to encrypt s3 bucket contents for some time now. It works rather well, but one thing that had been bugging me is that our IAM policies granted both read permissions on bucket objects and encrypt/decrypt on the relevant KMS key. That is, principals with the policies attached can use the key to encrypt/decrypt anything they otherwise have permission to access, not just objects in the bucket. It didn’t appear that there was a reasonable way to tighten this until I ran across references to the IAM kms:EncryptionContext: condition.

Using kms:EncryptionContext: it is possible to conditionally restrict a policy based on the ARN of the resource being acted upon. That is to say, one can use this condition to only allow a KMS key to be used to decrypt objects in a certain s3 bucket.

It took me a bit to figure this out as the docs didn’t quite spell out how to use an ARN as an encryption context (and, you know, encryption), so here’s a policy that shows it in action. The policy allows actions a certain KMS key to be used only in the context of the given bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "kmsKeyAccess",
      "Effect": "Allow",
      "Action": [
        "kms:ReEncrypt",
        "kms:GenerateDataKey*",
        "kms:Encrypt",
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:us-west-2:12345678901234:key/1234-abcd-567890-12345",
      "Condition": {
        "StringLike": {
          "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::davros-world-domination/*"
        }
      }
    }
  ]
}

These actions are only granted on objects in the specific s3 bucket and not denied explicitly to anything else. This is important, as we don’t want our efforts here to block a legitimate grant somewhere else.

This can be extended to apply to multiple buckets by changing the condition test to ForAnyValue:StringLike.

Conditional git Configuration

chris@weyl.io (Chris Weyl) — Thu, 21 Mar 2019 00:00:00 +0000

git has always(?) allowed for additional configuration files to be unconditionally included:

[include]
    path = path/to/gitconfig

Each individual git repo has always had the ability to maintain its own configuration at .git/config. However, sometimes on our systems we also have certain locations where we store multiple git projects, which may need different configuration from the global, but still common across that location.

Since … well, for the last year or two at least, git has allowed for the conditional inclusion of configuration files.

For example, I contribute to F/OSS projects using one email address, which lives in my global git config . However, for work projects, I want to use my work email everywhere — and accidentally pushing w/my personal email address is just embarrassing. All of my work projects live under a certain directory, so I can tell git that if a given repository’s gitdir lives under ~/work, it should also load an additional configuration file:

[includeIf "gitdir:~/work/"]
    path = ~/work/gitconfig

…and in there, I can set

; this is ~/work/gitconfig
[user]
    email = cweyl@work.com

In this way, I do not need to remember to change the email address of any repos I clone under ~/work to my work address. This is especially useful as I not infrequently find myself forking and submitting bugfix PR/MR’s to upstream, and if I do that for $work then I want to be using my work email address.

First Post

chris@weyl.io (Chris Weyl) — Wed, 28 Feb 2018 00:00:00 +0000

…um, kinda. I’m switching over from wordpress to Statocles, and porting my older posts over.

Still, who can resist “first post!” ;)

Fast Project Finding With fzf

chris@weyl.io (Chris Weyl) — Sat, 24 Feb 2018 00:00:00 +0000

fzf is a fantastic utility, written by an author with a history of writing useful things. He’s also a vim user, and in addition to his other vim plugins he has created an “enhancement” plugin called fzf.vim.

One of the neat things fzf.vim does is make it easy to create new commands for fuzzy searches. If you’re like me, you probably have some absurd number of project repositories you keep around and jump to, as necessary. Not everything is in the same directory (e.g. ~/work/), naturally, and with a laptop, desktop, and a couple other machines the less-frequently used repos may be where one least expects them to be — or not present at all.

It’s not hugely annoying, just a sort of mild pain to have to spend several extra seconds doing a fuzzy search manually, rather than having fzf do it. But we do have fzf, and it’s not difficult at all to build out a new search, so there’s really no reason to keep on inflicting that pain.

Create a `:Projects` command

Let’s create a new command in my vimrc, :Projects, that invokes fzf to search through all the different work directories I have.

command! -nargs=0 Projects
    \ call fzf#run(fzf#wrap('projects', {
    \   'source': 'find ~/work ~/.vim/plugged -name .git -maxdepth 3 -printf ''%h\n''',
    \   'sink': function('rsrchboy#fzf#FindOrOpenTab'),
    \   'options': '-m --prompt "Projects> "',
    \}, <bang>0))

What does this do?

Defines a new vim command, :Projects

No surprises here.
Invokes fzf#run() to run a fzf search

fzf#run() handles the actual execution and presentation of fzf, as well has dispatching the results back to the sink. fzf#wrap() is neat. It allows a command to take advantage of fzf.vim’s option handling – or not, by simply omitting it.
Uses find to look for repositories

We know roughly where to look(~/work/, ~/.vim/plugged) and how deep to look. Just about everything I do is backed by git, so we can look for repositories and return the parent of the found .git back to fzf.

Note that the find invocation deliberately omits a -type d argument. I do use git worktrees, meaning .git may well be a file (a “gitlink”).
Calls out to rsrchboy#fzf#FindOrOpenTab() with the project selected

The sink option tells fzf#run() what to do with the results. In our case we have provided fzf#run() with a callback function, but you can also use built-ins as sinks.

The callback “sink” function

fun! rsrchboy#fzf#FindOrOpenTab(work_dir) abort

    " loop over our tabs, looking for one with a t:git_workdir matching our
    " a:workdir; if found, change tab; if not fire up fzf again to find a file
    " to open in the new tab

    for l:tab in (gettabinfo())
        if get(l:tab.variables, 'git_workdir', '') ==# a:work_dir
            exe 'tabn ' . l:tab.tabnr
            return
        endif
    endfor

    call fzf#run(fzf#wrap('other-repo-git-ls', {
        \   'source': 'git ls-files',
        \   'dir': a:work_dir,
        \   'options': '--prompt "GitFiles in ' . a:work_dir . '> "',
        \   'sink': 'tabe ',
        \}, 0))

    return
endfun

In general, I use one tab per project (repository) in vim. For me, this is a nice balance of utility and sanity. It also allows me to do things like set t:git_dir and t:git_workdir to the git and workdir, respectively, of the repository associated with the tab.

Our callback function first attempts to find an open tab with the workdir requested; if found, it just switches to it and returns. (It should probably admonish me to read the tab line before invoking :Projects.)

If not found. the callback function invokes fzf#run() again. This time we use git ls-files to generate the source list for fzf, allowing us to pick a file to be opened by the given sink: tabe.

Hey, that wasn’t too hard!

Easier than writing this post, I’d say ;)

Happy hacking!