One thing I wish was embraced more, and not just by users but also platforms such as GitLab is re-usable actions, or workflows if you will. As they help spend less time away from your primary project. Also, containers.
Why Python rather than say Rust?
If you’re having a déjà vu, or perhaps you’d just be interested in a similar guide based around setting up a new project in Rust that’s also worth a read. The primary reason to use Python here besides being more familiar with it would be ease of contributions. Having used Rust I’ve not reached a point where I feel like I can prototype and hack away in it as quickly, or expect a drive-by contributor to do that compared to Python1. The inspiration for this post is a project with that in mind.
A little bit of poetry before we get to it
Well. I’m going to use poetry so if you don’t have it yet be sure to install it:
curl -sSL https://install.python-poetry.org | python3 - # Alternatively: pipx install poetry
Why use poetry? Because it replaces setup.py, requirements.txt, requirements-dev.txt, setup.cfg, Pipfile.lock and others files you may have used with a file called pyproject.toml.
Setting up a new Python project
This step seems easy enough. I’m creating a new repo on GitHub, calling it statiqa and selecting no license since I’d like to use the EUPL - alternatively you might want to use the LGPL2 or something else. I’m doing it this way around so that there’s no need to configure the local repo manually which has a main branch.
Note: You obviously want to replace statiqa with something else anywhere I mention it throughout this article.
Now there’s a repo to clone. And let’s get poetry to setup the barebones project.
git clone git@github.com:kalikiana/statiqa.git
cd statiqa
poetry init
You will be asked to confirm a few things such as your name, the description of the project and the license. If you already know what dependencies you’ll be using you can add them right away. And if you’ve not used poetry before you may be delighted to learn that its distinguishes development dependencies out of the box! But more on that later.
mkdir statiqa tests
touch {statiqa,tests}/__init__.py
touch README.md # Empty for now to avoid linting errors
Let’s add the boiler plate i.e. an initial piece of code implementing a class renderer in our new module, in a file called renderer.py:
class Renderer:
results: str
def __init__(self, results: str):
self.results = results
def process(self) -> {}:
print(f"Let's process {self.results}, shall we?")
details = {"results": self.results}
return details
Since we want to be able to run this from the command-line we’ll define a script:
echo '[tool.poetry.scripts]' >> pyproject.toml
echo 'statiqa="statiqa.__main__:__main__"' >> pyproject.toml
echo 'from . import cli; cli.CLI().run()' > statiqa/__main__.py
Whilst it is tempting to implement a trivial entry-point experience has taught me adding the 5 or so lines for a module that takes care of argument parsing which we’ll be able to unit-test later down the line will save us some frustration so here’s cli.py:
import argparse
from . import renderer
class CLI:
parser: argparse.ArgumentParser
def __init__(self):
self.parser = argparse.ArgumentParser(description="Render openQA test results")
self.parser.add_argument("results", type=str, help='results or "pool" folder')
def run(self) -> None:
return renderer.Renderer(self.parser.parse_args().results).process()
Let’s run this just to confirm that our setup is working so far.
poetry run statiqa # Alternatively: python3 -m statiqa
Can I haz unit tests?
Be sure to add unit tests early:
import unittest
from statiqa import renderer
class TestDetails(unittest.TestCase):
def setUp(self):
self.renderer = renderer.Renderer("example")
def test_details(self):
details = self.renderer.process()
self.assertEqual(details["results"], "example", "Results passed correctly")
While we’re at it, let’s also employ linting and code formatting. flake8 makes it real easy. Better yet let’s also integrate it with [black]((https://github.com/psf/black) and pylint via plugins so the only command you need to remember to run is this one:
poetry add --group=dev pytest flake8{-black,-pylint}
Note: As mentioned previously there’s a built-in distinction between different types of dependencies. We’ve used the dev group here so it’s easy to skip them.
Note that since we’re using poetry for everything the commands are contained within a virtual environment out of the box.
poetry run pytest
poetry run flake8
On a side note we were confronted with the question of licensing when creating the repo. That’s a great start. We can ensure our code is actually tagged consistently, especially whilst it is a new project. Let’s take advantage of REUSE here, which allows us to add license headers, download a copy of the license to our source repo and also check for any gaps in compliance:
poetry add --group=dev reuse
poetry run reuse download EUPL-1.2
poetry run reuse annotate --copyright='Liv Dywan <liv@twotoasts.de>' --license=EUPL-1.2 --recursive . --skip-unrecognised
echo '__pycache__' > .gitignore
poetry run reuse annotate --copyright='Liv Dywan <liv@twotoasts.de>' --license=EUPL-1.2 poetry.lock .gitignore --style python
poetry run reuse lint
Note how we took two special-cases into account that use an unusual format. In both cases it’s okay to assume Python style since the same syntax can be used. Any generated files listed in .gitignore won’t count against compliance so if something shows up here it’s probably a candidate for the list.
Let’s deploy this in a container
Fiddling with dependencies can be annoying. Let’s containerize this:
FROM python:3.11-alpine
COPY statiqa ./statiqa
COPY pyproject.toml poetry.lock README.md .
RUN apk add --no-cache poetry && \
poetry install --no-cache --without dev && \
apk del poetry
ENTRYPOINT ["poetry", "run", "statiqa"]
Note the use of --without which avoids pulling development dependencies into the container besides the use of a small base image.
You can build it locally with e.g. podman or docker:
podman build -t statiqa .
Make it actionable
Whilst containers are arguably the most portable way to deploy an app integration with GitHub Actions may be more convenient provided your project lives on GitHub… Codeberg or Gitea as more code hosting platforms are implementing support for compatible workflows. All it takes then is a little action.yaml:
---
name: Static (openQA) test result visualization
description: Render the results of test runs in your CI
branding:
icon: 'check-circle'
color: 'blue'
inputs:
results:
description: The folder with test results, sometimes called "pool" directory
required: false
default: 'pool'
runs:
using: docker
image: 'Dockerfile'
args:
- ${{ inputs.results }}
Well, that was too easy, wasn’t it? Having already containerized our little application we don’t really need to do much to run it. If you’d like to learn more about the inner workings you are welcome to checkout my full howto on publishing re-usable actions where I also explain composite actions and inputs in more detail.
Please integrate this continuously
It should go without saying that, although it’s a proof of concept for now, I’ll add a basic template for CI, in this case GitHub Actions just because it’s the simplest option when the project is hosted on GitHub. Time will tell if it’s going to do the job in the long run. So let’s add a file .github/workflows/test.yaml with the following contents:
name: Unit tests
on:
pull_request:
push:
branches:
- main
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: 3.11
- uses: abatilo/actions-poetry@v2
- name: Install dependencies
run: poetry install
- name: Python test cases
run: poetry run pytest -v
- name: Style checks
run: poetry run flake8
- name: License compliance
run: poetry run reuse lint
action:
runs-on: ubuntu-latest
steps:
- uses: ./
with:
results: example
- uses: actions/upload-artifact@v3
with:
name: Example test result visualization
path: ${{ github.workspace }}
if: always()
This includes formatting, linting, unit tests and using the action itself. We also know that the container works since the action is based on it.
Let’s also create a .github/workflows/container.yaml to automatically publish our container image with every change:
name: GitHub Container Registry
on:
push:
branches:
- main
permissions:
contents: read
packages: write
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build and publish Docker image for ARM64 and AMD64 architectures at the same time
uses: VaultVulp/gp-docker-action@1.6.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
image-name: min
image-platform: linux/arm64,linux/amd64
The gp-container-action takes care of building, as well as publishing to the GitHub Container Registry.
Ready for commitment?
We cheated earlier when poetry was looking for a README.md. Let’s add content to it, shall we?
## statiqa
### What's this project about?
Render the results of your [openQA](https://open.qa) tests directly in your CI. No need to have access to a full web UI. This is also handy to backup just the results without keeping all of the original data around.
### How do I build and run this?
The quickest way to run statiqa is via the container:
```bash
podman run -it --rm -v $(pwd):/w -w /w ghcr.io/kalikiana/statiqa/min:latest
```
For a typical development setup the necessary dependencies can be installed via [poetry](https://python-poetry.org):
```bash
poetry install
poetry run pytest -v
```
### Integrate with GitHub Actions
Using statiqa as part of a workflow can look something like this:
```yaml
name: openQA tests
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- kalikiana/isotovideo-action@main
with:
schedule: tests/foo/bar
- uses: kalikiana/statiqa
with:
results: ${{ github.workspace }}
- uses: actions/upload-artifact@v3
with:
name: Test result visualization
path: ${{ github.workspace }}
if: always()
```
Finally commit and push everything. Push early, push often. You don’t want to lose more hours or even days worth of work if you can help it. Remember that your local changes are temporary.
git commit -v
git push origin HEAD
If you feel like it, consider also bloging about your new project, attract contributors and get feedback. Good luck!
-
What did you expect me to say? This is an opinionated blog, and I have strong opinions especially when I talk about my two favorite programming languages. There’s a reason I don’t have one favorite. ↩︎