Introduction

Welcome to the documentation site of the mataroa blog project!

Main repository on GitHub
github.com/mataroa-blog/mataroa

Mirror repository on sr.ht
git.sr.ht/~sirodoht/mataroa

Report bugs on GitHub
github.com/mataroa-blog/mataroa/issues

Contribute on GitHub with Pull Requests
github.com/mataroa-blog/mataroa/pulls

Contribute (platform independent) with email patches
~sirodoht/public-inbox@lists.sr.ht

Community mailing list
lists.sr.ht/~sirodoht/mataroa-community

Start

Start learning about mataroa with reading the main repository README:

mataroa

Naked blogging platform.

Community

We have a mailing list at ~sirodoht/mataroa-community@lists.sr.ht for the mataroa community to introduce themselves, their blogs, and discuss anything that’s on their mind!

Archives at lists.sr.ht/~sirodoht/mataroa-community

Contributing

Open a PR on GitHub.

Send an email patch to ~sirodoht/public-inbox@lists.sr.ht. See how to contribute using email patches here: git-send-email.io.

Read our docs at docs.mataroa.blog

Development

This is a Django codebase. Check out the Django docs for general technical documentation.

The Django project is mataroa. There is one Django app, main, with all business logic. Application CLI commands are generally divided into two categories, those under python manage.py and those under make.

Set up subdomains

Because mataroa works primarily with subdomain, one cannot access the basic web app using the standard http://127.0.0.1:8000 or http://localhost:8000 URLs. What we do for local development is adding a few custom entries on our /etc/hosts system file.

Important note: there needs to be an entry of each user account created in the local development environment, so that the web server can respond to it.

The first line is the main needed: mataroalocal.blog. The rest are included as examples of other users one can create in their local environment. The easiest way to create them is to go through the sign up page (http://mataroalocal.blog:8000/accounts/create/ using default values).

# /etc/hosts

127.0.0.1 mataroalocal.blog

127.0.0.1 paul.mataroalocal.blog
127.0.0.1 random.mataroalocal.blog
127.0.0.1 anyusername.mataroalocal.blog

This will enable us to access mataroa locally (once we start the web server) at http://mataroalocal.blog:8000/ and if we make a user account with username paul, then we will be able to access it at http://paul.mataroalocal.blog:8000/

Docker

[!NOTE]
This is the last step for initial Docker setup. See the "Environment variables" section below, for further configuration details.

To set up a development environment with Docker and Docker Compose, run the following to start the web server and database:

docker compose up

If you have also configured hosts as described above in the "Set up subdomains" section, mataroa should now be locally accessible at http://mataroalocal.blog:8000/

Note: The database data are saved in the git-ignored docker-postgres-data docker volume, located in the root of the project.

Dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.dev.txt
pip install -r requirements.txt

Environment variables

A file named .envrc is used to define the environment variables required for this project to function. One can either export it directly or use direnv. There is an example environment file one can copy as base:

cp .envrc.example .envrc

.envrc should contain the following variables:

# .envrc

export DEBUG=1
export SECRET_KEY=some-secret-key
export DATABASE_URL=postgres://mataroa:db-password@db:5432/mataroa
export EMAIL_HOST_USER=smtp-user
export EMAIL_HOST_PASSWORD=smtp-password

When on production, also include/update the following variables (see Deployment and Backup):

# .envrc

export DEBUG=0
export PGPASSWORD=db-password

When on Docker, to change or populate environment variables, edit the environment key of the web service either directly on docker-compose.yml or by overriding it using the standard named git-ignored docker-compose.override.yml.

# docker-compose.override.yml

version: "3.8"

services:
  web:
    environment:
      EMAIL_HOST_USER=smtp-user
      EMAIL_HOST_PASSWORD=smtp-password

Finally, stop and start docker compose up again. It should pick up the override file as it has the default name docker-compose.override.yml.

Database

This project is using one PostreSQL database for persistence.

One can use the make pginit command to initialise a database in the postgres-data/ directory.

After setting the DATABASE_URL (see above), create the database schema with:

python manage.py migrate

Initialising the database with some sample development data is possible with:

python manage.py loaddata dev-data

dev-data is defined in main/fixtures/dev-data.json
Credentials of the fixtured user are admin / admin.

Serve

To run the Django development server:

python manage.py runserver

If you have also configured hosts as described above in the "Set up subdomains" section, mataroa should now be locally accessible at http://mataroalocal.blog:8000/

Testing

Using the Django test runner:

python manage.py test

For coverage, run:

coverage run --source='.' --omit '.venv/*' manage.py test
coverage report -m

Code linting & formatting

We use ruff for Python code formatting and linting.

To format:

ruff format

To lint:

ruff check
ruff check --fix

Python dependencies

We use pip-tools to manage our Python dependencies:

pip-compile -U requirements.in
pip install --upgrade pip
pip install -r requirements.txt

Deployment

See the Deployment document for an overview on steps required to deploy a mataroa instance.

Useful Commands

To reload the gunicorn process:

sudo systemctl reload mataroa

To reload Caddy:

systemctl restart caddy  # root only

gunicorn logs:

journalctl -fb -u mataroa

Caddy logs:

journalctl -fb -u caddy

Get an overview with systemd status:

systemctl status caddy
systemctl status mataroa

Backup

See Database Backup for details. In summary:

To create a database dump:

pg_dump -Fc --no-acl mataroa -h localhost -U mataroa -f /home/deploy/mataroa.dump -w

To restore a database dump:

pg_restore -v -h localhost -cO --if-exists -d mataroa -U mataroa -W mataroa.dump

Management

In addition to the standard Django management commands, there are also:

processnotifications: sends notification emails for new blog posts of existing records.
mailexports: emails users of their blog exports.

They are triggered using the standard manage.py Django way; eg:

python manage.py processnotifications

Billing

One can deploy mataroa without setting up billing functionalities. This is the default case. To handle payments and subscriptions this project uses Stripe. To enable Stripe and payments, one needs to have a Stripe account with a single Product (eg. "Mataroa Premium Plan").

To configure, add the following variables from your Stripe account to your .envrc:

export STRIPE_API_KEY="sk_test_XXX"
export STRIPE_PUBLIC_KEY="pk_test_XXX"
export STRIPE_PRICE_ID="price_XXX"

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3.

Coding Conventions

All files should end with a new line character.
Python code should be formatted with ruff.

Git Commit Message Guidelines

We follow some simple non-austere git commit message guidelines.

Start with a verb
- add
- change
- delete
- fix
- refactor
- tweak
- et al.
Start with a lowercase letter
- eg. change analytic page path to the same of page slug
Do not end with a fullstop

File Structure Walkthrough

Here, an overview of the project's code sources is presented. The purpose is for the reader to understand what kind of functionality is located where in the sources.

All business logic of the application is in one Django app: main.

Condensed and commented sources file tree:

.
├── .build.yml # SourceHut CI build config
├── .envrc.example # example direnv file
├── .github/ # GitHub Actions config files
├── Caddyfile # configuration for Caddy webserver
├── Dockerfile
├── LICENSE
├── Makefile # make-defined tasks
├── README.md
├── backup-database.sh
├── default.nix # nix profile
├── deploy.sh
├── docker-compose.yml
├── docs/
├── export_base_epub/ # base sources for epub export functionality
├── export_base_hugo/ # base sources for hugo export functionality
├── export_base_zola/ # base sources for zola export functionality
├── main/
│   ├── admin.py
│   ├── apps.py
│   ├── denylist.py # list of various keywords allowed and denied
│   ├── feeds.py # django rss functionality
│   ├── fixtures/
│   │   └── dev-data.json # sample development data
│   ├── forms.py
│   ├── management/ # commands under `python manage.py`
│   │   └── commands/
│   │       └── processnotifications.py
│   │       └── mailexports.py
│   ├── middleware.py # mostly subdomain routing
│   ├── migrations/
│   ├── models.py
│   ├── static/
│   ├── templates
│   │   ├── main/ # HTML templates for most pages
│   │   ├── assets/
│   │   │   ├── drag-and-drop-upload.js
│   │   │   └── style.css
│   │   ├── partials/
│   │   │   ├── footer.html
│   │   │   ├── footer_blog.html
│   │   │   └── webring.html
│   │   └── registration/
│   ├── tests/
│   │   ├── test_billing.py
│   │   ├── test_blog.py
│   │   ├── test_comments.py
│   │   ├── test_images.py
│   │   ├── test_management.py
│   │   ├── test_pages.py
│   │   ├── test_posts.py
│   │   ├── test_users.py
│   │   └── testdata/
│   ├── urls.py
│   ├── util.py
│   ├── validators.py # custom form and field validators
│   ├── views.py
│   ├── views_api.py
│   ├── views_billing.py
│   └── views_export.py
├── manage.py
├── mataroa
│   ├── asgi.py
│   ├── settings.py # django configuration file
│   ├── urls.py
│   └── wsgi.py
├── requirements.in # user-editable requirements file
├── requirements.txt # pip-compile generated version-locked dependencies
└── requirements.dev.txt # user-editable development requirements

`main/urls.py`

All urls are in this module. They are visually divided into several sections:

general, includes index, dashboard, static pages
user system, includes signup, settings, logout
blog posts, the CRUD opertions of
blog extras, includes rss and newsletter features
comments, related to the blog post comments
billing, subscription and card related
blog import, export, webring
images CRUD
analytics list and details
pages CRUD

`main/views.py`

The majority of business logic is in the views.py module.

It includes:

indexes, dashboard, static pages
user CRUD and login/logout
posts CRUD
comments CRUD
images CRUD
pages CRUD
webring
analytics
notifications subscribe/unsubscribe
moderation dashboard
sitemaps

Generally, Django class-based generic views are used most of the time as they provide useful functionality abstracted away.

The Django source code for generic views is also extremely readable:

base.py: base View and TemplateView
list.py: ListView
edit.py: UpdateView, DeleteView, FormView
detail.py: DetailView

Function-based views are used in cases where the CRUD/RESTful design pattern is not clear such as notification_unsubscribe_key where we unsubscribe an email via a GET operation.

`main/views_api.py`

This module contains all API related views. These views have their own api key based authentication.

`main/views_export.py`

This module contains all views related to the export capabilities of mataroa.

The way the exports work is by reading the base files from the repository root: export_base_hugo, export_base_zola, export_base_epub for Hugo, Zola, and epub respectively. After reading, we replace some strings on the configurations, generate posts as markdown strings, and zip-archive everything in-memory. Finally, we respond using the appropriate content type (application/zip or application/epub) and Content-Disposition attachment.

`main/views_billing.py`

This module contains all billing and subscription related views. It’s designed to support one payment processor, Stripe.

`main/tests/`

All tests are under this directory. They are divided into several modules, based on the functionality and the views they test.

Everything uses the built-in Python unittest module along with standard Django testing facilities.

`main/models.py` and `main/migrations/`

main/models.py is where the database schema is defined, translated into Django ORM-speak. This always displays the latest schema.

main/migrations/ includes all incremental migrations required to reach the schema defined in main/models.py starting from an empty database.

We use the built-in Django commands to generate and execute migrations, namely makemigrations and migrate. For example, the steps to make a schema change would be something like:

Make the change in main/models.py. See Django Model field reference.
Run python manage.py makemigrations to auto-generate the migrations.
Potentially refactor the auto-generated migration file (located at main/migrations/XXXX_auto_XXXXXXXX.py)
Run python manage.py migrate to execute migrations.
Also make format before committing.

`main/forms.py`

Here a collection of Django-based forms resides, mostly in regards to user creation, upload functionalities (for post import or image upload), and card details submission.

See Django Form fields reference.

`main/templates/assets/style.css`

On Mataroa, a user can enable an option, Theme Zia Lucia, and get a higher font size by default. Because we need to change the body font-size value, we render the CSS. It is not static. This is why it lives inside the templates directory.

Dependencies

Dependency Policy

The mataroa project has an unusually strict yet usually unclear dependency policy.

Vague rules include:

No third-party Django apps.
All Python / PyPI packages should be individually vetted.
- Packages should be published from community-trusted organisations or developers.
- Packages should be actively maintained (though not necessarily actively developed).
- Packages should hold a high quality of coding practices.
No JavaScript libraries / dependencies.

Current list of top-level PyPI dependencies (source at requirements.in):

Adding a new dependency

After approving a dependency, the process to add it is:

Assuming a venv is activated and requirements.dev.txt are installed.
Add new dependency in requirements.in.
Run pip-compile to generate requirements.txt
Run pip install -r requirements.txt

Upgrading dependencies

When a new Django version is out it’s a good idea to upgrade everything.

Steps:

Assuming a venv is activated and requirements.dev.txt are installed.
Run pip-compile -U to generate an upgraded requirements.txt.
Run git diff requirements.txt and spot non-patch level vesion bumps.
Examine release notes of each one.
Unless something comes up, make sure tests and smoke tests pass.
Deploy new dependency versions.

Deployment

Step 1: Ansible

We use ansible to provision a Debian 12 Linux server.

(1a) First, set up configuration files:

cd ansible/
# Make a copy of the example file
cp .envrc.example .envrc

# Edit parameters as required
vim .envrc

# Load variables into environment
source .envrc

(1b) Then, provision:

ansible-playbook playbook.yaml -v

Step 2: Wildcard certificates

We use Automatic DNS API integration with DNSimple:

https://github.com/acmesh-official/acme.sh?tab=readme-ov-file#1-how-to-install
https://github.com/acmesh-official/acme.sh/wiki/dnsapi#dns_dnsimple

Note: acme.sh's default SSL provider is ZeroSSL which does not accept email with plus-subaddressing. It will not error gracefully, just fail with a cryptic message (tested with acmesh v3.0.7).

curl https://get.acme.sh | sh -s email=person@example.com
# Note: Installation inserts a cronjob for auto-renewal

# Setup DNSimple API
echo 'export DNSimple_OAUTH_TOKEN="token-here"' >> /root/.acme.sh/acme.sh.env

# Issue cert
acme.sh --issue --dns dns_dnsimple -d mataroa.blog -d *.mataroa.blog

# We "install" (copy) the cert because we should not use the cert from acme.sh's internal store
acme.sh --install-cert -d mataroa.blog -d *.mataroa.blog --key-file /etc/caddy/mataroa-blog-key.pem --fullchain-file /etc/caddy/mataroa-blog-cert.pem --reloadcmd "chown caddy:www-data /etc/caddy/mataroa-blog-{cert,key}.pem && systemctl restart caddy"

Step 3: Cronjobs and Automated backups

There are a few cronjobs that need setting up and, of course, backups are essential:

(3a) Cronjobs
(3b) Database Backup

Cronjobs

We don't use cron but systemd timers for jobs that need to run recurringly.

Process email notifications

python manage.py processnotifications

Sends notification emails for new blog posts.

Triggers daily at 10AM server time.

Email blog exports

python manage.py mailexports

Emails users their blog exports.

Triggers monthly, first day of the month, 6AM server time.

Database backup

./backup-database.sh

Triggers every 6 hours.

Database Backup

Shell Script

We use the script backup-database.sh to dump the database and upload it into an S3-compatible object storage cloud using rclone. This script needs the database password as an environment variable. The key must be PGPASSWORD. The variable can live in .envrc as such:

export PGPASSWORD=db-password

Commands

To create a database dump run:

pg_dump -Fc --no-acl mataroa -h localhost -U mataroa -f /home/deploy/mataroa.dump -W

To restore a database dump run:

pg_restore --disable-triggers -j 4 -v -h localhost -cO --if-exists -d mataroa -U mataroa -W mataroa.dump

Initialise and configure backup script

cp /var/www/mataroa/backup-database.sh /home/deploy/

Setup rclone

Create bucket on Scaleway or any other S3-compatible object storage.
Find bucket URL.
- On Scaleway: it's on Bucket Settings.
Acquire IAM Access Key ID and Secret Key.
- On Scaleway: IAM -> Applications -> Project default -> API Keys

rclone config

Server Migration

Sadly or not, nothing lasts forever. One day you might do a server migration. Among many, mataroa is doing something naughty. We store everything, images including, in the Postgres database. Naughty indeed, yet makes it much easier to backup but also migrate.

To start with, one a migrator has setup their new server (see Deployment) we recommend testing everything in another domain, other than the main (existing) one.

Once everything works:

Verify all production variables and canonical server names exist in settings et al.
Disconnect production server from public IP. This is not a zero-downtime migration — to be clear.
Run backup-database.sh one last time.
Assign elastic/floating IP to new server.
Run TLS certificate (naked and wildcard) generations.
scp database dump into new server.
Restore database dump in new server.
Start mataroa and caddy systemd services

Later:

Setup cronjobs / systemd timers
Setup healthcheks for recurring jobs.
Verify DEBUG is 0.

The above assume the migrator has a floating IP that they can move around. If not, there are two problems. The migrator needs to coordinate DNS but much more problematically all custom domains stop working :/ For this reason we should implement CNAME custom domains. However, CNAME custom domains do not support root domains, so what's the point anyway you ask. Good question. I don't know. I only hope I never decide to switch away from Hetzner.

Peace.

Mataroa Documentation

main/models.py and main/migrations/

`main/models.py` and `main/migrations/`