Disclaimer: This is purely a bit of “what if?” fun, not serious advice

The basic idea is quite simple

1. Write a timer to pull a git repo
2. If a new change is found, trigger a oneshot service to build the project

Writing this for each project would get quite repetitive, so I’d like to generify it as follows

1. The timer will be parameterised with the git URL, so I can do systemctl enable systemd-ci-poll@$REPO
2. The builder service will expect an executable at $REPO/.systemd-ci/build

Assumptions

Since this is just a proof of concept, I make the followingg assumptions

1. Every project has a name marching [a-Z0-9]+, to saves character escaping headaches
2. Every project is already cloned in the workspace dir, and we’re only building the branch that’s been checked out
3. I’m not including any sort of artefact archiving or workspace cleaning. That could be implemented if needed, but for this PoC I’ll take the laissez-faire approach of leaving that to the build script
4. We can always fast-forward. No force pushes!

System setup

I deployed a fresh Ubuntu 20.04 LTS Linux container on Proxmox and made the following changes

• Installed git
• Add a systemd-ci user and group with the normal minimal permissions expected of a service user
• Create the workspace dir /var/lib/systemd-ci with owner systemd-ci:systemd-ci and mode 770

The builder service

This service is quite simple - just run an executable

/etc/systemd/system/systemd-ci-build@.service

[Unit]
Description=Build a systemd-ci project
After=multi-user.target

[Service]
Type=oneshot
WorkingDirectory=/var/lib/systemd-ci/%i
ExecStart=/var/lib/systemd-ci/%i/.systemd-ci/build
User=systemd-ci

We can test this in isolation, by just creating

/var/lib/systemd-ci/test/.systemd-ci/build (mode: +x)

#!/bin/bash -eux
touch success
echo it works

Then execute systemctl start systemd-ci-build@test (remembering to do a daemon-reload first). Logs can be found with journalctl -u systemd-ci-build@test

The trigger service

So for the timer, we need to repeatedly pull the repository, and take note of when it changes. In accordance with assumption 4, let’s be really hacky and just grep Fast-forward in the git output

/etc/systemd/system/systemd-ci-poll@.service

[Unit]
Description=Poll a systemd-ci project repo and trigger a build if necessary
After=multi-user.target
OnSuccess=systemd-ci-build@%i

[Service]
Type=oneshot
WorkingDirectory=/var/lib/systemd-ci/%i
ExecStart=/bin/bash -c "git pull | grep Fast-forward"
User=systemd-ci

And here we run into a problem. OnSuccess was added in systemd 249, but this version of Ubuntu has 245. Luckily, bash has a negation operator, so I can use OnFailure and ExecStart=/bin/bash -c "! git...

The polling timer

/etc/systemd/system/systemd-ci-poll@.timer

[Unit]
Description=Run a systemd-ci poll every 15m

[Timer]
OnUnitInactiveSec=15m

[Install]
WantedBy=timers.target

As far as I understand you need to enable systemd-ci-poll@.timer at least once, in addition to systemd-ci-poll@$REPO.timer. Well, I guess that provides an easy way to turn the whole thing off if needed

Project creation script

systemd-ci-init (mode +x)

#!/bin/bash -eux

# Syntax: sudo systemd-ci-init <repo url> <project name>

cd /var/lib/systemd-ci
git clone $1 $2
chown -R systemd-ci:systemd-ci $2
systemctl enable --now systemd-ci-poll@.timer
systemctl enable --now systemd-ci-poll@$2.timer

Conclusion

Yes, you can do it, no, you probably shouldn’t. The main problem is that you pollute the logs with “failed” polls, but I imagine this can be fixed if one is determined enough, but not on version 245

Minima partially supported this. There’s an official dark scheme, but it uses SCSS variables and hence has to be specified at compile time. I’ve converted those to native CSS variables/custom properties, but I did have to inline the use of lighten/darken. It would have been possible to work around this by indirecting the variables, but I didn’t see it as worth it

Here’s a code block to test syntax highlighting, since I don’t have any yet

def foo
  puts 'foo'
end

TL;DR

See the sections beneath the table for discussion of these points. Summary at the bottom of the article.

Licensing

Let’s start with the elephant in the room. SnapRAID is an open source project while Unraid costs between $60 and $130 depending on how many drives you use. Unless you have ethical objections to proprietary software I don’t think this is a big difference. The Unraid licenses are fairly cheap when you consider how much hardware costs, and they’re lifetime licenses too.

Some people are concerned that Unraid locks in your data - this isn’t true; I moved from Unraid to SnapRAID with just a parity sync. Your data drives are plain XFS or Btrfs, perhaps with LUKS. The parity drive format is undocumented afaik, but I’m sure someone’s reverse engineered it. Presumably the first parity drive is plain xor.

The Unraid system is just Slackware and you can easily SSH in and do whatever you want to it. The Unraid developers interact a lot with the community and often promote community plugins and tutorials, so they’re very open to hacking and experimenting on the OS.

The main disadvantage of Unraid is that it’s “all-or-nothing”, as discussed in the next section. I’m not sure how comfortable Unraid is running in a VM, or if you always need to dedicate a physical host.

Complete package vs just redundancy

The first difference to understand is that “Unraid” refers to a whole operating system that provides redundancy, union mounting, networked file access, monitoring, scheduling, notifications, VM and Docker management, and a web GUI to configure it all. On the other hand SnapRAID is just a utility for adding redundancy to a set of existing mountpoints, and everything else must be provided elsewhere. OpenMediaVault with plugins can offer an Unraid-like experience for SnapRAID+mergerfs - see here.

Almost everyone using SnapRAID will want union mounting (the ability to view all the disks as one pool). SnapRAID comes with built-in support for a read-only union view, in the form of a command which will populate a pool directory with symlinks to the actual files. Most users will want to replace this with something more powerful that can offer live updates and write support. The most popular solution is mergerfs, though I think Windows users often go with Stablebit Drivepool. I wrote a post comparing the write strategies of mergerfs and Unraid, but the executive summary is that union mounting is a fairly solved problem and both Unraid and mergerfs do it well. Mergerfs is more powerful, but assumes more expertise from users.

Another thing that you’ll really want to setup is SMART monitoring of your drives with email alerts. Unraid has this out-of-the-box. For custom SnapRAID systems you’ll probably want to use smartd and your preferred MTA or other notification dispatcher.

The flexibility of being able to add SnapRAID to an existing system allows me to use one machine as both a NAS and a media centre connected straight to my TV. When I was on Unraid I had to use a separate Raspberry Pi as the media client, which added the complication of network sharing to Kodi on the Pi. If I had had a spare discrete graphics card I may have been able to pass it through to a media client VM on Unraid, and kept the integrated graphics for the root console, but that is quite a bit of extra time and money to spend.

TL;DR

Blocks vs files

Another key difference is that Unraid works on block devices while SnapRAID works on files. This makes SnapRAID more convenient, but requires somewhat more resources.

Unraid is more like mdadm, where you take some raw block devices, create virtual RAID devices, and then work on top of those. The Unraid OS only supports XFS or Btrfs with optional LUKS, but in theory I expect the storage virtualisation layer could support any FS. This also means that adding a new disk requires a parity sync; the way this is usually done is by “pre-clearing” the new disk to all zeroes, since a xor 0 = a, meaning that the existing disks do not need updating. Unfortunately this means you can’t officially add a full disk to Unraid, though I expect it should be theoretically possible if you do a full parity sync afterwards. The other disadvantage of this is that a parity sync or check time is proportional to the size of your largest data drive, regardless of how much actual space is used.

SnapRAID OTOH does not really care about filesystems or block devices; it just works on directories, and is so relaxed about semantics that it even runs fine on Windows. You can start off with full drives, as I did when I moved from Unraid to SnapRAID. Even the parity itself is just plain files rather than requiring a raw block device, as Unraid does, and will be proportional to actual usage, meaning you can take a risk and temporarily overcommit a parity drive smaller than a data drive, as long as you don’t use more storage on a single drive than the parity drive can hold. Technically the parity file cannot use the whole disk, due to the FS overhead, but the data disks also have that “issue”, and in any case it should be a minor overhead.

The downside of the file parity is less efficient use of resources. SnapRAID can use quite a lot of RAM when performing a sync, as it has to hold data structures for the more complicated way its parity works, compared to A[x] xor B[x] = P[x]. The other major issue is that of fragmentation. Having files smaller than the parity block_size results in more space being allocated for parity than the file actually uses. If you have a lot of small files (e.g. photos), this can add up to a significant amount of space wasted. You can decrease the block size, but that increases the RAM used (see the manual, section 7.8). When I ran into this, I calculated that it was cheaper to have to spend a bit more on disks than upgrade my RAM. It’s also generally ill-advised to fill disks to the brim on any system, but keep in mind that just filling one data disk can induce this problem.

Live vs snapshot parity

The main difference between Unraid’s storage system and SnapRAID is that Unraid runs as a daemon, constantly maintaining the virtual devices discussed previously, while SnapRAID is actually just a set of terminating commands that you run manually or on a schedule. You add/delete/update files and run snapraid sync. If a disk fails, you install, format, and mount a new one, then tell SnapRAID to start restoring the files onto it. You won’t have access to all your files until this finishes. This also means your new files are unprotected until you run a sync, and if you delete files from one drive, you remove protection from some files on other drives until you sync again, though the integrity features described later mean it should never result in silent corruption of restored files.

Unraid is more like a traditional RAID setup. Parity is written before returning “success” to the writing program. Normal reads on a healthy array only use one data disk, but when a disk fails the system will immediately start using the other disks to emulate it from parity, allowing you to continue using the whole array without interruption. Ironically this means that Unraid is more “real” RAID than SnapRAID IMO, since I consider high-availability to be the main purpose of RAID.

This does mean that a file deleted from Unraid is deleted immediately, while SnapRAID allows you until the next sync to notice your mistake and restore it, although this is only certain to work if the mistake was confined to one disk (otherwise you may have deleted files from other disks which were providing each other with parity). This should be considered a little bonus rather than something to rely on. If you really need that, use a filesystem with proper snapshots like ZFS, and remember that snapshots and disk redundancy are still not a substitute for offsite backup.

The mover

Writing to hard drives can be slow, especially since Unraid array writes are limited by the speed of the parity disk. Unraid provides a built in utility to use an SSD as a write cache. Files are initially written to the SSD, and on a schedule (by default, in the early hours of the morning), they are moved to the array proper. Unlike the rest of Unraid, this works at a file, rather than block, level. The SSD is included in the union mount so it’s essentially transparent to the user. Since files on the cache are not covered by array parity, there is built in support for using two SSDs in RAID1 using Btrfs.

SnapRAID doesn’t have anything like this, but the concept is fairly simple, so it shouldn’t be too hard to write your own script using rsync and cron or systemd timers, or just do it by hand. Your writes aren’t bottlenecked by parity drives in the first place, due to the snapshot model.

Scrubbing and integrity

Both systems can perform scrubs/parity checks, however Unraid is more limited.

On a base install, Unraid parity checks are all-or-nothing, while SnapRAID defaults to scanning 8% of the files with the oldest scrub date, meaning that every file is checked every three months if you run the command weekly. As discussed before, Unraid has to scan the whole block device while SnapRAID just scans actual files.

Unraid suffers from the same problem as traditional RAID setups: if the calculated parity does not match the stored parity, is it the data or the parity which is wrong? SnapRAID is more similar to ZFS in that it stores actual file checksums, so can always answer correctly (beyond reasonable doubt). Presumably using dual parity on Unraid should help here.

Silent data corruption on hard disks is often considered to be virtually a myth, since it would be very unlikely to happen in such a way that the hard disk’s own error correction doesn’t detect it and return a media error, however it’s not the only way that the parity can become desynced. I believe Unraid is vulnerable to a “write hole” if power is lost during a write, whereas SnapRAID performs its syncs in a transactional manner, so can safely resume them.

As a bonus, SnapRAID also provides a dup command to list duplicate files, and also supports hard links, so it may be possible to leverage these for some limited deduplication support.

Summary

Both are great solutions. Unraid is more opinionated while SnapRAID is more flexible. There are tools to simplify SnapRAID setup, giving it an Unraid-like UX, but the reverse - using Unraid as a facet of an existing Linux install - is not possible. SnapRAID’s unique advantage is the file-based model, while Unraid’s is the fact that it provides high availability like a traditional RAID system. The checksumming built into SnapRAID is nice, but there are Unraid plugins that do similar things.

I intend to write a more thorough comparison of these two solutions in the future, but the short summary relevant to this article is that they use plain old filesystems like XFS on each disk and mount them independently, then provide a writeable union mount view. This isn’t as performant as “real” RAID, but for the scenario of media storage that’s usually not an issue, and in fact has the advantage of only spinning one disk at a time.

Creating a read-only union mount is simple enough, but when you add writeability into the equation you have to decide on a strategy to select a drive to write new files to. In theory this shouldn’t matter, but in practice I want to colocate whole TV shows onto one drive, so that a data loss scenario does not result in me having half of 10 shows rather than 5 complete and 5 completely missing shows. I also wanted to write some code and draw some pretty graphs.

Each solution offers a different unique feature that can help with this. Unraid has a “maximum split level”. For example if your schema is share root/TV/Show Name/Season 1/episode.mkv, then you could set the maximum split to two to colocate entire series, or three to colocate only seasons. Mergerfs includes “most-shared path” variants of most strategies, which will try to put new files in an existing directory on the same drive, then move up a level and try again if they won’t fit.

When using SnapRAID, one potential issue with strategies that fill disks unevenly is increasing the likelihood of wasted parity space. See this post, CTRL+F “fragmentation”.

In either case, the raw storage devices are exposed too, so you can always manually manage where things are put. Unraid comes with a web interface to show you where files are allocated, and the third-party “Unbalance” plugin makes it easy to relocate them. Mergerfs is a little more manual. I execute this script that I wrote in the root of the TV shows directory, and then manually use Midnight Commander to move things. I’m planning to work on a more automated solution.

Bin packing

The problem of selecting a bin (drive) to put an item (file) in is known as the bin-packing problem; specifically, the online variant, meaning we don’t have foresight of the files that will come in the future and we’d rather not move them once allocated.

The formal problem described in that article assumes all bins are the same size, which is not the case with my setup. It also considers items of radically different sizes and tries to optimise for the maximum total storage. This is not something I’m concerned with, since the drive are usually so much bigger than the files, and you’re ideally leaving them at <90% usage, so efficiency differences should be negligible.

The strategies are known by different names, so here’s a terminology table. Remember that lus and mfs are equivalent if all bins are the same size.

I wrote a little utility to simulate the different strategies of Unraid and mergerfs. Its main flaw is that it does not test the path-preservation feature of mergerfs, since it simulates all the files being dropped in the same directory.

I tested it using the scenario of storing 4GB files into drives with sizes (1, 4, 4, 2)TB, until they’re full. The key thing I was looking for in these plots is the separation of the lines. In many strategies the orange and green, and red and blue, lines follow almost the same trajectory, meaning that files are being alternatively assigned between the two during some periods, which would manifest as a fragmented TV show.

Now then

First

The simplest strategy, and does a decent job, as long as you don’t mind your drives being used somewhat disproportionally.

Least-free space

This is what I’m currently using on my setup.

Least-used space

This makes good proportionate use of the drives, but has poor separation of drives 1 and 2.

Most-free space

Good proportionate use, but pretty much the worst strategy for colocation.

High-water

This is an Unraid-specific strategy. Quoting from the UI

High-water Choose the lowest numbered disk with free space still above the current high water mark. The high water mark is initialized with the size of the largest data disk divided by 2. If no disk has free space above the current high water mark, divide the high water mark by 2 and choose again.

The goal of High-water is to write as much data as possible to each disk (in order to minimize how often disks need to be spun up), while at the same time, try to keep the same amount of free space on each disk (in order to distribute data evenly across the array).

This appears to fulfill its stated goal, making approximately equal use of drives and having good separation of lines. I’d probably use this if I were on Unraid.

Random

Basically a non-deterministic version of LUS.

Percentage-Free Random Distribution

An interesting strategy, but not good for my use case.

Conclusion

So in short, I would recommend

• If you’re on Unraid just use high-water tbh. I think it’s the default too
• If you want colocation, first off make use of the features like maximum split or path preservation that I described earlier, then use LFS or first
• If you want equal usage of drives (to reduce fragmentation), use LUS