DHLP: Project Feed

DHLP: Product Pipelines

This is the new product pipeline map based on recent updates:

DHLP: Cyrois is now Dyad

Cyrois was a placeholder name for the new fork of Pyrois.

Pyrois will be responsible for generating the early builds of Dark Horse Linux and its source based system compilation.

Dyad, previously called Cyrois, will be responsible for providing a step by step automation of a variant of LFS, to better enable those who should wish to create their own Linux distributions not beholden to existing interests.

DHLP: Cyrios: Forking Pyrois

After a great deal of thought, I’ve decided to fork Pyrois. The new project is called Cyrios.

Cyrios will be kept under the SILO GROUP umbrella and not the Dark Horse Linux Project space. Its source code is available at:

https://source.silogroup.org/SILO-GROUP/Cyrios

I will at a later date set up a downstream mirror on github.

Pyrois will continue to be part of the Dark Horse Linux Project and will remain where it is, but will be specialized to produce Dark Horse Linux images.

For those of you who are visual thinkers, this is the current state of the project structure:

And this is the future state of the project structure:

The reason why I’ve decided to fork my own project is a bit nuanced, but in essence, the goal of Pyrois is both important to me and limiting of its utility to the Dark Horse Linux Project at the same time. This solves both angles.

Here are the highlights:

• Pyrois was successful in providing a way to build a non-specialized ISO image for someone to be able to extend to create a new Linux distribution image to just about any configuration imaginable. It’s supposed to be a “generic distribution factory” that doesn’t make too many assumptions about what you want to build beyond a standard source-based system that is cross-compiled from raw sources. While there are many projects that will produce a linux image, I’m not aware of any that do not provide a highly specialized system beholden to upstream project interests with large sponsors behind them.
• Pyrois is currently used by Dark Horse Linux to create its images. Now that Dark Horse has reached a certain level of maturity in planning, I need to extend Pyrois to be specialized to generate the Dark Horse Linux image, and specifically an installer ISO.
• I didn’t want to destroy Pyrois’ current facilitation of distribution genesis by just specializing it to DHLP, so I’m forking it to a new project called Cyrios to serve that function.
• Cyrios won’t be able to get as many resources as Pyrois, so, I’m expecting it to largely stay out of date but to provide a base on which others can build by refreshing it using the LFS documentation.
• This will pave the way for a more developed DHLP image and eventually an installer image.

DHLP: Reduced ISO Size and Improved Build Safety

I was able to reduce the generated livecd ISO image from its eyesore of 8GB down to just under 2GB, putting the ISO size for the Dark Horse Linux livecd on par with other distributions.

You can grab the latest image here.

The issue was ultimately an overzealousness to make the image “provable” from a security perspective. The debuginfo in the compiled binaries of the system, particularly the kernel modules and shared libraries, accounted for about 6GB of the storage. Stripping that debuginfo consisted of almost all of the size reduction. 75% size reduction is really quite worth it.

This version also has some repaired filepaths, as well as additional safety in the scripts that generate the image.

You’ll probably see in the git logs that there are a bunch of fairly recent additions of set -u in the bash executables that Rex is kicking off. This tells bash to treat any reference to an unset variable as an error and exit immediately. That way if you do the admittedly dumb rm -Rf ${unset_path}/${also_unset_path}, bash will refuse to execute that line and fail the project run of rex due to a non-zero exit code. Obviously this is a stop-gap measure until I can do a full cycle focused on code safety rewrites.

Next is (besides some more clean up on the pyrois codebase) the introduction of RPM and presumably DNF, followed by some formation of what an installer ISO will look like.

It would potentially be a turning point for this project as it could be the point at which the distro moves from source based to precompiled binary package based, depending on how much infrastructure gets introduced during the RPM and DNF build/research work.

Make sure and subscribe to the new mailing list for updates as well.

DHLP: Email, Mailing lists and More

As part of the productization effort of Dark Horse Linux, one of the things that needed to be done is getting Dark Horse Linux resources off the SILO GROUP domain.

One of those included email. Another included mailing lists.

So, I’m bringing mailing lists back. If you’re looking for how to contribute, the dhlp-contributors mailing list is a great place to ask questions and find out about the project:

https://lists.darkhorselinux.org/mailman/listinfo/dhlp-contributors

If you want to contact someone personally, I can be reached at:

chris (dot) punches (at) darkhorselinux (dot) org

As always, I’m eager to bring more people onto the project, so if you know of any potential volunteers, please feel free to point them at that mailing list or to me directly.

I think I may be running out of excuses to focus on the installer and bringing in RPM.

Phanes' Canon: Kaizen

Well, I’m finally where I can take a couple days off and kick back.

I was evaluating where things are from a year ago, and three years ago, and five years ago, and my God, what a few years this has been!

And I am so tired that just sleeping in for a few days isn’t going to do it.

Dark Horse Linux

I got a new website up for Dark Horse Linux and there are a few issues to work out, but, there always will be. It needs a documentation generation system that’s themable and web-friendly, but also supports many formats.

For the immediate next cycle:

• I’ve decided to tackle the ISO size issue
• I will be putting RPM onto it, to open the door to all the cool things I’ve got planned.
• Along the way will be a new pass at the Pyrois project to clean up all the code along the way to make some of those commands a little safer to the build system, clean up dependencies, etc.
• This will involve either a tag and branch of Pyrois or a kind of snapshot fork to represent a generic distribution generator that is very close to LFS.

This will position that project nicely to start approaching both a package and patching ecosystem, as well as give users something they can builld on top of a little more easily for those spin-off possibilities.

Then, it’s installer, installer, installer and documentation, documentation, documentation.

Everything Else

I’m not taking on any new projects. In fact I might be trimming down the project list to focus more on DHLP, though, I really do want to sink my teeth into SMQ/S.

DHLP: New Website

As a result of community feedback, I have created a new website for dark horse linux, and it is in production now.

I am lukewarm with the results, but it does meet the intended purpose for now.

The documentation system is an acknowledged gap, one that presents an eyesore, and I’m not sure how I’ll approach it yet.

I’m not 100% convinced I should be writing the documentation so much as overseeing community contributions to that and ensuring the content is accurate.

Phanes' Canon: New Website Coming for Dark Horse Linux

I’ve hardened a preprod and prod environment for hosting the new site.

I’m about 3/4 of the way through the creation of the static site generator and the initial theme. Content is another story.

Interestingly enough, this is to address community feedback about the admittedly horrid theme on www.darkhorselinux.org currently — which is ironically based directly off the slackware.com website, after fixing it to make it valid html (the slackware.com website is not valid html).

But, feedback is feedback.

I really do need to get some other party involved on the theme stuff as HTML/CSS is really not something I enjoy doing or am even good at.

This stuff takes forever and I want to be spending this time improving the actual OS or writing some much needed documentation.

Phanes' Canon: Team Chris is Sleepy, and other Ramblings…

Summer keeps teasing me. It’ll come in hot and strong and I’ll rev up and have a great productive day. The next, it’s bleak, grey winds that cut through you and it’s cold again.

Can’t win with this. I get a crippling seasonal depression in the colder months. Maybe it’s time to move further south.

Hopefully we’ll get a “Brief History of the World Part 2” before Mel Brooks gets too old.

Work is still way too busy to put serious time into anything.

I’m working on a new static generator made to produce whole websites instead of just feed aggregation. This should be useful for improving the Dark Horse Linux website. Don’t know what to do about theming, though, that’s just really not something I want to learn how to do too well.

Stay tuned…

DHLP: New LiveCD ISO

Within the next hour or so, I’ll be uploading a new livecd to the downloads section of the site. It is a rather clean and faithful production of an LFS system, with overlayfs added as a dracut module so that the livecd can actually be used as a system with a writable root filesystem.

Changes won’t be persistent, but, this is a much better proof of work than what was up before it.

At this point I do encourage people to try it by downloading it and firing it up in an emulator and provide feedback. I’d be thrilled to hear someone got it running on a physical machine.

While cool in and of itself, this paves the way for transitioning the image to an installer ISO, where Pyrois copies the sysroot to a subdirectory in itself to put on to a target machine to boot from local disk after some basic configuration prompts.

ISO Image file size is still an issue at 8GB. I’ll have to depart from LFS to fix that. So, this might be the “goodbye LFS” post I’ve been building up to. Not sure yet.

Phanes' Canon: The Next Phase

I am still resting up in between the current next phase of Pyrois / Dark Horse Linux to work on other projects so I don’t fall behind.

Aside from the improvement of themes for DarkFeed, which is really out of my wheelhouse as I’m not a graphic designer, I’m also looking to build out a basic working time series prediction engine that works off data pulled from yahoo finance to test out some of the newer LSTM models’ capabilities.

I may also soon start to finally build pieces of the SMQ/S together now that I’ve refound the Raft protocol.

I may need to take another vacation before then, as between $dayjob and everything else I’m getting worn out pretty well.

Bing art is cool.

DHLP: Progress Update

I am excited to announce a milestone in the Dark Horse Linux project.

The code in the repo has finished creating the initial target system, including the kernel compilation using the default Fedora Linux kernel config.

There is an exception, the fstab file, which has a chicken-egg problem to resolve but I consider this to be minor and can be resolved during the ISO file generation, which is the next piece.

I should be able to use grub-mkrescue or genisoimage/xorriso or some combination of these with a thought out init ram disk to complete that part.

Once complete, I’ll start polishing the codebase for Pyrois up, branding a fork of it for ALFS-NG, and tag and release that for now.

For the long term, the `ALFS-NG` spinoff is just a side project that got spawned by this effort. I haven’t decided if this is something I want to give to the LFS folks to revive/modernize their seemingly abandoned ALFS project, or if it should remain wholly separate due to the Darcy infection. I have a desire to “play well with others”, but, that can’t occur in a vacuum, and I need to be certain it stays shielded from Canonical based on previous behaviours.

On the other hand, it’s likely to go very stale if I keep it under my thumb as it’s not a priority for me to keep it up to date more than to facilitate it as an instructional device. I suppose that begs the consideration that they’ve already abandoned that project once, so, it may not be a priority for them either. They likely wouldn’t want it.

I will then want to decide whether I want to introduce librpm for the next phase, or whether I want to create an installer disk, presumably with something like `anaconda`, though, I am hesitant about introducing an upstream dependency on the RH ecosystem based on the goals of the DHLP project.

I will want this system to be very familiar to RH-based distro users for many reasons, but, all cooperation with external projects must be a voluntary effort that can be easily severed for this mission to work. This is, after all, a contagion firewall. I’m already going the librpm route, which is maintained by RH, but, many distros use librpm. Not many use anaconda, so, their maneuvreability is different with that piece. Not that I think anything would happen there, but, it’s a risk surface that needs gaurded.

I suppose I’ve answered my own question thinking it out — anaconda is a package-based distro installer, so, if I’m going the librpm route I need to do that first before anaconda.

I suppose one approach might be to build my own basic installer that is a modification of the sysroot I’ve created that just copies the unmodified sysroot to a target system after the user selects and mounts all their target mounts. That’s a thought. Copy it off, when it’s done in the build, then engage in a 6th stage that turns one of the copies into an installer that puts the intact copy onto the target system.

I may do that first to move forward on a known working path, and then do my reductionist cross-reference with other similar projects to synthesize a new process. It’s a bit of a longer path but it’s one that can provide consistent progress.

Of course, things are always subject to pivot based on what I learn as I go through this. I have no experience with this aspect of things, so, I’m certainly open to ideas.

DHLP: Entering Chapter 9

Automation of the 11.3-systemd rc1 has been successful on a Fedora build system up to Chapter 9.

At this point what’s left is:

• populate some of the system config files (/etc/*)
• set up udev
• set up the timezone
• some minor systemd disablement
• set the system locale
• compile the kernel
• set up kernel module load orders
• configure grub to be on an iso and install the bootloader to that iso (this will require deviation from the LFS book)

The fun part is that I’m not going to be using stock values for these. It’ll give you curses dialogs asking for input on alot of the stuff you’d normally configure on your OS, though, I may be reserving alot of that implementation for an “installer ISO” so that the values aren’t baked into the ISO in future versions.

After that, it’s an RPM variant.

After that it’s a reductionist rewrite cross-referenced with other unrelated projects to create a more minimal system without sacrificing on the core components.

All in all, looking back through the work, this process would have only taken a couple days, maybe a few hours for someone who’s done it before, but, automating it — I think that’s where this project’s value is.

I would like to see someone fork it, containerize the process, and then use pull requests with automated unit tests to accelerate OS development.

In the meantime once this ALFS-NG baby project is wrapped up, I’ll rebrand it as that, and then use the Dark Horse Linux label to build out the real deal.

DHLP: Project Direction Changes

After some review of the dual licensing of LFS, it’s occured to me I need more segmentation in my planning for how to get to a new distro that I can offer commercial support for and distribute for commercial use.

That’s not the core reason for doing this, but, I’d like to be compensated eventually for all the hard work, and perhaps hire people and build a company around it.

I think in order to get there, I’ll need to make some pivots on approach on this.

To get around the non-commercial shit in LFS’ licensing, I’ll probably, once I get to a releasable state, release this build as a project called “ALFS- NG” that inherits the crap they snuck in there and have DHLP be a non-derivative rewrite.

I’m still thinking about it, but, if that’s the direction, once ALFS-NG is good to a certain point, I’ll be able to apply what I’ve learned to Dark Horse Linux. I feel as I’m going through this process that LFS has many, many unnecessary steps to do something like this.

I may even do a second build with one of the RPM variants of LFS/BLFS to get an idea of what I’m in for in terms of introducing binary package management cleanly.

Phanes' Canon: Dark Horse Linux: Stage 4

At this point, DHLP is compiling up to LFS CH. 8.

This is the point where it finally starts to get “real”, as we’ve finished the compilation of all the temporary tools and are starting to build the “real” permanent system.

This is also an exciting part because I got to introduce the first of those classic ncurses dialog prompts.

Stay tuned.

Phanes' Canon: Plans for Dark Horse Linux

So, as it’s nearing a bootable ISO, some stages of development seem to be forming in terms of expectations.

A Whole Image

First, I’ll end up with something very close to an LFS build that needs alot of work to get installed onto a different system.

Gentooesque Genesis, Slackwaresque Form

Then, I’ll end up with some kind of installer that seems like it will work very similarly to slackware’s and FreeBSD’s installer, but compiled in a very gentoo kind of way beforehand. I see this piece taking quite a bit of time to get through if I pursue that as far would be expected. The important pieces will be drive formatting/partitioning dialogs and tools, bootloader installation. Copying to the mounts appropriately etc.

A Familiar Set of Tools

Some ways after that, unless it doesn’t work the way I expect, I’ll probably pivot to an rpm-based system that, a certain point after librpm is compiled, binary packages are downloaded that replace the packages already compiled and installed. Systemd. Firewalld. Then the decision needs made about whether I’ll use DNF or an inhouse built package manager. One drawback I see about this is, some users will probably use rpms from SUSE and RH and Rocky and Fedora and run into build compatibility problems. On the other hand, some of the time it’ll actually work with a little tweaking of binaries. On the other hand, using a familiar ecosystem for the core pieces of the system will bring with them the stability those pieces are known for, they are tried and true, and will be already familiar by many, many users. So there’s a tradeoff to consider. It also prevents me from having to reinvent package management. Life’s just too short and I’m one person.

On Package Management

It’s dangerous to go alone.

Take this.

Leaving it without package management is just not an option in the currrent security theatre. Modern systems need not just build systems for distributable packages but patching systems built in with pipelines established for those patches to reduce overhead. Patches that are free, open source, and available in other package-based distributions. In some cases, with associated CVEs. This will require lots of infrastructure to do sustainably even if I weren’t just one person.

I think what I ought to do is find some way to faciliate that infrastructure, designed in such a way where it waits for contributors to do the things. So, if someone thinks its too slow, a pull request from a stranger (or a familiar maintainer, even) gets submitted so that I don’t have to do the work. It gets reviewed and merged. The more people involved the better, but, it /could/ work with one person, just very slowly.