A look at Zef, a perl6 package manager, and how it works.

Everything has to start somewhere. A Perl6 package's story usually starts at its META.info file, a file containing various bits of information that describe the package and what it contains. So for any package we should expect a META.info file, and it is from this which we create our base class: Zef::Distribution^[1]. In addition to containing the META information, it serves to:

1. Normalize certain values (version/vers, auth/auth/authority, and their combinations)
1. Save the base path of the package so we can reconstruct absolute paths
1. Helper methods/attributes to transform meta information, generally related to using #2 to handle and handling relative file paths.

It can be thought of as a mix of Distribution^[2] and CompUnitRepo::Locally^[3]. Even better, it appears that in the future they will provide most of this functionality (or make it extremely trivial to implement). This means one more thing (or two more depending on how you look at it) that people smarter than me will maintain, a win/win.

# assuming Zef's META.info is located at /tmp/Zef/META.info
my $dist = Zef::Distribution.new( path => '/tmp/Zef' );

Now we have our $dist, but right now it doesn't do much. We will want to do various combinations of precompile/test/report/install, while also taking hooks into account. In the interest of curiosity, when a user does zef test we do not want to have any install related stuff inherited. Roles^[4] will make this easy for us.

Parallelization was/is a primary goal of Zef. On one hand it could use Perl6's built in concurrency features to do pieces of work. On the other we could spawn separate processes. The former was the first choice, but in the end spawning separate processes was chosen. One factor involved in this was that some modules were able to crash in such a way that they would pollute the rest of the test run. By spawning a separate process we could avoid such problems.

my $dist = Zef::Distribution.new( path => '/tmp/Zef' );
$dist does Zef::Roles::Testing;

$dist now can do whatever Zef::Roles::Testing allows. Initially it was to provide a test method (and others would provide the build, install, and so forth). However, another idea on the back burner was to be able to generate a stand alone installer authors can distribute. So instead these roles supply test-cmds() and precomp-cmds(). There is no install-cmds(), as we cannot do the actual install phase for multiple packages at once so the tuits are lacking, but ideally it will also move to a install-cmds(). This will make it easier to attempt creating 'the build/test/install as a stand alone script' generator. Here is what we are working with:

use Zef::Utils::PathTools;

role Zef::Roles::Testing {
    method test-cmds(Bool :$shuffle) {
        my $test-dir   := $.path.IO.child('t');
        my @test-files  = $test-dir.ls(:r, :f)\
            .grep(*.extension eq 't')\
            .map({ ?$_.IO.is-relative ?? $_.IO.relative !! $_.IO.relative($.path).IO });
        @test-files = ?$shuffle ?? @test-files.pick(*) !! @test-files.sort;

        my @cmds = @test-files.map: { [$*EXECUTABLE, $_] };
        return @cmds;
    }
}

Pretty simple. Look at the /t directory, create a list of all files with a .t extension, sort or shuffle that list, and finally generate the command necessary to execute each test file. We follow a nearly identical pattern for Zef::Roles::Precompiling^[5] and Zef::Roles::Hooking^[6], as they too are just generating command strings from a list of file paths.

To make use of the values from the various -cmds() methods we will use a simple process manager, Zef::Processing^[7]. For our purposes the process manager will handle grouping the appropriate processes for parallelization, as well as providing any process helper methods.

role Zef::Roles::Processing[Bool :$async] {
    has @.processes;

    method queue-processes(*@groups) {
    }

    method start-processes {
    }

    method passes     {  }
    method failures   {  }
}

For this post we do not need to go into greater detail here. We have created an easy way to handle groups of processes like [[A,B], [C], [D,E,F]] we can use with test-cmds(), precomp-cmds(), or hook-cmds()

hook-cmds()? For our implementation of hooks/^[7:1] we match the phase name (build, test, install, maybe more in the future) and time frame (before or after) against a basic file name template. before-install.pl6, after-test.pl6, and so forth. This will be improved further, such as hooks on success/failure. This improves on the current Build.pm method by:

1. Not requiring any dependencies (and thus no package manager)
1. Not composing an outside role into the package manager itself
1. Allows a module author to write hooks to implement the entire install procedure

So what does our package manager generally look like when we put the pieces together?

my $dist = Zef::Distribution.new(path => '/tmp/Zef');
$dist does Zef::Roles::Processing;
$dist does Zef::Roles::Testing;
$dist does Zef::Roles::Precompiling;
$dist does Zef::Roles::Installing;
$dist does Zef::Roles::Hooking;

# test + hooks
$dist.queue-processes: [$dist.hook-cmds(TEST, :before)],
                       [$dist.test-cmds],
                       [$dist.hook-cmds(TEST, :after)];
await $dist.start-processes;

# build + hooks
$dist.queue-processes: [$dist.hook-cmds(BUILD, :before)];
$dist.queue-processes($_) for $dist.precomp-cmds
$dist.queue-processes: [$dist.hook-cmds(BUILD, :after)];
await $dist.start-processes;

# install + hooks
$dist.queue-processes: [$dist.hook-cmds(INSTALL, :before)];
await $dist.start-processes;
$dist.install;
$dist.queue-processes: [$dist.hook-cmds(INSTALL, :after)];
await $dist.start-processes;

Yes, it is a little verbose, but the code is somewhat self explanatory while remaining highly configurable. You will notice the install + hooks is slightly different. As previously mentioned, install does not get executed in a separate process. So to properly execute the hooks for install (which are executed in a separate process) we have to call .start-processes on 2 occasions.