I have been tracking the "binpkg-multi-instance" bug in Portage since 2007 when I was looking to use binpkgs for a slew of VMs. They tended to be grouped such that each group needed different use flags for the same packages - a perfect use case. Now that zmedico's binpkg-multi-instances feature work is in a stable version of portage (2.2.20) there has been some small discussion on the original bug as to the best way to use/automate this feature.
A few months ago, after some discussion with cchildress, who has been tracking the bug for the same reason, I started some work on a "reserve scheduler". The basic idea is the opposite of what is found in HPC schedulers - rather than schedule a queue of jobs on many hosts to run in parallel, the scheduler needs to serialize many requests on a single host. I chose a general solution where commands are run remotely rather than some form of IPC. This is riskier but is more flexible and intended for use in a closed environment. Most significant though is when a build or job is complete, the requester is notified and can execute a script. This is very useful for chaining events such as installing a freshly built package.
I have a very primitive, working command queue or cq that can handle build requests: cq on github I have used it successfully and repeatedly in my testing environment. However, it's not really documented yet and it's useful for more than just a build host so it's own documentation might take another form. I'll try to document a good use of it with the Portage binpkg-multi-instance feature here.
For this to work some shared filesystem or a mechanism to deliver packages to clients (NFS/web server/FTP/etc) is necessary but I'm assuming that's already in place. Add binpkg-multi-instance to make.conf on all the involved hosts - clients and the build host. Add my junkdrawer overlay and unmask app-admin/cq. This will also pull in an updated version of munge. Once installed, ensure a consistent munge key is installed on all the hosts (/etc/munge/munge.key) and has safe permissions (400) and start munge.
At the time of writing cq_server does not daemonize. Just run cq_server, optionally with -h or -p flags for host and ports unless the default 0.0.0.0:48005 is acceptable. (I intend to push changes soon so it will run as a daemon and log to file(s)/syslog.)
On the client side some scripting is required for automation but the basic idea is to run cq_client with the following options:
- -h host
- -p port (optional)
- -P script (optional)
- -E port (optional)
- -- command string
Multiple -E flags may be specified for things like USE, ABI_X86, etc while -P can be a local script to run once the build is finished (such as an install script that calls emerge). The script will receive two parameters. The first is the internal status from cq (did we succeed in running the command remotely). I propose a script that will take (or discover) packages to be updated with their appropriate environment, generate an install script, and then run cq_client with the appropriate options. (This script left as an exercise, blah, blah, blah...)
There is still a good amount of work to be done. A few planned items are recorded in the README. Other theorized improvements include:
- A hook for incoming commands - when building packages it could be useful for finding duplicate requests in the queue and eliminating them.
- Multiple build hosts - the architecture already supports this (for a different reason) but some small changes would be necessary to add options to enable it.