March 30 2020

Linux kernel 5.6.0 iwlwifi bug

Mike Pagano (mpagano) March 30, 2020, 13:27

Quick note that the Linux kernel 5.6.0 has an iwlwifi bug that will prevent network connectivity. [1]

A patch is out but did not make 5.6.0. This patch IS included in gentoo-sources-5.6.0. It will be in a future vanilla-sources 5.6.X once upstream releases a new version.

[1] www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-Broken-Intel-IWLWIFI

[2] git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/patch/?id=be8c827f50a0bcd56361b31ada11dc0a3c2fd240

Quick note that the Linux kernel 5.6.0 has an iwlwifi bug that will prevent network connectivity. [1]

A patch is out but did not make 5.6.0. This patch IS included in gentoo-sources-5.6.0. It will be in a future vanilla-sources 5.6.X once upstream releases a new version.

[1] https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-Broken-Intel-IWLWIFI

[2] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/patch/?id=be8c827f50a0bcd56361b31ada11dc0a3c2fd240

February 27 2020

Amavisd crashes immediately on start with little in the logs

Nathan Zachary (nathanzachary) February 27, 2020, 17:02

Recently I was updating amavisd and other portions of the mail stack on one of my mail servers (running Gentoo Linux). This particular set of updates didn’t go as smoothly as I would have liked primarily due to the acct-user/amavis package being created on Thursday, 13 February 2020 as a part of the ongoing effort to standardise user accounts used within applications. Some other problems with the upgrade of amavisd (to version 2.12.0-r1) and the mail stack were that some dependencies were erroneously removed from the ebuild (bug 710842) and (though not directly related) permissions within the new amavis user’s home directory being inadequate for ClamAV to successfully run. I will address these problems and their respective fixes below.

Missing dependencies in 2.12.0-r1 (bug 710842)

Some of the Perl modules (since amavisd-new is written in Perl) were erroneously removed from the dependency list in version 2.12.0-r1. The fix here was to simply reinstall the modules manually, which in my case entailed:

emerge -av MIME-tools Net-Server Mail-DKIM MailTools Net-LibIDN

However, my situation required one additional module that wasn’t mentioned in the bug, and was discovered when trying to manually start amavisd because of this error message:

# /usr/sbin/amavisd
Problem in Amavis::Unpackers code: Can't locate Archive/Zip.pm in @INC (you may need to install the Archive::Zip module) (@INC contains: /etc/perl /usr/local/lib64/perl5/5.30.1/x86_64-linux /usr/local/lib64/perl5/5.30.1 /usr/lib64/perl5/vendor_perl/5.30.1/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.1 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl/5.30.0/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.0 /usr/lib64/perl5/vendor_perl/5.28.0 /usr/lib64/perl5/vendor_perl/5.26.2 /usr/lib64/perl5/vendor_perl/5.26.1 /usr/lib64/perl5/vendor_perl/5.24.0 /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/5.30.1/x86_64-linux /usr/lib64/perl5/5.30.1) at (eval 101) line 47.
BEGIN failed--compilation aborted at (eval 101) line 47.

That one was also simple enough to solve by installing dev-perl/Archive-Zip:

emerge -av dev-perl/Archive-Zip

Fortunately, this ‘missing dependencies’ problem was fixed with version 2.12.0-r2.

Changes in the ‘amavis’ user’s home directory

With the addition of the acct-user/amavis package, a few things changes about the default setup for the ‘amavis’ user. When installing that package, Portage warns about some steps that need to be implemented in order to make amavis work again after migrating its home directory from /var/amavis/ to /var/lib/amavishome. Even after following those steps, I saw the following error message when starting amavis:

Feb 27 00:08:31 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:08:31 [amavis] will bind to /var/amavis/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Amavis (type Net::Server::PreForkSimple) starting! pid(1738)
Feb 27 00:08:31 [amavis] Net::Server: Binding to UNIX socket file "/var/amavis/amavisd.sock"
Feb 27 00:08:31 [amavis] (!)Net::Server: 2020/02/27-00:08:31 Can't connect to UNIX socket at file /var/amavis/amavisd.sock [No such file or directory]\n  at line 66 in file /usr/lib64/perl5/vendor_perl/5.30.1/Net/Server/Proto/UNIX.pm
Feb 27 00:08:31 [amavis] sd_notify (no socket): STOPPING=1\nSTATUS=Server rundown, notifying child processes.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Server closing!
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Child processes have been stopped.

In that error message, I noticed that there were still references to /var/amavis/ instead of the new /var/lib/amavishome/ directory, so I updated it using the $MYHOME variable in /etc/amavisd.conf:

# grep -e '^$MYHOME' /etc/amavisd.conf 
$MYHOME = '/var/lib/amavishome';   # a convenient default for other settings, -H

Thereafter, the start-up logs indicated that it was binding to a UNIX socket in the correct home directory:

Feb 27 00:12:15 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:12:15 [amavis] will bind to /var/lib/amavishome/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:12:15 [amavis] Net::Server: 2020/02/27-00:12:15 Amavis (type Net::Server::PreForkSimple) starting! pid(1952)
Feb 27 00:12:15 [amavis] Net::Server: Binding to UNIX socket file "/var/lib/amavishome/amavisd.sock"
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host 127.0.0.1 with IPv4
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host ::1 with IPv6
ClamAV permissions within the amavis home directory:

Though this error wasn’t directly related to the upgrades (and had likely existed for quite some time beforehand), I only just now noticed it whilst combing through the logs (the first error is from my system’s mail log, and the second error is from the clamd log):

Feb 27 00:15:10 [amavis] (01980-01) run_av (ClamAV-clamd) result: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n
Feb 27 00:15:10 [amavis] (01980-01) (!)run_av (ClamAV-clamd) FAILED - unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n"
Feb 27 00:15:10 [amavis] (01980-01) (!)ClamAV-clamd av-scanner FAILED: CODE(0x5611198fa5d8) unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n" at (eval 85) line 951.
# grep 'lstat' /var/log/clamav/clamd.log
Wed Feb 26 23:26:53 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180007-00529-S1gbY8cd/parts
Wed Feb 26 23:28:32 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180038-00592-FK2_Uj2T/parts
Wed Feb 26 23:31:30 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T165427-32346-V4WeP0YX/parts
Thu Feb 27 00:15:10 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts
Thu Feb 27 00:26:21 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002621-01981-l36mWT4P/parts
Thu Feb 27 00:26:30 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002630-01982-k0qgJdjl/parts
Thu Feb 27 00:29:12 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002912-01983-G57aKCmK/parts
Thu Feb 27 00:31:55 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003155-01984-M9r9r1Gc/parts
Thu Feb 27 00:33:07 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003307-01985-8n_wS6pQ/parts
Thu Feb 27 00:40:20 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004020-01986-gH4PrFAk/parts
Thu Feb 27 00:45:45 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004545-01987-J4YGuNbw/parts

These errors were fixed by 1) adding the ‘clamav’ user to the ‘amavis’ group, 2) setting g+w permissions on the /var/lib/amavishome/tmp/ directory, and 3) restarting clamd and amavisd

# gpasswd -a clamav amavis
Adding user clamav to group amavis

# ls -alh /var/lib/amavishome/ | grep tmp
drwxr-xr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# chmod 775 /var/lib/amavishome/tmp

# ls -alh /var/lib/amavishome/ | grep tmp
drwxrwxr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# /etc/init.d/clamd restart && /etc/init.d/amavisd restart

Now ClamAV is readily able to access files under /var/lib/amavishome/tmp/:

Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Connecting to socket  /var/run/clamav/clamd.sock
Feb 27 01:00:12 [amavis] (03346-01) new socket by IO::Socket::UNIX to /var/run/clamav/clamd.sock, timeout set to 10
Feb 27 01:00:12 [amavis] (03346-01) connected to /var/run/clamav/clamd.sock successfully
Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Sending CONTSCAN /var/lib/amavishome/tmp/amavis-20200227T010012-03346-YjbUpYOk/parts\n to socket /var/run/clamav/clamd.sock

Hopefully, if you run into these errors, you will be able to take this information and apply it to your particular mail stack.

Cheers,

Zach

Recently I was updating amavisd and other portions of the mail stack on one of my mail servers (running Gentoo Linux). This particular set of updates didn’t go as smoothly as I would have liked primarily due to the acct-user/amavis package being created on Thursday, 13 February 2020 as a part of the ongoing effort to standardise user accounts used within applications. Some other problems with the upgrade of amavisd (to version 2.12.0-r1) and the mail stack were that some dependencies were erroneously removed from the ebuild (bug 710842) and (though not directly related) permissions within the new amavis user’s home directory being inadequate for ClamAV to successfully run. I will address these problems and their respective fixes below.

Missing dependencies in 2.12.0-r1 (bug 710842)

Some of the Perl modules (since amavisd-new is written in Perl) were erroneously removed from the dependency list in version 2.12.0-r1. The fix here was to simply reinstall the modules manually, which in my case entailed:

emerge -av MIME-tools Net-Server Mail-DKIM MailTools Net-LibIDN

However, my situation required one additional module that wasn’t mentioned in the bug, and was discovered when trying to manually start amavisd because of this error message:

# /usr/sbin/amavisd
Problem in Amavis::Unpackers code: Can't locate Archive/Zip.pm in @INC (you may need to install the Archive::Zip module) (@INC contains: /etc/perl /usr/local/lib64/perl5/5.30.1/x86_64-linux /usr/local/lib64/perl5/5.30.1 /usr/lib64/perl5/vendor_perl/5.30.1/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.1 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl/5.30.0/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.0 /usr/lib64/perl5/vendor_perl/5.28.0 /usr/lib64/perl5/vendor_perl/5.26.2 /usr/lib64/perl5/vendor_perl/5.26.1 /usr/lib64/perl5/vendor_perl/5.24.0 /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/5.30.1/x86_64-linux /usr/lib64/perl5/5.30.1) at (eval 101) line 47.
BEGIN failed--compilation aborted at (eval 101) line 47.

That one was also simple enough to solve by installing dev-perl/Archive-Zip:

emerge -av dev-perl/Archive-Zip

Fortunately, this ‘missing dependencies’ problem was fixed with version 2.12.0-r2.

Changes in the ‘amavis’ user’s home directory

With the addition of the acct-user/amavis package, a few things changes about the default setup for the ‘amavis’ user. When installing that package, Portage warns about some steps that need to be implemented in order to make amavis work again after migrating its home directory from /var/amavis/ to /var/lib/amavishome. Even after following those steps, I saw the following error message when starting amavis:

Feb 27 00:08:31 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:08:31 [amavis] will bind to /var/amavis/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Amavis (type Net::Server::PreForkSimple) starting! pid(1738)
Feb 27 00:08:31 [amavis] Net::Server: Binding to UNIX socket file "/var/amavis/amavisd.sock"
Feb 27 00:08:31 [amavis] (!)Net::Server: 2020/02/27-00:08:31 Can't connect to UNIX socket at file /var/amavis/amavisd.sock [No such file or directory]\n  at line 66 in file /usr/lib64/perl5/vendor_perl/5.30.1/Net/Server/Proto/UNIX.pm
Feb 27 00:08:31 [amavis] sd_notify (no socket): STOPPING=1\nSTATUS=Server rundown, notifying child processes.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Server closing!
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Child processes have been stopped.

In that error message, I noticed that there were still references to /var/amavis/ instead of the new /var/lib/amavishome/ directory, so I updated it using the $MYHOME variable in /etc/amavisd.conf:

# grep -e '^$MYHOME' /etc/amavisd.conf 
$MYHOME = '/var/lib/amavishome';   # a convenient default for other settings, -H

Thereafter, the start-up logs indicated that it was binding to a UNIX socket in the correct home directory:

Feb 27 00:12:15 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:12:15 [amavis] will bind to /var/lib/amavishome/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:12:15 [amavis] Net::Server: 2020/02/27-00:12:15 Amavis (type Net::Server::PreForkSimple) starting! pid(1952)
Feb 27 00:12:15 [amavis] Net::Server: Binding to UNIX socket file "/var/lib/amavishome/amavisd.sock"
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host 127.0.0.1 with IPv4
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host ::1 with IPv6

ClamAV permissions within the amavis home directory:

Though this error wasn’t directly related to the upgrades (and had likely existed for quite some time beforehand), I only just now noticed it whilst combing through the logs (the first error is from my system’s mail log, and the second error is from the clamd log):

Feb 27 00:15:10 [amavis] (01980-01) run_av (ClamAV-clamd) result: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n
Feb 27 00:15:10 [amavis] (01980-01) (!)run_av (ClamAV-clamd) FAILED - unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n"
Feb 27 00:15:10 [amavis] (01980-01) (!)ClamAV-clamd av-scanner FAILED: CODE(0x5611198fa5d8) unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n" at (eval 85) line 951.
# grep 'lstat' /var/log/clamav/clamd.log
Wed Feb 26 23:26:53 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180007-00529-S1gbY8cd/parts
Wed Feb 26 23:28:32 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180038-00592-FK2_Uj2T/parts
Wed Feb 26 23:31:30 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T165427-32346-V4WeP0YX/parts
Thu Feb 27 00:15:10 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts
Thu Feb 27 00:26:21 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002621-01981-l36mWT4P/parts
Thu Feb 27 00:26:30 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002630-01982-k0qgJdjl/parts
Thu Feb 27 00:29:12 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002912-01983-G57aKCmK/parts
Thu Feb 27 00:31:55 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003155-01984-M9r9r1Gc/parts
Thu Feb 27 00:33:07 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003307-01985-8n_wS6pQ/parts
Thu Feb 27 00:40:20 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004020-01986-gH4PrFAk/parts
Thu Feb 27 00:45:45 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004545-01987-J4YGuNbw/parts

These errors were fixed by 1) adding the ‘clamav’ user to the ‘amavis’ group, 2) setting g+w permissions on the /var/lib/amavishome/tmp/ directory, and 3) restarting clamd and amavisd

# gpasswd -a clamav amavis
Adding user clamav to group amavis

# ls -alh /var/lib/amavishome/ | grep tmp
drwxr-xr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# chmod 775 /var/lib/amavishome/tmp

# ls -alh /var/lib/amavishome/ | grep tmp
drwxrwxr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# /etc/init.d/clamd restart && /etc/init.d/amavisd restart

Now ClamAV is readily able to access files under /var/lib/amavishome/tmp/:

Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Connecting to socket  /var/run/clamav/clamd.sock
Feb 27 01:00:12 [amavis] (03346-01) new socket by IO::Socket::UNIX to /var/run/clamav/clamd.sock, timeout set to 10
Feb 27 01:00:12 [amavis] (03346-01) connected to /var/run/clamav/clamd.sock successfully
Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Sending CONTSCAN /var/lib/amavishome/tmp/amavis-20200227T010012-03346-YjbUpYOk/parts\n to socket /var/run/clamav/clamd.sock

Hopefully, if you run into these errors, you will be able to take this information and apply it to your particular mail stack.

Cheers,

Zach

February 23 2020

Searx and Gentoo wiki search

Alice Ferrazzi (alicef) February 23, 2020, 15:00

Two years ago I started to get interested in selfhosting services. I started to go away from private services and implementing selfhosting, manly because private services was disabling most of the features that I liked and I had no way to contribute or see how they was working.
That is what made look into old.reddit.com/r/selfhosted/ and www.privacytools.io/ That is when I disovered searx, as the github page say searx is a "Privacy-respecting metasearch engine".
As any selfhost service, you can install easily on your server or
also on the local computer.
For the installation instruction go here or use the searx-docker project.
As I use selfhosted services also because I like to contribute back,
after a few look I decided to add a meta-engine in searx.
Specifically a Gentoo wiki search meta-engine. Pull requeset #1368

Gentoo wiki search is usually enabled by default in the
it tab :)

Gentoo wiki search can also be used by the searx shortcut system (same as bang in duckduckgo if you have familiarity)

The Gentoo wiki search shortcut is !ge
for example
search.stinpriza.org/?q=%21ge%20project%3Akernel&categories=none&language=en

will give you this:

for concluding, have fun with searx and Gentoo!

I also highly recommend to have your own searx instance but you can play with public instances.

Two years ago I started to get interested in selfhosting services. I started to go away from private services and implementing selfhosting, manly because private services was disabling most of the features that I liked and I had no way to contribute or see how they was working.
That is what made look into https://old.reddit.com/r/selfhosted/ and https://www.privacytools.io/ That is when I disovered searx, as the github page say searx is a "Privacy-respecting metasearch engine".
As any selfhost service, you can install easily on your server or
also on the local computer.
For the installation instruction go here or use the searx-docker project.
As I use selfhosted services also because I like to contribute back,
after a few look I decided to add a meta-engine in searx.
Specifically a Gentoo wiki search meta-engine. Pull requeset #1368

Gentoo wiki search is usually enabled by default in the
it tab :)

searx_gentoo_default

Gentoo wiki search can also be used by the searx shortcut system (same as bang in duckduckgo if you have familiarity)

The Gentoo wiki search shortcut is !ge
for example
https://search.stinpriza.org/?q=%21ge%20project%3Akernel&categories=none&language=en

will give you this:
searx_shortcut_ge

for concluding, have fun with searx and Gentoo!

I also highly recommend to have your own searx instance but you can play with public instances.

February 21 2020

Gentoo Python Guide

Michał Górny (mgorny) February 21, 2020, 10:35

Gentoo provides one of the best frameworks for providing Python support in packages among operating systems. This includes support for running multiple versions of Python (while most other distributions avoid going beyond simultaneous support for Python 2 and one version of Python 3), alternative implementations of Python, reliable tests, deep QA checks. While we aim to keep things simple, this is not always possible.

At the same time, the available documentation is limited and not always up-to-date. Both the built-in eclass documentation and Python project wiki page provide bits of documentation but they are mostly in reference form and not very suitable for beginners nor people who do not actively follow the developments within the ecosystem. This results in suboptimal ebuilds, improper dependencies, missing tests.

Gentoo Python Guide aims to fill the gap by providing a good, complete, by-topic (rather than reference-style) documentation for the ecosystem in Gentoo and the relevant eclasses. Combined with examples, it should help you write good ebuilds and solve common problems as simply as possible.

Gentoo Python Guide sources are available on GitHub. Suggestions and improvements are welcome.

mgorny (mgorny ) February 21, 2020, 10:35

Gentoo provides one of the best frameworks for providing Python support in packages among operating systems. This includes support for running multiple versions of Python (while most other distributions avoid going beyond simultaneous support for Python 2 and one version of Python 3), alternative implementations of Python, reliable tests, deep QA checks. While we aim to keep things simple, this is not always possible.

At the same time, the available documentation is limited and not always up-to-date. Both the built-in eclass documentation and Python project wiki page provide bits of documentation but they are mostly in reference form and not very suitable for beginners nor people who do not actively follow the developments within the ecosystem. This results in suboptimal ebuilds, improper dependencies, missing tests.

Gentoo Python Guide aims to fill the gap by providing a good, complete, by-topic (rather than reference-style) documentation for the ecosystem in Gentoo and the relevant eclasses. Combined with examples, it should help you write good ebuilds and solve common problems as simply as possible.

Gentoo Python Guide sources are available on GitHub. Suggestions and improvements are welcome.

February 10 2020

No more PYTHON_TARGETS in single-r1

Michał Górny (mgorny) February 10, 2020, 7:39

Since its inception in 2012, python-single-r1 has been haunting users with two sets of USE flags: PYTHON_TARGETS and PYTHON_SINGLE_TARGET. While this initially seemed a necessary part of the grand design, today I know we could have done better. Today this chymera is disappearing for real, and python-single-r1 are going to use PYTHON_SINGLE_TARGET flags only.

I would like to take this opportunity to explain why the eclass has been designed this way in the first place, and what has been done to change that.

Why PYTHON_SINGLE_TARGET?

Why did we need a second variable in the first place? After all, we could probably get away with using PYTHON_TARGETS everywhere, and adding an appropriate REQUIRED_USE constraint.

Back in the day we have established that for users’ convenience we need to default to enabling one version of Python 2 and one version of Python 3. If we enabled only one of them, the users would end up having to enable the other for a lot of packages. On the other had, if we combined both with using PT for single-r1 packages, the users would have to disable the extra implementation for a lot of them. Neither option was good.

The primary purpose of PYTHON_SINGLE_TARGET was to provide a parallel sensible setting for those packages. It was not only to make the default work out of the box but also to let users change it in one step.

Today, with the demise of Python 2 and the effort to remove Python 2 from default PT, it may seem less important to keep the distinction. Nevertheless, a number of developers and at least some users keep multiple versions of Python in PT to test their packages. Having PST is still helpful to them.

Why additional PYTHON_TARGETS then?

PST is only half of the story. What I explained above does not justify having PYTHON_TARGETS on those packages as well, and a REQUIRED_USE constraint to make them superset of enabled PST. Why did we need to have two flag sets then?

The answer is: PYTHON_USEDEP. The initial design goal was that both python-r1 eclasses would use the same approach to declaring USE dependencies between packages. This also meant that this variable must work alike on dependencies that are multi-impl and single-r1 packages. In the end, this meant a gross hack.

Without getting into details, the currently available USE dependency syntax does not permit directly depending on PT flags based on PST-based conditions. This needs to be done using the more verbose expanded syntax:

pst2_7? ( foo[pt2_7] )
pst3_7? ( foo[pt3_7] )

While this was doable back in the day, it was not possible with PYTHON_USEDEP-based approach. Hence, all single-r1 packages gained additional set of flags merely to construct dependencies conveniently.

What is the problem with that?

I suppose some of you see the problem already. Nevertheless, let’s list them explicitly.

Firstly, enabling additional implementations is inconvenient. Whenever you need to do that, you need to add both PST and PT flags.

Secondly, the PT flags are entirely redundant and meaningless for the package in question. Whenever your value of PT changes, all single-r1 packages trigger rebuilds even if their PST value stays the same.

Thirdly, the PT flags overspecify dependencies. If your PT flags specify multiple implementations (which is normally the case), all dependencies will also have to be built for those interpreters even though PST requires only one of them.

The solution

The user-visible part of the solution is that PYTHON_TARGETS are disappearing from single-r1 packages. From now on, only PYTHON_SINGLE_TARGET will be necessary. Furthermore, PT enforcement on dependencies (if necessary) will be limited to the single implementation selected by PST rather than all of PT.

The developer-oriented part is that PYTHON_USEDEP is no longer valid in single-r1 packages. Instead, PYTHON_SINGLE_USEDEP is provided for dependencies on other single-r1 packages, and PYTHON_MULTI_USEDEP placeholder is used for multi-impl packages. The former is available as a global variable, the latter only as a placeholder in python_gen_cond_dep (the name is a bit of misnomer now but I’ve decided not to introduce additional function).

All existing uses have been converted, and the eclasses will now fail if someone tries to use the old logic. The conversion of existing ebuilds is rather simple:

  1. Replace all ${PYTHON_USEDEP}s with ${PYTHON_SINGLE_USEDEP} when the dep is single-r1, or with ${PYTHON_MULTI_USEDEP} otherwise.
  2. Wrap all dependencies containing ${PYTHON_MULTI_USEDEP} in a python_gen_cond_dep. Remember that the variable must be a literal placeholder, i.e. use single quotes.

An example of the new logic follows:

RDEPEND="
  dev-libs/libfoo[python,${PYTHON_SINGLE_USEDEP}]
  $(python_gen_cond_dep '
    dev-python/foo[${PYTHON_MULTI_USEDEP}]
    dev-python/bar[${PYTHON_MULTI_USEDEP}]
  ')
"

If you get the dependency type wrong, repoman/pkgcheck will complain about bad dependency.

Since its inception in 2012, python-single-r1 has been haunting users with two sets of USE flags: PYTHON_TARGETS and PYTHON_SINGLE_TARGET. While this initially seemed a necessary part of the grand design, today I know we could have done better. Today this chymera is disappearing for real, and python-single-r1 are going to use PYTHON_SINGLE_TARGET flags only.

I would like to take this opportunity to explain why the eclass has been designed this way in the first place, and what has been done to change that.

Why PYTHON_SINGLE_TARGET?

Why did we need a second variable in the first place? After all, we could probably get away with using PYTHON_TARGETS everywhere, and adding an appropriate REQUIRED_USE constraint.

Back in the day we have established that for users’ convenience we need to default to enabling one version of Python 2 and one version of Python 3. If we enabled only one of them, the users would end up having to enable the other for a lot of packages. On the other had, if we combined both with using PT for single-r1 packages, the users would have to disable the extra implementation for a lot of them. Neither option was good.

The primary purpose of PYTHON_SINGLE_TARGET was to provide a parallel sensible setting for those packages. It was not only to make the default work out of the box but also to let users change it in one step.

Today, with the demise of Python 2 and the effort to remove Python 2 from default PT, it may seem less important to keep the distinction. Nevertheless, a number of developers and at least some users keep multiple versions of Python in PT to test their packages. Having PST is still helpful to them.

Why additional PYTHON_TARGETS then?

PST is only half of the story. What I explained above does not justify having PYTHON_TARGETS on those packages as well, and a REQUIRED_USE constraint to make them superset of enabled PST. Why did we need to have two flag sets then?

The answer is: PYTHON_USEDEP. The initial design goal was that both python-r1 eclasses would use the same approach to declaring USE dependencies between packages. This also meant that this variable must work alike on dependencies that are multi-impl and single-r1 packages. In the end, this meant a gross hack.

Without getting into details, the currently available USE dependency syntax does not permit directly depending on PT flags based on PST-based conditions. This needs to be done using the more verbose expanded syntax:

pst2_7? ( foo[pt2_7] )
pst3_7? ( foo[pt3_7] )

While this was doable back in the day, it was not possible with PYTHON_USEDEP-based approach. Hence, all single-r1 packages gained additional set of flags merely to construct dependencies conveniently.

What is the problem with that?

I suppose some of you see the problem already. Nevertheless, let’s list them explicitly.

Firstly, enabling additional implementations is inconvenient. Whenever you need to do that, you need to add both PST and PT flags.

Secondly, the PT flags are entirely redundant and meaningless for the package in question. Whenever your value of PT changes, all single-r1 packages trigger rebuilds even if their PST value stays the same.

Thirdly, the PT flags overspecify dependencies. If your PT flags specify multiple implementations (which is normally the case), all dependencies will also have to be built for those interpreters even though PST requires only one of them.

The solution

The user-visible part of the solution is that PYTHON_TARGETS are disappearing from single-r1 packages. From now on, only PYTHON_SINGLE_TARGET will be necessary. Furthermore, PT enforcement on dependencies (if necessary) will be limited to the single implementation selected by PST rather than all of PT.

The developer-oriented part is that PYTHON_USEDEP is no longer valid in single-r1 packages. Instead, PYTHON_SINGLE_USEDEP is provided for dependencies on other single-r1 packages, and PYTHON_MULTI_USEDEP placeholder is used for multi-impl packages. The former is available as a global variable, the latter only as a placeholder in python_gen_cond_dep (the name is a bit of misnomer now but I’ve decided not to introduce additional function).

All existing uses have been converted, and the eclasses will now fail if someone tries to use the old logic. The conversion of existing ebuilds is rather simple:

  1. Replace all ${PYTHON_USEDEP}s with ${PYTHON_SINGLE_USEDEP} when the dep is single-r1, or with ${PYTHON_MULTI_USEDEP} otherwise.
  2. Wrap all dependencies containing ${PYTHON_MULTI_USEDEP} in a python_gen_cond_dep. Remember that the variable must be a literal placeholder, i.e. use single quotes.

An example of the new logic follows:

RDEPEND="
  dev-libs/libfoo[python,${PYTHON_SINGLE_USEDEP}]
  $(python_gen_cond_dep '
    dev-python/foo[${PYTHON_MULTI_USEDEP}]
    dev-python/bar[${PYTHON_MULTI_USEDEP}]
  ')
"

If you get the dependency type wrong, repoman/pkgcheck will complain about bad dependency.

January 03 2020

FOSDEM 2020

Gentoo News (GentooNews) January 03, 2020, 0:00

It’s FOSDEM time again! Join us at Université libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2020 will be held on February 1st and 2nd.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building K where we will also celebrate 20 years compiling! Visit this year’s wiki page to see who’s coming.

FOSDEM logo

It’s FOSDEM time again! Join us at Université libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2020 will be held on February 1st and 2nd.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building K where we will also celebrate 20 years compiling! Visit this year’s wiki page to see who’s coming.

December 28 2019

Scylla Summit 2019

Alexys Jacob (ultrabug) December 28, 2019, 19:04

I’ve had the pleasure to attend again and present at the Scylla Summit in San Francisco and the honor to be awarded the Most innovative use case of Scylla.

It was a great event, full of friendly people and passionate conversations. Peter did a great full write-up of it already so I wanted to share some of my notes instead…

This a curated set of topics that I happened to question or discuss in depth so this post is not meant to be taken as a full coverage of the conference.

Scylla Manager version 2

The upcoming version of scylla-manager is dropping its dependency on SSH setup which will be replaced by an agent, most likely shipped as a separate package.

On the features side, I was a bit puzzled by the fact that ScyllaDB is advertising that its manager will provide a repair scheduling window so that you can control when it’s running or not.

Why did it struck me you ask?

Because MongoDB does the same thing within its balancer process and I always thought of this as a patch to a feature that the database should be able to cope with by itself.

And that database-do-it-better-than-you motto is exactly one of the promises of Scylla, the boring database, so smart at handling workload impacts on performance that you shouldn’t have to start playing tricks to mitigate them… I don’t want this time window feature on scylla-manager to be a trojan horse on the demise of that promise!

Kubernetes

They almost got late on this but are working hard to play well with the new toy of every tech around the world. Helm charts are also being worked on!

The community developed scylla operator by Yannis is now being worked on and backed by ScyllaDB. It can deploy, scale up and down a cluster.

Few things to note:

  • it’s using a configmap to store the scylla config
  • no TLS support yet
  • no RBAC support yet
  • kubernetes networking is lighter on the network performance hit that was seen on Docker
  • use placement strategies to dedicate kubernetes nodes to scylla!
Change Data Capture

Oh boy this one was awaited… but it’s now coming soon!

I inquired about it’s performance impact since every operation will be written to a table. Clearly my questioning was a bit alpha since CDC is still being worked on.

I had the chance to discuss ideas with Kamil, Tzach and Dor: one of the thing that one of my colleague Julien asked for was the ability for the CDC to generate an event when a tombstone is written so we could actually know when a specific data expired!

I want to stress a few other things too:

  • default TTL on CDC table is 24H
  • expect I/O impact (logical)
  • TTL tombstones can have a hidden disk space cost and nobody was able to tell me if the CDC table was going to be configured with a lower gc_grace_period than the default 10 days so that’s something we need to keep in mind and check for
  • there was no plan to add user information that would allow us to know who actually did the operation, so that’s something I asked for because it could be used as a cheap and open source way to get auditing!
LightWeight Transactions

Another so long awaited feature is also coming from the amazing work and knowledge of Konstantin. We had a great conversation about the differences between the currently worked on Paxos based LWT implementation and the maybe later Raft one.

So yes, the first LWT implementation will be using Paxos as a consensus algorithm. This will make the LWT feature very consistent while having it slower that what could be achieved using Raft. That’s why ScyllaDB have plans on another implementation that could be faster with less data consistency guarantees.

User Defined Functions / Aggregations

This one is bringing the Lua language inside Scylla!

To be precise, it will be a Lua JIT as its footprint is low and Lua can be cooperative enough but the ScyllaDB people made sure to monitor its violations (when it should yield but does not) and act strongly upon them.

I got into implementation details with Avi, this is what I noted:

  • lua function return type is not checked at creation but at execution, so expect runtime errors if your lua code is bad
  • since lua is lightweight, there’s no need to assign a core to lua execution
  • I found UDA examples, like top-k rows, to be very similar to the Map/Reduce logic
  • UDF will allow simpler token range full table scans thanks to syntax sugar
  • there will be memory limits applied to result sets from UDA, and they will be tunable
Text search

Dejan is the text search guy at ScyllaDB and the one who kindly implemented the LIKE feature we asked for and that will be released in the upcoming 3.2 version.

We discussed ideas and projected use cases to make sure that what’s going to be worked on will be used!

Redis API

I’ve always been frustrated about Redis because while I love the technology I never trusted its clustering and scaling capabilities.

What if you could scale your Redis like Scylla without giving up on performance? That’s what the implementation of the Redis API backed by Scylla will get us!

I’m desperately looking forward to see this happen!

ultrabug (ultrabug ) December 28, 2019, 19:04

I’ve had the pleasure to attend again and present at the Scylla Summit in San Francisco and the honor to be awarded the Most innovative use case of Scylla.

It was a great event, full of friendly people and passionate conversations. Peter did a great full write-up of it already so I wanted to share some of my notes instead…

This a curated set of topics that I happened to question or discuss in depth so this post is not meant to be taken as a full coverage of the conference.

Scylla Manager version 2

The upcoming version of scylla-manager is dropping its dependency on SSH setup which will be replaced by an agent, most likely shipped as a separate package.

On the features side, I was a bit puzzled by the fact that ScyllaDB is advertising that its manager will provide a repair scheduling window so that you can control when it’s running or not.

Why did it struck me you ask?

Because MongoDB does the same thing within its balancer process and I always thought of this as a patch to a feature that the database should be able to cope with by itself.

And that database-do-it-better-than-you motto is exactly one of the promises of Scylla, the boring database, so smart at handling workload impacts on performance that you shouldn’t have to start playing tricks to mitigate them… I don’t want this time window feature on scylla-manager to be a trojan horse on the demise of that promise!

Kubernetes

They almost got late on this but are working hard to play well with the new toy of every tech around the world. Helm charts are also being worked on!

The community developed scylla operator by Yannis is now being worked on and backed by ScyllaDB. It can deploy, scale up and down a cluster.

Few things to note:

  • it’s using a configmap to store the scylla config
  • no TLS support yet
  • no RBAC support yet
  • kubernetes networking is lighter on the network performance hit that was seen on Docker
  • use placement strategies to dedicate kubernetes nodes to scylla!

Change Data Capture

Oh boy this one was awaited… but it’s now coming soon!

I inquired about it’s performance impact since every operation will be written to a table. Clearly my questioning was a bit alpha since CDC is still being worked on.

I had the chance to discuss ideas with Kamil, Tzach and Dor: one of the thing that one of my colleague Julien asked for was the ability for the CDC to generate an event when a tombstone is written so we could actually know when a specific data expired!

I want to stress a few other things too:

  • default TTL on CDC table is 24H
  • expect I/O impact (logical)
  • TTL tombstones can have a hidden disk space cost and nobody was able to tell me if the CDC table was going to be configured with a lower gc_grace_period than the default 10 days so that’s something we need to keep in mind and check for
  • there was no plan to add user information that would allow us to know who actually did the operation, so that’s something I asked for because it could be used as a cheap and open source way to get auditing!

LightWeight Transactions

Another so long awaited feature is also coming from the amazing work and knowledge of Konstantin. We had a great conversation about the differences between the currently worked on Paxos based LWT implementation and the maybe later Raft one.

So yes, the first LWT implementation will be using Paxos as a consensus algorithm. This will make the LWT feature very consistent while having it slower that what could be achieved using Raft. That’s why ScyllaDB have plans on another implementation that could be faster with less data consistency guarantees.

User Defined Functions / Aggregations

This one is bringing the Lua language inside Scylla!

To be precise, it will be a Lua JIT as its footprint is low and Lua can be cooperative enough but the ScyllaDB people made sure to monitor its violations (when it should yield but does not) and act strongly upon them.

I got into implementation details with Avi, this is what I noted:

  • lua function return type is not checked at creation but at execution, so expect runtime errors if your lua code is bad
  • since lua is lightweight, there’s no need to assign a core to lua execution
  • I found UDA examples, like top-k rows, to be very similar to the Map/Reduce logic
  • UDF will allow simpler token range full table scans thanks to syntax sugar
  • there will be memory limits applied to result sets from UDA, and they will be tunable

Text search

Dejan is the text search guy at ScyllaDB and the one who kindly implemented the LIKE feature we asked for and that will be released in the upcoming 3.2 version.

We discussed ideas and projected use cases to make sure that what’s going to be worked on will be used!

Redis API

I’ve always been frustrated about Redis because while I love the technology I never trusted its clustering and scaling capabilities.

What if you could scale your Redis like Scylla without giving up on performance? That’s what the implementation of the Redis API backed by Scylla will get us!

I’m desperately looking forward to see this happen!

December 24 2019

Handling PEP 517 (pyproject.toml) packages in Gentoo

Michał Górny (mgorny) December 24, 2019, 22:59

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.

Instead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we’ve explored the possibility of converting the pyproject.toml files into setup.py scripts. Since the new formats are declarative, this should not be that hard.

We’ve found poetry-setup project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.

Finally, I’ve managed to dedicate some time into building an in-house solution instead. pyproject2setuppy is a small-ish (<100 SLOC) pyproject.toml-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it’s quite limited, it’s good enough to build and install the packages that we needed to deal with so far.

The design is quite simple — it reads pyproject.toml and calls setuptools’ setup() function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py script in other packages. In fact, this is how its own setup.py works — it carries flit-compatible pyproject.toml and uses itself to install itself via setuptools.

dev-python/pyproject2setuppy is already packaged in Gentoo. I’ve sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml packages should be as simple as adding the following declaration into ebuilds:

DISTUTILS_USE_SETUPTOOLS=pyproject.toml

This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world. It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.

Instead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we’ve explored the possibility of converting the pyproject.toml files into setup.py scripts. Since the new formats are declarative, this should not be that hard.

We’ve found poetry-setup project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.

Finally, I’ve managed to dedicate some time into building an in-house solution instead. pyproject2setuppy is a small-ish (<100 SLOC) pyproject.toml-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it’s quite limited, it’s good enough to build and install the packages that we needed to deal with so far.

The design is quite simple — it reads pyproject.toml and calls setuptools’ setup() function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py script in other packages. In fact, this is how its own setup.py works — it carries flit-compatible pyproject.toml and uses itself to install itself via setuptools.

dev-python/pyproject2setuppy is already packaged in Gentoo. I’ve sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml packages should be as simple as adding the following declaration into ebuilds:

DISTUTILS_USE_SETUPTOOLS=pyproject.toml

This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world. It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.

December 19 2019

A distribution kernel for Gentoo

Michał Górny (mgorny) December 19, 2019, 12:32

The traditional Gentoo way of getting a kernel is to install the sources, and then configure and build one yourself. For those who didn’t want to go through the tedious process of configuring it manually, an alternative route of using genkernel was provided. However, neither of those variants was able to really provide the equivalent of kernels provided by binary distributions.

I have manually configured the kernels for my private systems long time ago. Today, I wouldn’t really have bothered. In fact, I realized that for some time I’m really hesitant to even upgrade them because of the effort needed to update configuration. The worst part is, whenever a new kernel does not boot, I have to ask myself: is it a real bug, or is it my fault for configuring it wrong?

I’m not alone in this. Recently Михаил Коляда (zlogene) has talked to me about providing binary kernels for Gentoo. While I have not strictly implemented what he had in mind, he inspired me to start working on a distribution kernel. The goal was to create a kernel package that users can install to get a working kernel with minimal effort, and that would be upgraded automatically as part of regular @world upgrades.

Pros and cons of your own kernel

If I am to justify switching from the old tradition of custom kernels to a universal kernel package, I should start by discussing the reasons why you may want to configure a custom kernel in the first place.

In my opinion, the most important feature of a custom kernel is that you can fine-tune it to your hardware. You just have to build the drivers you need (or may need), and the features you care about. The modules for my last custom kernel have occupied 44 MiB. The modules for the distribution kernel occupy 294 MiB. Such a difference in size also comes with a proportional increase of build time. This can be an important argument for people with low-end hardware. On the other hand, the distribution kernel permits building reusable binary packages that can save more computing power.

The traditional Gentoo argument is performance. However, these days I would be very careful arguing about that. I suppose you are able to reap benefits if you know how to configure your kernel towards a specific workload. But then — a misconfiguration can have the exact opposite effect. We must not forget that binary distributions are important players in the field — and the kernel must also be able to achieve good performance when not using a dedicated configuration.

At some point I have worked on achieving a very fast startup. For this reason I’ve switched to using LILO as the bootloader, and a kernel suitable for booting my system without an initramfs. A universal kernel naturally needs an initramfs, and is slower to boot.

The main counterargument is the effort. As mentioned earlier, I’ve personally grown tired of having to manually deal with my kernel. Do the potential gains mentioned outweigh the loss of human time on configuring and maintaining a custom kernel?

Creating a truly universal kernel

A distribution kernel makes sense only if it works on a wide range of systems. Furthermore, I didn’t forget the original idea of binary kernel packages. I didn’t want to write an ebuild that can install a working kernel anywhere. I wanted to create an ebuild that can be used to build a binary package that’s going to work on a wide range of setups — including not only different hardware but also bootloaders and /boot layout. A package that would work fine both for my ‘traditional’ LILO setup and UEFI systemd-boot setup.

The first part of a distribution kernel is the right configuration. I wanted to use a well-tested configuration known to build kernels used on many systems, while at the same time minimizing the maintenance effort on our end. Reusing the configuration from a binary distro was the obvious solution. I went for using the config from Arch Linux’s kernel package with minimal changes (e.g. changing the default hostname to Gentoo).

The second part is an initramfs. Since we need to support a wide variety of setups, we can’t get away without it. To follow the configuration used, Dracut was the natural choice.

The third and hardest part is installing it all. Since I’ve already set a goal of reusing the same binary package on different filesystem layouts, the actual installation needed to be moved to postinst phase. Our distribution kernel package installs the kernel into an interim location which is entirely setup-independent, rendering the binary packages setup-agnostic as well. The initramfs is created and installed into the final location along with the kernel in pkg_postinst.

Support for different install layouts is provided by reusing the installkernel tool, originally installed by debianutils. As part of the effort, it was extended with initramfs support and moved into a separate sys-kernel/installkernel-gentoo package. Furthermore, an alternative sys-kernel/installkernel-systemd-boot package was created to provide an out-of-the-box support for systemd-boot layout. If neither of those two work for you, you can easily create your own /usr/local/bin/installkernel that follows your own layout.

Summary

The experimental versions of the distribution kernel are packaged as sys-kernel/vanilla-kernel (in distinction from sys-kernel/vanilla-sources that install the sources). Besides providing the default zero-effort setup, the package supports using your own configuration via savedconfig (but no easy way to update it at the moment). It also provides a forced flag that can be used by expert users to disable the initramfs.

The primary goal at the moment is to test the package and find bugs that could prevent our users from using it. In the future, we’re planning to extend it to other architectures, kernel variants (Gentoo patch set in particular) and LTS versions. We’re also considering providing prebuilt binary packages — however, this will probably be a part of a bigger effort into providing an official Gentoo binhost.

The traditional Gentoo way of getting a kernel is to install the sources, and then configure and build one yourself. For those who didn’t want to go through the tedious process of configuring it manually, an alternative route of using genkernel was provided. However, neither of those variants was able to really provide the equivalent of kernels provided by binary distributions.

I have manually configured the kernels for my private systems long time ago. Today, I wouldn’t really have bothered. In fact, I realized that for some time I’m really hesitant to even upgrade them because of the effort needed to update configuration. The worst part is, whenever a new kernel does not boot, I have to ask myself: is it a real bug, or is it my fault for configuring it wrong?

I’m not alone in this. Recently Михаил Коляда (zlogene) has talked to me about providing binary kernels for Gentoo. While I have not strictly implemented what he had in mind, he inspired me to start working on a distribution kernel. The goal was to create a kernel package that users can install to get a working kernel with minimal effort, and that would be upgraded automatically as part of regular @world upgrades.

Pros and cons of your own kernel

If I am to justify switching from the old tradition of custom kernels to a universal kernel package, I should start by discussing the reasons why you may want to configure a custom kernel in the first place.

In my opinion, the most important feature of a custom kernel is that you can fine-tune it to your hardware. You just have to build the drivers you need (or may need), and the features you care about. The modules for my last custom kernel have occupied 44 MiB. The modules for the distribution kernel occupy 294 MiB. Such a difference in size also comes with a proportional increase of build time. This can be an important argument for people with low-end hardware. On the other hand, the distribution kernel permits building reusable binary packages that can save more computing power.

The traditional Gentoo argument is performance. However, these days I would be very careful arguing about that. I suppose you are able to reap benefits if you know how to configure your kernel towards a specific workload. But then — a misconfiguration can have the exact opposite effect. We must not forget that binary distributions are important players in the field — and the kernel must also be able to achieve good performance when not using a dedicated configuration.

At some point I have worked on achieving a very fast startup. For this reason I’ve switched to using LILO as the bootloader, and a kernel suitable for booting my system without an initramfs. A universal kernel naturally needs an initramfs, and is slower to boot.

The main counterargument is the effort. As mentioned earlier, I’ve personally grown tired of having to manually deal with my kernel. Do the potential gains mentioned outweigh the loss of human time on configuring and maintaining a custom kernel?

Creating a truly universal kernel

A distribution kernel makes sense only if it works on a wide range of systems. Furthermore, I didn’t forget the original idea of binary kernel packages. I didn’t want to write an ebuild that can install a working kernel anywhere. I wanted to create an ebuild that can be used to build a binary package that’s going to work on a wide range of setups — including not only different hardware but also bootloaders and /boot layout. A package that would work fine both for my ‘traditional’ LILO setup and UEFI systemd-boot setup.

The first part of a distribution kernel is the right configuration. I wanted to use a well-tested configuration known to build kernels used on many systems, while at the same time minimizing the maintenance effort on our end. Reusing the configuration from a binary distro was the obvious solution. I went for using the config from Arch Linux’s kernel package with minimal changes (e.g. changing the default hostname to Gentoo).

The second part is an initramfs. Since we need to support a wide variety of setups, we can’t get away without it. To follow the configuration used, Dracut was the natural choice.

The third and hardest part is installing it all. Since I’ve already set a goal of reusing the same binary package on different filesystem layouts, the actual installation needed to be moved to postinst phase. Our distribution kernel package installs the kernel into an interim location which is entirely setup-independent, rendering the binary packages setup-agnostic as well. The initramfs is created and installed into the final location along with the kernel in pkg_postinst.

Support for different install layouts is provided by reusing the installkernel tool, originally installed by debianutils. As part of the effort, it was extended with initramfs support and moved into a separate sys-kernel/installkernel-gentoo package. Furthermore, an alternative sys-kernel/installkernel-systemd-boot package was created to provide an out-of-the-box support for systemd-boot layout. If neither of those two work for you, you can easily create your own /usr/local/bin/installkernel that follows your own layout.

Summary

The experimental versions of the distribution kernel are packaged as sys-kernel/vanilla-kernel (in distinction from sys-kernel/vanilla-sources that install the sources). Besides providing the default zero-effort setup, the package supports using your own configuration via savedconfig (but no easy way to update it at the moment). It also provides a forced flag that can be used by expert users to disable the initramfs.

The primary goal at the moment is to test the package and find bugs that could prevent our users from using it. In the future, we’re planning to extend it to other architectures, kernel variants (Gentoo patch set in particular) and LTS versions. We’re also considering providing prebuilt binary packages — however, this will probably be a part of a bigger effort into providing an official Gentoo binhost.

December 12 2019

A better ebuild workflow with pure git and pkgcheck

Michał Górny (mgorny) December 12, 2019, 20:04

Many developers today continue using repoman commit as their primary way of committing to Gentoo. While this tool was quite helpful, if not indispensable in times of CVS, today it’s a burden. The workflow using a single serial tool to check your packages and commit to them is not very efficient. Not only it wastes your time and slows you down — it discourages you from splitting your changes into more atomic commits.

Upon hearing the pkgcheck advocacy, many developers ask whether it can commit for you. It won’t do that, that’s not its purpose. Not only it’s waste of time to implement that — it would actually make it a worse tool. With its parallel engine pkgcheck really shines when dealing with multiple packages — forcing it to work on one package is a waste of its potential.

Rather than trying to proliferate your bad old habits, you should learn how to use git and pkgcheck efficiently. This post aims to give you a few advices.

pkgcheck after committing

Repoman was built under the assumption that checks should be done prior to committing. That is understandable when you’re working on a ‘live’ repository as the ones used by CVS or Subversion. However, in case of VCS-es involving staging commits such as Git there is no real difference between checking prior to or post commit. The most efficient pkgcheck workflow is to check once all changes are committed and you are ready to push.

The most recent version of pkgcheck has a command just for that:

$ pkgcheck scan --commits

Yes, it’s that simple. It checks what you’ve committed compared to origin (note: you’ll need to have a correct origin remote), and runs scan on all those packages. Now, if you’re committing changes to multiple packages (which should be pretty common), the scan is run in parallel to utilize your CPU power better.

You might say: but repoman ensures that my commit message is neat these days! Guess what. The --commits option does exactly that — it raises warnings if your commit message is bad. Admittedly, it only checks summary line at the moment but that’s something that can (and will) be improved easily.

And I’ve forgotten the most cool thing of all: pkgcheck also reports if you accidentally remove the newest ebuild with stable keywords on given arch!

One more tip. You can use the following option to include full live verification of URLs:

$ pkgcheck scan --net --commits

Again, this is a feature missing entirely from repoman.

pkgcommit to ease committing to ebuilds

While the majority of repoman’s VCS support is superficial or better implemented elsewhere, there’s one killer feature worth keeping: automatically prepending the package name to the summary line. Since that is a really trivial thing, I’ve reimplemented it in a few lines of bash as pkgcommit.

When run in a package directory, it runs an editor with pre-filled commit message template to let you type it in, then passes it along with its own arguments to git. Usually, I use it as (I like to be explicit about signoffs and signing, you can make .git/config take care of it):

$ pkgcommit -sS .

Its extra feature is that it processes -m option and lets you skip the editor for simple messages:

$ pkgcommit -sS . -m 'Bump to 1.2.3'

Note that it does not go out of its way to figure out what to commit. You need to either stage changes yourself via git add, or pass appropriate paths to the command. What’s important is that it does not limit you to committing to one directory — you can e.g. include some profile changes easily.

You’ll also need pkg script from the same repository. Or you just install the whole bundle of app-portage/mgorny-dev-scripts.

Amending commits via fixups

Most of you know probably know that you can update commits via git commit --amend. However, that’s useful only for editing the most recent commit. You can also use interactive rebase to choose specific commits for editing, and then amend them. Yet, usually there’s a much more convenient way of doing that.

In order to commit a fixup to a particular past commit, use:

$ git commit --fixup OLD_COMMIT_ID

This will create a specially titled commit that will be automatically picked up and ordered by the interactive rebase:

$ git rebase -i -S origin

Again, I have a tool of greater convenience. Frequently, I just want to update the latest commit to a particular package (directory). git-fixup does exactly that — it finds the identifier of the latest commit to a particular file/directory (or the current directory when no parameter is given) and commits a fixup to that:

$ git-fixup .

Note that if you try to push fixups into the repository, nothing will stop you. This is one of the reasons that I don’t enable signoffs and signing on all commits by default. This way, if I forget to rebase my fixups, the git hook will reject them as lacking signoff and/or signature.

Again, it is part of app-portage/mgorny-dev-scripts.

Interactive rebase to the rescue

When trivial tools are no longer sufficient, interactive rebase is probably one of the best tools for editing your commits. Start by initiating it for all commits since the last push:

$ git rebase -i -S origin

It will bring your editor with a list of all commits. Using this list, you can do a lot: reorder commits, drop them, reword their commit messages, use squash or fixup to merge them into other commits, and finally: edit them (open for amending).

The interactive rebase is probably the most powerful porcelain git command. I’ve personally found the immediate tips given by git good enough but I realize that many people find it hard nevertheless. Since it’s not my goal here to provide detailed instructions on using git, I’m going to suggest looking online for tutorials and guides. The Rewriting History section of the Git Book also has a few examples.

Before pushing: git log

git log seems to be one of the most underappreciated pre-push tools. However, it can be of great service to you. When run prior to pushing, it can help you verify that what you’re pushing is actually what you’ve meant to push.

$ git log --stat

will list all staged commits along with a pretty summary of affected files. This can help you notice that you’ve forgotten to git add a patch, or that you’ve accidentally committed some extraneous change, or that you’ve just mixed changes from two commits.

Of course, you can go even further and take a look at the changes in patch form:

$ git log -p

While I realize this is nothing new or surprising to you, sometimes it’s worthwhile to reiterate the basics in a different context to make you realize something obvious.

Many developers today continue using repoman commit as their primary way of committing to Gentoo. While this tool was quite helpful, if not indispensable in times of CVS, today it’s a burden. The workflow using a single serial tool to check your packages and commit to them is not very efficient. Not only it wastes your time and slows you down — it discourages you from splitting your changes into more atomic commits.

Upon hearing the pkgcheck advocacy, many developers ask whether it can commit for you. It won’t do that, that’s not its purpose. Not only it’s waste of time to implement that — it would actually make it a worse tool. With its parallel engine pkgcheck really shines when dealing with multiple packages — forcing it to work on one package is a waste of its potential.

Rather than trying to proliferate your bad old habits, you should learn how to use git and pkgcheck efficiently. This post aims to give you a few advices.

pkgcheck after committing

Repoman was built under the assumption that checks should be done prior to committing. That is understandable when you’re working on a ‘live’ repository as the ones used by CVS or Subversion. However, in case of VCS-es involving staging commits such as Git there is no real difference between checking prior to or post commit. The most efficient pkgcheck workflow is to check once all changes are committed and you are ready to push.

The most recent version of pkgcheck has a command just for that:

$ pkgcheck scan --commits

Yes, it’s that simple. It checks what you’ve committed compared to origin (note: you’ll need to have a correct origin remote), and runs scan on all those packages. Now, if you’re committing changes to multiple packages (which should be pretty common), the scan is run in parallel to utilize your CPU power better.

You might say: but repoman ensures that my commit message is neat these days! Guess what. The --commits option does exactly that — it raises warnings if your commit message is bad. Admittedly, it only checks summary line at the moment but that’s something that can (and will) be improved easily.

And I’ve forgotten the most cool thing of all: pkgcheck also reports if you accidentally remove the newest ebuild with stable keywords on given arch!

One more tip. You can use the following option to include full live verification of URLs:

$ pkgcheck scan --net --commits

Again, this is a feature missing entirely from repoman.

pkgcommit to ease committing to ebuilds

While the majority of repoman’s VCS support is superficial or better implemented elsewhere, there’s one killer feature worth keeping: automatically prepending the package name to the summary line. Since that is a really trivial thing, I’ve reimplemented it in a few lines of bash as pkgcommit.

When run in a package directory, it runs an editor with pre-filled commit message template to let you type it in, then passes it along with its own arguments to git. Usually, I use it as (I like to be explicit about signoffs and signing, you can make .git/config take care of it):

$ pkgcommit -sS .

Its extra feature is that it processes -m option and lets you skip the editor for simple messages:

$ pkgcommit -sS . -m 'Bump to 1.2.3'

Note that it does not go out of its way to figure out what to commit. You need to either stage changes yourself via git add, or pass appropriate paths to the command. What’s important is that it does not limit you to committing to one directory — you can e.g. include some profile changes easily.

You’ll also need pkg script from the same repository. Or you just install the whole bundle of app-portage/mgorny-dev-scripts.

Amending commits via fixups

Most of you know probably know that you can update commits via git commit --amend. However, that’s useful only for editing the most recent commit. You can also use interactive rebase to choose specific commits for editing, and then amend them. Yet, usually there’s a much more convenient way of doing that.

In order to commit a fixup to a particular past commit, use:

$ git commit --fixup OLD_COMMIT_ID

This will create a specially titled commit that will be automatically picked up and ordered by the interactive rebase:

$ git rebase -i -S origin

Again, I have a tool of greater convenience. Frequently, I just want to update the latest commit to a particular package (directory). git-fixup does exactly that — it finds the identifier of the latest commit to a particular file/directory (or the current directory when no parameter is given) and commits a fixup to that:

$ git-fixup .

Note that if you try to push fixups into the repository, nothing will stop you. This is one of the reasons that I don’t enable signoffs and signing on all commits by default. This way, if I forget to rebase my fixups, the git hook will reject them as lacking signoff and/or signature.

Again, it is part of app-portage/mgorny-dev-scripts.

Interactive rebase to the rescue

When trivial tools are no longer sufficient, interactive rebase is probably one of the best tools for editing your commits. Start by initiating it for all commits since the last push:

$ git rebase -i -S origin

It will bring your editor with a list of all commits. Using this list, you can do a lot: reorder commits, drop them, reword their commit messages, use squash or fixup to merge them into other commits, and finally: edit them (open for amending).

The interactive rebase is probably the most powerful porcelain git command. I’ve personally found the immediate tips given by git good enough but I realize that many people find it hard nevertheless. Since it’s not my goal here to provide detailed instructions on using git, I’m going to suggest looking online for tutorials and guides. The Rewriting History section of the Git Book also has a few examples.

Before pushing: git log

git log seems to be one of the most underappreciated pre-push tools. However, it can be of great service to you. When run prior to pushing, it can help you verify that what you’re pushing is actually what you’ve meant to push.

$ git log --stat

will list all staged commits along with a pretty summary of affected files. This can help you notice that you’ve forgotten to git add a patch, or that you’ve accidentally committed some extraneous change, or that you’ve just mixed changes from two commits.

Of course, you can go even further and take a look at the changes in patch form:

$ git log -p

While I realize this is nothing new or surprising to you, sometimes it’s worthwhile to reiterate the basics in a different context to make you realize something obvious.

Gentoo eclass design pitfalls

Michał Górny (mgorny) November 06, 2019, 7:57

I have written my share of eclasses, and I have made my share of mistakes. Designing good eclasses is a non-trivial problem, and there are many pitfalls you should be watching for. In this post, I would like to highlight three of them.

Not all metadata variables are combined

PMS provides a convenient feature for eclass writers: cumulative handling of metadata variables. Quoting the relevant passage:

The IUSE, REQUIRED_USE, DEPEND, BDEPEND, RDEPEND and PDEPEND variables are handled specially when set by an eclass. They must be accumulated across eclasses, appending the value set by each eclass to the resulting value after the previous one is loaded. Then the eclass-defined value is appended to that defined by the ebuild. […]

Package Manager Specification (30th April 2018), 10.2 Eclass-defined Metadata Keys

That’s really handy! However, the important thing that’s not obvious from this description is that not all metadata variables work this way. The following multi-value variables don’t: HOMEPAGE, SRC_URI, LICENSE, KEYWORDS, PROPERTIES and RESTRICT. Surely, some of them are not supposed to be set in eclasses but e.g. the last two are confusing.

This means that technically you need to append when defining them, e.g.:

# my.eclass
RESTRICT+=" !test? ( test )"

However, that’s not the biggest problem. The real issue is that those variables are normally set in ebuilds after inherit, so you actually need to make sure that all ebuilds append to them. For example, the ebuild needs to do:

# my-1.ebuild
inherit my
RESTRICT+=" bindist"

Therefore, this design is prone to mistakes at ebuild level. I’m going to discuss an alternative solution below.

Declarative vs functional

It is common to use declarative style in eclasses — create a bunch of variables that ebuilds can use to control the eclass behavior. However, this style has two significant disadvantages.

Firstly, it is prone to typos. If someone recalls the variable name wrong, and its effects are not explicitly visible, it is very easy to commit an ebuild with a silly bug. If the effects are visible, it can still give you some quality debugging headache.

Secondly, in order to affect global scope, the variables need to be set before inherit. This is not trivially enforced, and it is easy to miss that the variable doesn’t work (or partially misbehaves) when set too late.

The alternative is to use functional style, especially for affecting global scope variables. Instead of immediately editing variables in global scope and expecting ebuilds to control the behavior via variables, give them a function to do it:

# my.eclass
my_enable_pytest() {
  IUSE+=" test"
  RESTRICT+=" !test? ( test )"
  BDEPEND+=" test? ( dev-python/pytest[${PYTHON_USEDEP}] )"
  python_test() {
    pytest -vv || die
  }
}

Note that this function is evaluated in ebuild context, so all variables need appending. Its main advantage is that it works independently of where in ebuild it’s called (but if you call it early, remember to append!), and in case of typo you get an explicit error. Example use in ebuild:

# my-1.ebuild
inherit my
IUSE="randomstuff"
RDEPEND="randomstuff? ( dev-libs/random )"
my_enable_pytest
Think what phases to export

Exporting phase functions is often a matter of convenience. However, doing it poorly can cause ebuild writers more pain than if they weren’t exported in the first place. An example of this is vala.eclass as of today. It wrongly exports dysfunctional src_prepare(), and all ebuilds have to redefine it anyway.

It is often a good idea to consider how your eclass is going to be used. If there are both use cases for having the phases exported and for providing utility functions without any phases, it is probably a good idea to split the eclass in two: into -utils eclass that just provides the functions, and main eclass that combines them with phase functions. A good examples today are xdg and xdg-utils eclasses.

When you do need to export phases, it is wortwhile to consider how different eclasses are going to be combined. Generally, a few eclass types could be listed:

  • Unpacking (fetching) eclasses; e.g. git-r3 with src_unpack(),
  • Build system eclasses; e.g. cmake-utils, src_prepare() through src_install(),
  • Post-install eclasses; e.g. xdg, pkg_*inst(), pkg_*rm(),
  • Build environment setup eclasses; e.g. python-single-r1, pkg_setup().

Generally, it’s best to fit your eclass into as few of those as possible. If you do that, there’s a good chance that the ebuild author would be able to combine multiple eclasses easily:

# my-1.ebuild
PYTHON_COMPAT=( python3_7 )
inherit cmake-utils git-r3 python-single-r1

Note that since each of those eclasses uses a different phase function set to do its work, they combine just fine! The inherit order is also irrelevant. If we e.g. need to add llvm to the list, we just have to redefine pkg_setup().

I have written my share of eclasses, and I have made my share of mistakes. Designing good eclasses is a non-trivial problem, and there are many pitfalls you should be watching for. In this post, I would like to highlight three of them.

Not all metadata variables are combined

PMS provides a convenient feature for eclass writers: cumulative handling of metadata variables. Quoting the relevant passage:

The IUSE, REQUIRED_USE, DEPEND, BDEPEND, RDEPEND and PDEPEND variables are handled specially when set by an eclass. They must be accumulated across eclasses, appending the value set by each eclass to the resulting value after the previous one is loaded. Then the eclass-defined value is appended to that defined by the ebuild. […]

Package Manager Specification (30th April 2018), 10.2 Eclass-defined Metadata Keys

That’s really handy! However, the important thing that’s not obvious from this description is that not all metadata variables work this way. The following multi-value variables don’t: HOMEPAGE, SRC_URI, LICENSE, KEYWORDS, PROPERTIES and RESTRICT. Surely, some of them are not supposed to be set in eclasses but e.g. the last two are confusing.

This means that technically you need to append when defining them, e.g.:

# my.eclass
RESTRICT+=" !test? ( test )"

However, that’s not the biggest problem. The real issue is that those variables are normally set in ebuilds after inherit, so you actually need to make sure that all ebuilds append to them. For example, the ebuild needs to do:

# my-1.ebuild
inherit my
RESTRICT+=" bindist"

Therefore, this design is prone to mistakes at ebuild level. I’m going to discuss an alternative solution below.

Declarative vs functional

It is common to use declarative style in eclasses — create a bunch of variables that ebuilds can use to control the eclass behavior. However, this style has two significant disadvantages.

Firstly, it is prone to typos. If someone recalls the variable name wrong, and its effects are not explicitly visible, it is very easy to commit an ebuild with a silly bug. If the effects are visible, it can still give you some quality debugging headache.

Secondly, in order to affect global scope, the variables need to be set before inherit. This is not trivially enforced, and it is easy to miss that the variable doesn’t work (or partially misbehaves) when set too late.

The alternative is to use functional style, especially for affecting global scope variables. Instead of immediately editing variables in global scope and expecting ebuilds to control the behavior via variables, give them a function to do it:

# my.eclass
my_enable_pytest() {
  IUSE+=" test"
  RESTRICT+=" !test? ( test )"
  BDEPEND+=" test? ( dev-python/pytest[${PYTHON_USEDEP}] )"
  python_test() {
    pytest -vv || die
  }
}

Note that this function is evaluated in ebuild context, so all variables need appending. Its main advantage is that it works independently of where in ebuild it’s called (but if you call it early, remember to append!), and in case of typo you get an explicit error. Example use in ebuild:

# my-1.ebuild
inherit my
IUSE="randomstuff"
RDEPEND="randomstuff? ( dev-libs/random )"
my_enable_pytest

Think what phases to export

Exporting phase functions is often a matter of convenience. However, doing it poorly can cause ebuild writers more pain than if they weren’t exported in the first place. An example of this is vala.eclass as of today. It wrongly exports dysfunctional src_prepare(), and all ebuilds have to redefine it anyway.

It is often a good idea to consider how your eclass is going to be used. If there are both use cases for having the phases exported and for providing utility functions without any phases, it is probably a good idea to split the eclass in two: into -utils eclass that just provides the functions, and main eclass that combines them with phase functions. A good examples today are xdg and xdg-utils eclasses.

When you do need to export phases, it is wortwhile to consider how different eclasses are going to be combined. Generally, a few eclass types could be listed:

  • Unpacking (fetching) eclasses; e.g. git-r3 with src_unpack(),
  • Build system eclasses; e.g. cmake-utils, src_prepare() through src_install(),
  • Post-install eclasses; e.g. xdg, pkg_*inst(), pkg_*rm(),
  • Build environment setup eclasses; e.g. python-single-r1, pkg_setup().

Generally, it’s best to fit your eclass into as few of those as possible. If you do that, there’s a good chance that the ebuild author would be able to combine multiple eclasses easily:

# my-1.ebuild
PYTHON_COMPAT=( python3_7 )
inherit cmake-utils git-r3 python-single-r1

Note that since each of those eclasses uses a different phase function set to do its work, they combine just fine! The inherit order is also irrelevant. If we e.g. need to add llvm to the list, we just have to redefine pkg_setup().

October 13 2019

Improving distfile mirror structure

Michał Górny (mgorny) October 13, 2019, 14:34

The Gentoo distfile mirror network is essential in distributing sources to our users. It offloads upstream download locations, improves throughput and reliability, guarantees distfile persistency.

The current structure of distfile mirrors dates back to 2002. It might have worked well back when we mirrored around 2500 files but it proved not to scale well. Today, mirrors hold almost 70 000 files, and this number has been causing problems for mirror admins.

The most recent discussion on restructuring mirrors started in January 2015. I have started the preliminary research in January 2017, and it resulted in GLEP 75 being created in January 2018. With the actual implementation effort starting in October 2019, I’d like to summarize all the data and update it with fresh statistics.

Continue reading

The Gentoo distfile mirror network is essential in distributing sources to our users. It offloads upstream download locations, improves throughput and reliability, guarantees distfile persistency.

The current structure of distfile mirrors dates back to 2002. It might have worked well back when we mirrored around 2500 files but it proved not to scale well. Today, mirrors hold almost 70 000 files, and this number has been causing problems for mirror admins.

The most recent discussion on restructuring mirrors started in January 2015. I have started the preliminary research in January 2017, and it resulted in GLEP 75 being created in January 2018. With the actual implementation effort starting in October 2019, I’d like to summarize all the data and update it with fresh statistics.

Continue reading

October 04 2019

[gentoo] Network Bridge device for Qemu kvm

Thomas Raschbacher (lordvan) October 04, 2019, 8:53

So I needed to set up qemu+kvm on a new server (After the old one died)

Seems like i forgot to mention how I set up the bridge network on my previous blog post so here you go:

First let me mention that I am using the second physical Interface on the server for the bridge. Depending on your available hardware or use case you might need / want to change this:

So this is fairly simple (if one has a properly configured kernel of course - details on the Gentoo Wiki article on Network Bridges):

First add this in your /etc/conf.d/net (adjust the interface names as needed):

# QEMU / KVM bridge
bridge_br0="enp96s0f1"
config_br0="dhcp"

then add an init script and start it:

cd /etc/init.d; ln -s net.lo net.br0
/etc/init.d/net.br0 start # to test it

So then I get this error when trying to start my kvm/qemu instance:

 * creating qtap (auto allocating one) ...
/usr/sbin/qtap-manipulate: line 28: tunctl: command not found
tunctl failed
 * failed to create qtap interface

seems like I was missing sys-apps/usermode-utilities .. so just emerge that, only to get this:

 * creating qtap (auto allocating one) ...
/usr/sbin/qtap-manipulate: line 29: brctl: command not found
brctl failed
 * failed to create qtap interface

yepp I forgot to install that too ^^ .. so Install net-misc/bridge-utils and now it starts the VM

So I needed to set up qemu+kvm on a new server (After the old one died)

Seems like i forgot to mention how I set up the bridge network on my previous blog post so here you go:

First let me mention that I am using the second physical Interface on the server for the bridge. Depending on your available hardware or use case you might need / want to change this:

So this is fairly simple (if one has a properly configured kernel of course - details on the Gentoo Wiki article on Network Bridges):

First add this in your /etc/conf.d/net (adjust the interface names as needed):

# QEMU / KVM bridge
bridge_br0="enp96s0f1"
config_br0="dhcp"

then add an init script and start it:

cd /etc/init.d; ln -s net.lo net.br0
/etc/init.d/net.br0 start # to test it

So then I get this error when trying to start my kvm/qemu instance:

 * creating qtap (auto allocating one) ...
/usr/sbin/qtap-manipulate: line 28: tunctl: command not found
tunctl failed
 * failed to create qtap interface

seems like I was missing sys-apps/usermode-utilities .. so just emerge that, only to get this:

 * creating qtap (auto allocating one) ...
/usr/sbin/qtap-manipulate: line 29: brctl: command not found
brctl failed
 * failed to create qtap interface

yepp I forgot to install that too ^^ .. so Install net-misc/bridge-utils and now it starts the VM

September 30 2019

"Network is unreachable" error - Gentoo, silly netifrc configuration mistake

Thomas Raschbacher (lordvan) September 30, 2019, 8:36

So on a new install I was just sitting there and wondering .. what did I do wrong .. why do I keep getting those errors:

# ping lordvan.com
connect: Network is unreachable

then I realized something when checking the output of route -n:

 # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 enp96s0f0
192.168.0.254   0.0.0.0         255.255.255.255 UH    2      0        0 enp96s0f0

It should be:

 # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.254   0.0.0.0         UG    2      0        0 enp96s0f0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 enp96s0f0

Turns out I had forgotten something quite simple, yet important: add "default via <router IP>" to /etc/conf.d/net.. So after changing it from

routes_enp96s0f0="192.168.0.254"

to

routes_enp96s0f0="default via 192.168.0.254"

and restarting the interface everything works just fine ;)

Silly mistake, easy fix .. can be a pain to realize what went wrong though .. maybe someone will make the same mistake and find this blog post hopefully to fix it faster than me ;)

So on a new install I was just sitting there and wondering .. what did I do wrong .. why do I keep getting those errors:

# ping lordvan.com
connect: Network is unreachable

then I realized something when checking the output of route -n:

 # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 enp96s0f0
192.168.0.254   0.0.0.0         255.255.255.255 UH    2      0        0 enp96s0f0

It should be:

 # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.254   0.0.0.0         UG    2      0        0 enp96s0f0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 enp96s0f0

Turns out I had forgotten something quite simple, yet important: add "default via <router IP>" to /etc/conf.d/net.. So after changing it from

routes_enp96s0f0="192.168.0.254"

to

routes_enp96s0f0="default via 192.168.0.254"

and restarting the interface everything works just fine ;)

Silly mistake, easy fix .. can be a pain to realize what went wrong though .. maybe someone will make the same mistake and find this blog post hopefully to fix it faster than me ;)

September 22 2019

py3status v3.20 – EuroPython 2019 edition

Alexys Jacob (ultrabug) September 22, 2019, 14:27

Shame on me to post this so long after it happened… Still, that’s a funny story to tell and a lot of thank you to give so let’s go!

The py3status EuroPython 2019 sprint

I’ve attended all EuroPython conferences since 2013. It’s a great event and I encourage everyone to get there!

The last two days of the conference week are meant for Open Source projects collaboration: this is called sprints.

I don’t know why but this year I decided that I would propose a sprint to welcome anyone willing to work on py3status to come and help…

To be honest I was expecting that nobody would be interested so when I sat down at an empty table on saturday I thought that it would remain empty… but hey, I would have worked on py3status anyway so every option was okay!

Then two students came. They ran Windows and Mac OS and never heard of i3wm or py3status but were curious so I showed them. They could read C so I asked them if they could understand how i3status was reading its horrible configuration file… and they did!

Then Oliver Bestwalter (main maintainer of tox) came and told me he was a long time py3status user… followed by Hubert Bryłkowski and Ólafur Bjarni! Wow..

We joined forces to create a py3status module that allows the use of the great PewPew hardware device created by Radomir Dopieralski (which was given to all attendees) to control i3!

And we did it and had a lot of fun!

Oliver’s major contribution

The module itself is awesome okay… but thanks to Oliver’s experience with tox he proposed and contributed one of the most significant feature py3status has had: the ability to import modules from other pypi packages!

The idea is that you have your module or set of modules. Instead of having to contribute them to py3status you could just publish them to pypi and py3status will automatically be able to detect and load them!

The usage of entry points allow custom and more distributed modules creation for our project!

Read more about this amazing feature on the docs.

All of this happened during EuroPython 2019 and I want to extend once again my gratitude to everyone who participated!

Thank you contributors

Version 3.20 is also the work of cool contributors.
See the changelog.

  • Daniel Peukert
  • Kevin Pulo
  • Maxim Baz
  • Piotr Miller
  • Rodrigo Leite
  • lasers
  • luto

Shame on me to post this so long after it happened… Still, that’s a funny story to tell and a lot of thank you to give so let’s go!

The py3status EuroPython 2019 sprint

I’ve attended all EuroPython conferences since 2013. It’s a great event and I encourage everyone to get there!

The last two days of the conference week are meant for Open Source projects collaboration: this is called sprints.

I don’t know why but this year I decided that I would propose a sprint to welcome anyone willing to work on py3status to come and help…

To be honest I was expecting that nobody would be interested so when I sat down at an empty table on saturday I thought that it would remain empty… but hey, I would have worked on py3status anyway so every option was okay!

Then two students came. They ran Windows and Mac OS and never heard of i3wm or py3status but were curious so I showed them. They could read C so I asked them if they could understand how i3status was reading its horrible configuration file… and they did!

Then Oliver Bestwalter (main maintainer of tox) came and told me he was a long time py3status user… followed by Hubert Bryłkowski and Ólafur Bjarni! Wow..

We joined forces to create a py3status module that allows the use of the great PewPew hardware device created by Radomir Dopieralski (which was given to all attendees) to control i3!

And we did it and had a lot of fun!

Oliver’s major contribution

The module itself is awesome okay… but thanks to Oliver’s experience with tox he proposed and contributed one of the most significant feature py3status has had: the ability to import modules from other pypi packages!

The idea is that you have your module or set of modules. Instead of having to contribute them to py3status you could just publish them to pypi and py3status will automatically be able to detect and load them!

The usage of entry points allow custom and more distributed modules creation for our project!

Read more about this amazing feature on the docs.

All of this happened during EuroPython 2019 and I want to extend once again my gratitude to everyone who participated!

Thank you contributors

Version 3.20 is also the work of cool contributors.
See the changelog.

  • Daniel Peukert
  • Kevin Pulo
  • Maxim Baz
  • Piotr Miller
  • Rodrigo Leite
  • lasers
  • luto

September 13 2019

Vim pulling in xorg dependencies in Gentoo

Nathan Zachary (nathanzachary) September 13, 2019, 17:10

Today I went to update one of my Gentoo servers, and noticed that it wanted to pull in a bunch of xorg dependencies. This is a simple music server without any type of graphical environment, so I don’t really want any xorg libraries or other GUI components installed. Looking through the full output, I couldn’t see a direct reason that these components were now requirements.

To troubleshoot, I started adding packages to /etc/portage/package.mask, starting with cairo (which was the package directly requesting the ‘X’ USE flag be enabled). That didn’t get me very far as it still just indicated that GTK+ needed to be installed. After following the dependency chain for a bit, I noticed that something was pulling in libcanberra and found that the default USE flags now include ‘sound’ and that vim now had it enabled. It looks like this USE flag was added between vim-8.1.1486 and vim-8.1.1846.

For my needs, the most straightforward solution was to just remove the ‘sound’ USE flag from vim by adding the following to /etc/portage/package.use:

# grep vim /etc/portage/package.use 
app-editors/vim -sound

Today I went to update one of my Gentoo servers, and noticed that it wanted to pull in a bunch of xorg dependencies. This is a simple music server without any type of graphical environment, so I don’t really want any xorg libraries or other GUI components installed. Looking through the full output, I couldn’t see a direct reason that these components were now requirements.

To troubleshoot, I started adding packages to /etc/portage/package.mask, starting with cairo (which was the package directly requesting the ‘X’ USE flag be enabled). That didn’t get me very far as it still just indicated that GTK+ needed to be installed. After following the dependency chain for a bit, I noticed that something was pulling in libcanberra and found that the default USE flags now include ‘sound’ and that vim now had it enabled. It looks like this USE flag was added between vim-8.1.1486 and vim-8.1.1846.

For my needs, the most straightforward solution was to just remove the ‘sound’ USE flag from vim by adding the following to /etc/portage/package.use:

# grep vim /etc/portage/package.use 
app-editors/vim -sound

September 06 2019

Adobe Flash and Firefox 68+ in Gentoo Linux

Nathan Zachary (nathanzachary) September 06, 2019, 19:17

Though many sites have abandoned Adobe Flash in favour of HTML5 these days, there are still some legacy applications (e.g. older versions of VMWare’s vSphere web client) that depend on it. Recent versions of Firefox in Linux (68+) started failing to load Flash content for me, and it took some digging to find out why. First off, I noticed that the content wouldn’t load even on Adobe’s Flash test page. Second off, I found that the plugin wasn’t listed in Firefox’s about:plugins page.

So, I realised that the problem was due to the Adobe Flash plugin not integrating properly with Firefox. I use Gentoo Linux, so these instructions may not directly apply to other distributions, but I would imagine that the directory structures are at least similar. To start, I made sure that I had installed the www-plugins/adobe-flash ebuild with the ‘npapi’ USE flag enabled:

$ eix adobe-flash
[I] www-plugins/adobe-flash
     Available versions:  (22) 32.0.0.238^ms
       {+nsplugin +ppapi ABI_MIPS="n32 n64 o32" ABI_RISCV="lp64 lp64d" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  32.0.0.238(22)^ms(03:13:05 22/08/19)(nsplugin -ppapi ABI_MIPS="-n32 -n64 -o32" ABI_RISCV="-lp64 -lp64d" ABI_S390="-32 -64" ABI_X86="64 -32 -x32")
     Homepage:            www.adobe.com/products/flashplayer.html get.adobe.com/flashplayer/ helpx.adobe.com/security/products/flash-player.html
     Description:         Adobe Flash Player

That ebuild installs the libflashplayer.so (shared object) in the /usr/lib64/nsbrowser/plugins/ directory by default.

However, through some digging, I found that Firefox 68+ was looking in another directory for the plugin (in my particular situation, that directory was /usr/lib64/mozilla/plugins/, which actually didn’t exist on my system). Seeing as the target directory didn’t exist, I had to firstly create it, and then I decided to symlink the shared object there so that future updates to the www-plugins/adobe-flash package would work without any further manual intervention:

mkdir -p /usr/lib64/mozilla/plugins/
cd $_
ln -s /usr/lib64/nsbrowser/plugins/libflashplayer.so .

After restarting Firefox, the Adobe Flash test page started working as did other sites that use Flash. So, though your particular Linux distribution, version of Firefox, and version of Adobe Flash may require the use of different directories than the ones I referenced above, I hope that these instructions can help you troubleshoot the problem with Adobe Flash not showing in the Firefox about:plugins page.

Though many sites have abandoned Adobe Flash in favour of HTML5 these days, there are still some legacy applications (e.g. older versions of VMWare’s vSphere web client) that depend on it. Recent versions of Firefox in Linux (68+) started failing to load Flash content for me, and it took some digging to find out why. First off, I noticed that the content wouldn’t load even on Adobe’s Flash test page. Second off, I found that the plugin wasn’t listed in Firefox’s about:plugins page.

So, I realised that the problem was due to the Adobe Flash plugin not integrating properly with Firefox. I use Gentoo Linux, so these instructions may not directly apply to other distributions, but I would imagine that the directory structures are at least similar. To start, I made sure that I had installed the www-plugins/adobe-flash ebuild with the ‘npapi’ USE flag enabled:

$ eix adobe-flash
[I] www-plugins/adobe-flash
     Available versions:  (22) 32.0.0.238^ms
       {+nsplugin +ppapi ABI_MIPS="n32 n64 o32" ABI_RISCV="lp64 lp64d" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  32.0.0.238(22)^ms(03:13:05 22/08/19)(nsplugin -ppapi ABI_MIPS="-n32 -n64 -o32" ABI_RISCV="-lp64 -lp64d" ABI_S390="-32 -64" ABI_X86="64 -32 -x32")
     Homepage:            https://www.adobe.com/products/flashplayer.html https://get.adobe.com/flashplayer/ https://helpx.adobe.com/security/products/flash-player.html
     Description:         Adobe Flash Player

That ebuild installs the libflashplayer.so (shared object) in the /usr/lib64/nsbrowser/plugins/ directory by default.

However, through some digging, I found that Firefox 68+ was looking in another directory for the plugin (in my particular situation, that directory was /usr/lib64/mozilla/plugins/, which actually didn’t exist on my system). Seeing as the target directory didn’t exist, I had to firstly create it, and then I decided to symlink the shared object there so that future updates to the www-plugins/adobe-flash package would work without any further manual intervention:

mkdir -p /usr/lib64/mozilla/plugins/
cd $_
ln -s /usr/lib64/nsbrowser/plugins/libflashplayer.so .

After restarting Firefox, the Adobe Flash test page started working as did other sites that use Flash. So, though your particular Linux distribution, version of Firefox, and version of Adobe Flash may require the use of different directories than the ones I referenced above, I hope that these instructions can help you troubleshoot the problem with Adobe Flash not showing in the Firefox about:plugins page.

August 11 2019

AArch64 (arm64) profiles are now stable!

Gentoo News (GentooNews) August 11, 2019, 0:00

The ARM64 project is pleased to announce that all ARM64 profiles are now stable.

While our developers and users have contributed significantly in this accomplishment, we must also thank our Packet sponsor for their contribution. Providing the Gentoo developer community with access to bare metal hardware has accelerated progress in achieving the stabilization of the ARM64 profiles.

About Packet.com

This access has been kindly provided to Gentoo by bare metal cloud Packet via their Works on Arm project. Learn more about their commitment to supporting open source here.

About Gentoo

Gentoo Linux is a free, source-based, rolling release meta distribution that features a high degree of flexibility and high performance. It empowers you to make your computer work for you, and offers a variety of choices at all levels of system configuration.

As a community, Gentoo consists of approximately two hundred developers and over fifty thousand users globally.

Packet.com logo

The ARM64 project is pleased to announce that all ARM64 profiles are now stable.

While our developers and users have contributed significantly in this accomplishment, we must also thank our Packet sponsor for their contribution. Providing the Gentoo developer community with access to bare metal hardware has accelerated progress in achieving the stabilization of the ARM64 profiles.

About Packet.com

This access has been kindly provided to Gentoo by bare metal cloud Packet via their Works on Arm project. Learn more about their commitment to supporting open source here.

About Gentoo

Gentoo Linux is a free, source-based, rolling release meta distribution that features a high degree of flexibility and high performance. It empowers you to make your computer work for you, and offers a variety of choices at all levels of system configuration.

As a community, Gentoo consists of approximately two hundred developers and over fifty thousand users globally.

July 09 2019

Verifying Gentoo election results via Votrify

Michał Górny (mgorny) July 09, 2019, 14:15

Gentoo elections are conducted using a custom software called votify. During the voting period, the developers place their votes in their respective home directories on one of the Gentoo servers. Afterwards, the election officials collect the votes, count them, compare their results and finally announce them.

The simplified description stated above suggests two weak points. Firstly, we rely on honesty of election officials. If they chose to conspire, they could fake the result. Secondly, we rely on honesty of all Infrastructure members, as they could use root access to manipulate the votes (or the collection process).

To protect against possible fraud, we make the elections transparent (but pseudonymous). This means that all votes cast are public, so everyone can count them and verify the result. Furthermore, developers can verify whether their personal vote has been included. Ideally, all developers would do that and therefore confirm that no votes were manipulated.

Currently, we are pretty much implicitly relying on developers doing that, and assuming that no protest implies successful verification. However, this is not really reliable, and given the unfriendly nature of our scripts I have reasons to doubt that the majority of developers actually verify the election results. In this post, I would like to shortly explain how Gentoo elections work, how they could be manipulated and introduce Votrify — a tool to explicitly verify election results.

Gentoo voting process in detail

Once the nomination period is over, an election official sets the voting process up by creating control files for the voting scripts. Those control files include election name, voting period, ballot (containing all vote choices) and list of eligible voters.

There are no explicit events corresponding to the beginning or the end of voting period. The votify script used by developers reads election data on each execution, and uses it to determine whether the voting period is open. During the voting period, it permits the developer to edit the vote, and finally to ‘submit’ it. Both draft and submitted vote are stored as appropriate files in the developer’s home directory, ‘submitted’ votes are not collected automatically. This means that the developer can still manually manipulate the vote once voting period concludes, and before the votes are manually collected.

Votes are collected explicitly by an election official. When run, the countify script collects all vote files from developers’ home directories. An unique ‘confirmation ID’ is generated for each voting developer. All votes along with their confirmation IDs are placed in so-called ‘master ballot’, while mapping from developer names to confirmation IDs is stored separately. The latter is used to send developers their respective confirmation IDs, and can be discarded afterwards.

Each of the election officials uses the master ballot to count the votes. Afterwards, they compare their results and if they match, they announce the election results. The master ballot is attached to the announcement mail, so that everyone can verify the results.

Possible manipulations

The three methods of manipulating the vote that I can think of are:

  1. Announcing fake results. An election result may be presented that does not match the votes cast. This is actively prevented by having multiple election officials, and by making the votes transparent so that everyone can count them.
  2. Manipulating votes cast by developers. The result could be manipulated by modifying the votes cast by individual developers. This is prevented by including pseudonymous vote attribution in the master ballot. Every developer can therefore check whether his/her vote has been reproduced correctly. However, this presumes that the developer is active.
  3. Adding fake votes to the master ballot. The result could be manipulated by adding votes that were not cast by any of the existing developers. This is a major problem, and such manipulation is entirely plausible if the turnout is low enough, and developers who did not vote fail to check whether they have not been added to the casting voter list.

Furthermore, the efficiency of the last method can be improved if the attacker is able to restrict communication between voters and/or reliably deliver different versions of the master ballot to different voters, i.e. convince the voters that their own vote was included correctly while manipulating the remaining votes to achieve the desired result. The former is rather unlikely but the latter is generally feasible.

Finally, the results could be manipulated via manipulating the voting software. This can be counteracted through verifying the implementation against the algorithm specification or, to some degree, via comparing the results a third party tool. Robin H. Johnson and myself were historically working on this (or more specifically, on verifying whether the Gentoo implementation of Schulze method is correct) but neither of us was able to finish the work. If you’re interested in the topic, you can look at my election-compare repository. For the purpose of this post, I’m going to consider this possibility out of scope.

Verifying election results using Votrify

Votrify uses a two-stage verification model. It consists of individual verification which is performed by each voter separately and produces signed confirmations, and community verification that uses the aforementioned files to provide final verified election result.

The individual verification part involves:

  1. Verifying that the developer’s vote has been recorded correctly. This takes part in detecting whether any votes have been manipulated. The positive result of this verification is implied by the fact that a confirmation is produced. Additionally, developers who did not cast a vote also need to produce confirmations, in order to detect any extraneous votes.
  2. Counting the votes and producing the election result. This produces the election results as seen from the developer’s perspective, and therefore prevents manipulation via announcing fake results. Furthermore, comparing the results between different developers helps finding implementation bugs.
  3. Hashing the master ballot. The hash of master ballot file is included, and comparing it between different results confirms that all voters received the same master ballot.

If the verification is positive, a confirmation is produced and signed using developer’s OpenPGP key. I would like to note that no private data is leaked in the process. It does not even indicate whether the dev in question has actually voted — only that he/she participates in the verification process.

Afterwards, confirmations from different voters are collected. They are used to perform community verification which involves:

  1. Verifying the OpenPGP signature. This is necessary to confirm the authenticity of the signed confirmation. The check also involves verifying that the key owner was an eligible voter and that each voter produced only one confirmation. Therefore, it prevents attempts to~fake the verification results.
  2. Comparing the results and master ballot hashes. This confirms that everyone participating received the same master ballot, and produced the same results.

If the verification for all confirmations is positive, the election results are repeated, along with explicit quantification of how trustworthy they are. The number indicates how many confirmations were used, and therefore how many of the votes (or non-votes) in master ballot were confirmed. The difference between the number of eligible voters and the number of confirmations indicates how many votes may have been altered, planted or deleted. Ideally, if all eligible voters produced signed confirmations, the election would be 100% confirmed.

Gentoo elections are conducted using a custom software called votify. During the voting period, the developers place their votes in their respective home directories on one of the Gentoo servers. Afterwards, the election officials collect the votes, count them, compare their results and finally announce them.

The simplified description stated above suggests two weak points. Firstly, we rely on honesty of election officials. If they chose to conspire, they could fake the result. Secondly, we rely on honesty of all Infrastructure members, as they could use root access to manipulate the votes (or the collection process).

To protect against possible fraud, we make the elections transparent (but pseudonymous). This means that all votes cast are public, so everyone can count them and verify the result. Furthermore, developers can verify whether their personal vote has been included. Ideally, all developers would do that and therefore confirm that no votes were manipulated.

Currently, we are pretty much implicitly relying on developers doing that, and assuming that no protest implies successful verification. However, this is not really reliable, and given the unfriendly nature of our scripts I have reasons to doubt that the majority of developers actually verify the election results. In this post, I would like to shortly explain how Gentoo elections work, how they could be manipulated and introduce Votrify — a tool to explicitly verify election results.

Gentoo voting process in detail

Once the nomination period is over, an election official sets the voting process up by creating control files for the voting scripts. Those control files include election name, voting period, ballot (containing all vote choices) and list of eligible voters.

There are no explicit events corresponding to the beginning or the end of voting period. The votify script used by developers reads election data on each execution, and uses it to determine whether the voting period is open. During the voting period, it permits the developer to edit the vote, and finally to ‘submit’ it. Both draft and submitted vote are stored as appropriate files in the developer’s home directory, ‘submitted’ votes are not collected automatically. This means that the developer can still manually manipulate the vote once voting period concludes, and before the votes are manually collected.

Votes are collected explicitly by an election official. When run, the countify script collects all vote files from developers’ home directories. An unique ‘confirmation ID’ is generated for each voting developer. All votes along with their confirmation IDs are placed in so-called ‘master ballot’, while mapping from developer names to confirmation IDs is stored separately. The latter is used to send developers their respective confirmation IDs, and can be discarded afterwards.

Each of the election officials uses the master ballot to count the votes. Afterwards, they compare their results and if they match, they announce the election results. The master ballot is attached to the announcement mail, so that everyone can verify the results.

Possible manipulations

The three methods of manipulating the vote that I can think of are:

  1. Announcing fake results. An election result may be presented that does not match the votes cast. This is actively prevented by having multiple election officials, and by making the votes transparent so that everyone can count them.
  2. Manipulating votes cast by developers. The result could be manipulated by modifying the votes cast by individual developers. This is prevented by including pseudonymous vote attribution in the master ballot. Every developer can therefore check whether his/her vote has been reproduced correctly. However, this presumes that the developer is active.
  3. Adding fake votes to the master ballot. The result could be manipulated by adding votes that were not cast by any of the existing developers. This is a major problem, and such manipulation is entirely plausible if the turnout is low enough, and developers who did not vote fail to check whether they have not been added to the casting voter list.

Furthermore, the efficiency of the last method can be improved if the attacker is able to restrict communication between voters and/or reliably deliver different versions of the master ballot to different voters, i.e. convince the voters that their own vote was included correctly while manipulating the remaining votes to achieve the desired result. The former is rather unlikely but the latter is generally feasible.

Finally, the results could be manipulated via manipulating the voting software. This can be counteracted through verifying the implementation against the algorithm specification or, to some degree, via comparing the results a third party tool. Robin H. Johnson and myself were historically working on this (or more specifically, on verifying whether the Gentoo implementation of Schulze method is correct) but neither of us was able to finish the work. If you’re interested in the topic, you can look at my election-compare repository. For the purpose of this post, I’m going to consider this possibility out of scope.

Verifying election results using Votrify

Votrify uses a two-stage verification model. It consists of individual verification which is performed by each voter separately and produces signed confirmations, and community verification that uses the aforementioned files to provide final verified election result.

The individual verification part involves:

  1. Verifying that the developer’s vote has been recorded correctly. This takes part in detecting whether any votes have been manipulated. The positive result of this verification is implied by the fact that a confirmation is produced. Additionally, developers who did not cast a vote also need to produce confirmations, in order to detect any extraneous votes.
  2. Counting the votes and producing the election result. This produces the election results as seen from the developer’s perspective, and therefore prevents manipulation via announcing fake results. Furthermore, comparing the results between different developers helps finding implementation bugs.
  3. Hashing the master ballot. The hash of master ballot file is included, and comparing it between different results confirms that all voters received the same master ballot.

If the verification is positive, a confirmation is produced and signed using developer’s OpenPGP key. I would like to note that no private data is leaked in the process. It does not even indicate whether the dev in question has actually voted — only that he/she participates in the verification process.

Afterwards, confirmations from different voters are collected. They are used to perform community verification which involves:

  1. Verifying the OpenPGP signature. This is necessary to confirm the authenticity of the signed confirmation. The check also involves verifying that the key owner was an eligible voter and that each voter produced only one confirmation. Therefore, it prevents attempts to~fake the verification results.
  2. Comparing the results and master ballot hashes. This confirms that everyone participating received the same master ballot, and produced the same results.

If the verification for all confirmations is positive, the election results are repeated, along with explicit quantification of how trustworthy they are. The number indicates how many confirmations were used, and therefore how many of the votes (or non-votes) in master ballot were confirmed. The difference between the number of eligible voters and the number of confirmations indicates how many votes may have been altered, planted or deleted. Ideally, if all eligible voters produced signed confirmations, the election would be 100% confirmed.

Autoclicker for Linux

Thomas Raschbacher (lordvan) July 09, 2019, 14:02

So I wanted an autoclicker for linux - for one of my browser based games that require a lot of clicking.

Looked around and tried to find something useful, but all i could find was old pages outdated download links,..

In the end I stumbled upon something simple yet immensely more powerful:xdotool (github) or check out the xdotool website

As an extra bonus it is in the Gentoo repository so a simple

emerge xdotool

Got it installed. it also has minimal dependencies which is nice.

The good part, but also a bit of a downside is that there is no UI (maybe I'll write one when I get a chance .. just as a wrapper).

anyway to do what I wanted was simply this:

xdotool click --repeat 1000 --delay 100 1

Pretty self explainatory, but here's a short explaination anyway:

  • click .. simulate a mouse click
  • --repeat 1000 ... repeat 1000 times
  • --delay 100 ... wait 100ms between clicks
  • 1  .. mouse button 1

The only problem is I need to know how many clicks I need beforehand - which can also be a nice feature of course.

There is one way to stop it if you have the terminal you ran this command from visible (which i always have - and set it to always on top): click with your left mouse button - this stops the click events being registered since it is mouse-down and waits for mouse-up i guess .. but not sure if that is the reason. then move to the terminal and either close it or ctrl+c abort the command -- or just wait for the program to exit after finishing the requested number of clicks. -- On a side note if you don't like that way of stopping it you could always just ctrl+alt+f1 (or whatever terminal you want to use) and log in there and kill the xdotool process (either find thepid and kill it or just killall xdotool - which will of course kill all, but i doubt you'll run more than one at once)

So I wanted an autoclicker for linux - for one of my browser based games that require a lot of clicking.

Looked around and tried to find something useful, but all i could find was old pages outdated download links,..

In the end I stumbled upon something simple yet immensely more powerful:xdotool (github) or check out the xdotool website

As an extra bonus it is in the Gentoo repository so a simple

emerge xdotool

Got it installed. it also has minimal dependencies which is nice.

The good part, but also a bit of a downside is that there is no UI (maybe I'll write one when I get a chance .. just as a wrapper).

anyway to do what I wanted was simply this:

xdotool click --repeat 1000 --delay 100 1

Pretty self explainatory, but here's a short explaination anyway:

  • click .. simulate a mouse click
  • --repeat 1000 ... repeat 1000 times
  • --delay 100 ... wait 100ms between clicks
  • 1  .. mouse button 1

The only problem is I need to know how many clicks I need beforehand - which can also be a nice feature of course.

There is one way to stop it if you have the terminal you ran this command from visible (which i always have - and set it to always on top): click with your left mouse button - this stops the click events being registered since it is mouse-down and waits for mouse-up i guess .. but not sure if that is the reason. then move to the terminal and either close it or ctrl+c abort the command -- or just wait for the program to exit after finishing the requested number of clicks. -- On a side note if you don't like that way of stopping it you could always just ctrl+alt+f1 (or whatever terminal you want to use) and log in there and kill the xdotool process (either find thepid and kill it or just killall xdotool - which will of course kill all, but i doubt you'll run more than one at once)

July 04 2019

SKS poisoning, keys.openpgp.org / Hagrid and other non-solutions

Michał Górny (mgorny) July 04, 2019, 11:23

The recent key poisoning attack on SKS keyservers shook the world of OpenPGP. While this isn’t a new problem, it has not been exploited on this scale before. The attackers have proved how easy it is to poison commonly used keys on the keyservers and effectively render GnuPG unusably slow. A renewed discussion on improving keyservers has started as a result. It also forced Gentoo to employ countermeasures. You can read more on them in the ‘Impact of SKS keyserver poisoning on Gentoo’ news item.

Coincidentally, the attack happened shortly after the launch of keys.openpgp.org, that advertises itself as both poisoning-resistant and GDPR-friendly keyserver. Naturally, many users see it as the ultimate solution to the issues with SKS. I’m afraid I have to disagree — in my opinion, this keyserver does not solve any problems, it merely cripples OpenPGP in order to avoid being affected by them, and harms its security in the process.

In this article, I’d like to shortly explain what the problem is, and which of the different solutions proposed so far to it (e.g. on gnupg-users mailing list) make sense, and which make things even worse. Naturally, I will also cover the new Hagrid keyserver as one of the glorified non-solutions.

The attack — key poisoning

OpenPGP uses a distributed design — once the primary key is created, additional packets can be freely appended to it and recombined on different systems. Those packets include subkeys, user identifiers and signatures. Signatures are used to confirm the authenticity of appended packets. The packets are only meaningful if the client can verify the authenticity of their respective signatures.

The attack is carried through third-party signatures that normally are used by different people to confirm the authenticity of the key — that is, to state that the signer has verified the identity of the key owner. It relies on three distinct properties of OpenPGP:

  1. The key can contain unlimited number of signatures. After all, it is natural that very old keys will have a large number of signatures made by different people on them.
  2. Anyone can append signatures to any OpenPGP key. This is partially keyserver policy, and partially the fact that SKS keyserver nodes are propagating keys one to another.
  3. There is no way to distinguish legitimate signatures from garbage. To put it other way, it is trivial to make garbage signatures look like the real deal.

The attacker abuses those properties by creating a large number of garbage signatures and sending them to keyservers. When users fetch key updates from the keyserver, GnuPG normally appends all those signatures to the local copy. As a result, the key becomes unusually large and causes severe performance issues with GnuPG, preventing its normal usage. The user ends up having to manually remove the key in order to fix the installation.

The obvious non-solutions and potential solutions

Let’s start by analyzing the properties I’ve listed above. After all, removing at least one of the requirements should prevent the attack from being possible. But can we really do that?

Firstly, we could set a hard limit on number of signatures or key size. This should obviously prevent the attacker from breaking user systems via huge keys. However, it will make it entirely possible for the attacker to ‘brick’ the key by appending garbage up to the limit. Then it would no longer be possible to append any valid signatures to the key. Users would suffer less but the key owner will lose the ability to use the key meaningfully. It’s a no-go.

Secondly, we could limit key updates to the owner. However, the keyserver update protocol currently does not provide any standard way of verifying who the uploader is, so it would effectively require incompatible changes at least to the upload protocol. Furthermore, in order to prevent malicious keyservers from propagating fake signatures we’d also need to carry the verification along when propagating key updates. This effectively means an extension of the key format, and it has been proposed e.g. in ‘Abuse-Resistant OpenPGP Keystores’ draft. This is probably a wortwhile option but it will take time before it’s implemented.

Thirdly, we could try to validate signatures. However, any validation can be easily worked around. If we started requiring signing keys to be present on the keyserver, the attackers can simply mass-upload keys used to create garbage signatures. If we went even further and e.g. started requiring verified e-mail addresses for the signing keys, the attackers can simply mass-create e-mail addresses and verify them. It might work as a temporary solution but it will probably cause more harm than good.

There were other non-solutions suggested — most notably, blacklisting poisoned keys. However, this is even worse. It means that every victim of poisoning attack would be excluded from using the keyserver, and in my opinion it will only provoke the attackers to poison even more keys. It may sound like a good interim solution preventing users from being hit but it is rather short-sighted.

keys.openpgp.org / Hagrid — a big non-solution

A common suggestion for OpenPGP users — one that even Gentoo news item mentions for lack of alternative — is to switch to keys.openpgp.org keyserver, or switch keyservers to their Hagrid software. It is not vulnerable to key poisoning attack because it strips away all third-party signatures. However, this and other limitations make it a rather poor replacement, and in my opinion can be harmful to security of OpenPGP.

Firstly, stripping all third-party signatures is not a solution. It simply avoids the problem by killing a very important portion of OpenPGP protocol — the Web of Trust. Without it, the keys obtained from the server can not be authenticated otherwise than by direct interaction between the individuals. For example, Gentoo Authority Keys can’t work there. Most of the time, you won’t be able to tell whether the key on keyserver is legitimate or forged.

The e-mail verification makes it even worse, though not intentionally. While I agree that many users do not understand or use WoT, Hagrid is implicitly going to cause users to start relying on e-mail verification as proof of key authenticity. In other words, people are going to assume that if a key on keys.openpgp.org has verified e-mail address, it has to be legitimate. This makes it trivial for an attacker that manages to gain unauthorized access to the e-mail address or the keyserver to publish a forged key and convince others to use it.

Secondly, Hagrid does not support UID revocations. This is an entirely absurd case where GDPR fear won over security. If your e-mail address becomes compromised, you will not be able to revoke it. Sure, the keyserver admins may eventually stop propagating it along with your key, but all users who fetched the key before will continue seeing it as a valid UID. Of course, if users send encrypted mail the attacker won’t be able to read it. However, the users can be trivially persuaded to switch to a new, forged key.

Thirdly, Hagrid rejects all UIDs except for verified e-mail-based UIDs. This is something we could live with if key owners actively pursue having their identities verified. However, this also means you can’t publish a photo identity or use keybase.io. The ‘explicit consent’ argument used by upstream is rather silly — apparently every UID requires separate consent, while at the same time you can trivially share somebody else’s PII as the real name of a valid e-mail address.

Apparently, upstream is willing to resolve the first two of those issues once satisfactory solutions are established. However, this doesn’t mean that it’s fine to ignore those problems. Until they are resolved, and necessary OpenPGP client updates are sufficiently widely deployed, I don’t believe Hagrid or its instance at keys.openpgp.org are good replacements for SKS and other keyservers.

So what are the solutions?

Sadly, I am not aware of any good global solution at the moment. The best workaround for GnuPG users so far is the new self-sigs-only option that prevents it from importing third-party signatures. Of course, it shares the first limitation of Hagrid keyserver. The future versions of GnuPG will supposedly fallback to this option upon meeting excessively large keys.

For domain-limited use cases such as Gentoo’s, running a local keyserver with restricted upload access is an option. However, it requires users to explicitly specify our keyserver, and effectively end up having to specify multiple different keyservers for each domain. Furthermore, WKD can be used to distribute keys. Sadly, at the moment GnuPG uses it only to locate new keys and does not support refreshing keys via WKD (gemato employs a cheap hack to make it happen). In both cases, the attack is prevented via isolating the infrastructure and preventing public upload access.

The long-term solution probably lies in the ‘First-party-attested Third-party Certifications‘ section of the ‘Abuse-Resistant OpenPGP Keystores’ draft. In this proposal, every third-party signature must be explicitly attested by the key owner. Therefore, only the key owner can append additional signatures to the key, and keyservers can reject any signatures that were not attested. However, this is not currently supported by GnuPG, and once it is, deploying it will most likely take significant time.

The recent key poisoning attack on SKS keyservers shook the world of OpenPGP. While this isn’t a new problem, it has not been exploited on this scale before. The attackers have proved how easy it is to poison commonly used keys on the keyservers and effectively render GnuPG unusably slow. A renewed discussion on improving keyservers has started as a result. It also forced Gentoo to employ countermeasures. You can read more on them in the ‘Impact of SKS keyserver poisoning on Gentoo’ news item.

Coincidentally, the attack happened shortly after the launch of keys.openpgp.org, that advertises itself as both poisoning-resistant and GDPR-friendly keyserver. Naturally, many users see it as the ultimate solution to the issues with SKS. I’m afraid I have to disagree — in my opinion, this keyserver does not solve any problems, it merely cripples OpenPGP in order to avoid being affected by them, and harms its security in the process.

In this article, I’d like to shortly explain what the problem is, and which of the different solutions proposed so far to it (e.g. on gnupg-users mailing list) make sense, and which make things even worse. Naturally, I will also cover the new Hagrid keyserver as one of the glorified non-solutions.

The attack — key poisoning

OpenPGP uses a distributed design — once the primary key is created, additional packets can be freely appended to it and recombined on different systems. Those packets include subkeys, user identifiers and signatures. Signatures are used to confirm the authenticity of appended packets. The packets are only meaningful if the client can verify the authenticity of their respective signatures.

The attack is carried through third-party signatures that normally are used by different people to confirm the authenticity of the key — that is, to state that the signer has verified the identity of the key owner. It relies on three distinct properties of OpenPGP:

  1. The key can contain unlimited number of signatures. After all, it is natural that very old keys will have a large number of signatures made by different people on them.
  2. Anyone can append signatures to any OpenPGP key. This is partially keyserver policy, and partially the fact that SKS keyserver nodes are propagating keys one to another.
  3. There is no way to distinguish legitimate signatures from garbage. To put it other way, it is trivial to make garbage signatures look like the real deal.

The attacker abuses those properties by creating a large number of garbage signatures and sending them to keyservers. When users fetch key updates from the keyserver, GnuPG normally appends all those signatures to the local copy. As a result, the key becomes unusually large and causes severe performance issues with GnuPG, preventing its normal usage. The user ends up having to manually remove the key in order to fix the installation.

The obvious non-solutions and potential solutions

Let’s start by analyzing the properties I’ve listed above. After all, removing at least one of the requirements should prevent the attack from being possible. But can we really do that?

Firstly, we could set a hard limit on number of signatures or key size. This should obviously prevent the attacker from breaking user systems via huge keys. However, it will make it entirely possible for the attacker to ‘brick’ the key by appending garbage up to the limit. Then it would no longer be possible to append any valid signatures to the key. Users would suffer less but the key owner will lose the ability to use the key meaningfully. It’s a no-go.

Secondly, we could limit key updates to the owner. However, the keyserver update protocol currently does not provide any standard way of verifying who the uploader is, so it would effectively require incompatible changes at least to the upload protocol. Furthermore, in order to prevent malicious keyservers from propagating fake signatures we’d also need to carry the verification along when propagating key updates. This effectively means an extension of the key format, and it has been proposed e.g. in ‘Abuse-Resistant OpenPGP Keystores’ draft. This is probably a wortwhile option but it will take time before it’s implemented.

Thirdly, we could try to validate signatures. However, any validation can be easily worked around. If we started requiring signing keys to be present on the keyserver, the attackers can simply mass-upload keys used to create garbage signatures. If we went even further and e.g. started requiring verified e-mail addresses for the signing keys, the attackers can simply mass-create e-mail addresses and verify them. It might work as a temporary solution but it will probably cause more harm than good.

There were other non-solutions suggested — most notably, blacklisting poisoned keys. However, this is even worse. It means that every victim of poisoning attack would be excluded from using the keyserver, and in my opinion it will only provoke the attackers to poison even more keys. It may sound like a good interim solution preventing users from being hit but it is rather short-sighted.

keys.openpgp.org / Hagrid — a big non-solution

A common suggestion for OpenPGP users — one that even Gentoo news item mentions for lack of alternative — is to switch to keys.openpgp.org keyserver, or switch keyservers to their Hagrid software. It is not vulnerable to key poisoning attack because it strips away all third-party signatures. However, this and other limitations make it a rather poor replacement, and in my opinion can be harmful to security of OpenPGP.

Firstly, stripping all third-party signatures is not a solution. It simply avoids the problem by killing a very important portion of OpenPGP protocol — the Web of Trust. Without it, the keys obtained from the server can not be authenticated otherwise than by direct interaction between the individuals. For example, Gentoo Authority Keys can’t work there. Most of the time, you won’t be able to tell whether the key on keyserver is legitimate or forged.

The e-mail verification makes it even worse, though not intentionally. While I agree that many users do not understand or use WoT, Hagrid is implicitly going to cause users to start relying on e-mail verification as proof of key authenticity. In other words, people are going to assume that if a key on keys.openpgp.org has verified e-mail address, it has to be legitimate. This makes it trivial for an attacker that manages to gain unauthorized access to the e-mail address or the keyserver to publish a forged key and convince others to use it.

Secondly, Hagrid does not support UID revocations. This is an entirely absurd case where GDPR fear won over security. If your e-mail address becomes compromised, you will not be able to revoke it. Sure, the keyserver admins may eventually stop propagating it along with your key, but all users who fetched the key before will continue seeing it as a valid UID. Of course, if users send encrypted mail the attacker won’t be able to read it. However, the users can be trivially persuaded to switch to a new, forged key.

Thirdly, Hagrid rejects all UIDs except for verified e-mail-based UIDs. This is something we could live with if key owners actively pursue having their identities verified. However, this also means you can’t publish a photo identity or use keybase.io. The ‘explicit consent’ argument used by upstream is rather silly — apparently every UID requires separate consent, while at the same time you can trivially share somebody else’s PII as the real name of a valid e-mail address.

Apparently, upstream is willing to resolve the first two of those issues once satisfactory solutions are established. However, this doesn’t mean that it’s fine to ignore those problems. Until they are resolved, and necessary OpenPGP client updates are sufficiently widely deployed, I don’t believe Hagrid or its instance at keys.openpgp.org are good replacements for SKS and other keyservers.

So what are the solutions?

Sadly, I am not aware of any good global solution at the moment. The best workaround for GnuPG users so far is the new self-sigs-only option that prevents it from importing third-party signatures. Of course, it shares the first limitation of Hagrid keyserver. The future versions of GnuPG will supposedly fallback to this option upon meeting excessively large keys.

For domain-limited use cases such as Gentoo’s, running a local keyserver with restricted upload access is an option. However, it requires users to explicitly specify our keyserver, and effectively end up having to specify multiple different keyservers for each domain. Furthermore, WKD can be used to distribute keys. Sadly, at the moment GnuPG uses it only to locate new keys and does not support refreshing keys via WKD (gemato employs a cheap hack to make it happen). In both cases, the attack is prevented via isolating the infrastructure and preventing public upload access.

The long-term solution probably lies in the ‘First-party-attested Third-party Certifications‘ section of the ‘Abuse-Resistant OpenPGP Keystores’ draft. In this proposal, every third-party signature must be explicitly attested by the key owner. Therefore, only the key owner can append additional signatures to the key, and keyservers can reject any signatures that were not attested. However, this is not currently supported by GnuPG, and once it is, deploying it will most likely take significant time.

July 03 2019

Case label for Pocket Science Lab V5

Marek Szuba (marecki) July 03, 2019, 17:28

tl;dr: Here (PDF, 67 kB) is a case label for Pocket Science Lab version 5 that is compatible with the design for a laser-cut case published by FOSSAsia.

In case you haven’t heard about it, Pocket Science Lab [1] is a really nifty board developed by the FOSSAsia community which combines a multichannel, megahertz-range oscilloscope, a multimeter, a logic probe, several voltage sources and a current source, several wave generators, UART and I2C interfaces… and all of this in the form factor of an Arduino Mega, i.e. only somewhat larger than that of a credit card. Hook it up over USB to a PC or an Android device running the official (free and open source, of course) app and you are all set.

Well, not quite set yet. What you get for your 50-ish EUR is just the board itself. You will quite definitely need a set of probe cables (sadly, I have yet to find even an unofficial adaptor allowing one to equip PSLab with standard industry oscilloscope probes using BNC connectors) but if you expect to lug yours around anywhere you go, you will quite definitely want to invest in a case of some sort. While FOSSAsia does not to my knowledge sell PSLab cases, they provide a design for one [2]. It is meant to be laser-cut but I have successfully managed to 3D-print it as well, and for the more patient among us it shouldn’t be too difficult to hand-cut one with a jigsaw either.

Of course in addition to making sure your Pocket Science Lab is protected against accidental damage it would also be nice to have all the connectors clearly labelled. Documentation bundled with PSLab software does show not a few “how to connect instrument X” diagrams but unfortunately said diagrams picture a version 4 of the board and the current major version, V5, features radically different pinout (compare [3] with [4]/[5] and you will see immediately what I mean), not to mention that having to stare at a screen while wiring your circuit isn’t always optimal. Now, all versions of the board feature a complete set of header labels (along with LEDs showing the device is active) on the front side and at least the more recent ones additionally show more detailed descriptions on the back, clearly suggesting the optimal way to go is to make your case our of transparent material. But what if looking at the provided labels directly is not an option, for instance because you have gone eco-friendly and made your case out of wood? Probably stick a label to the front of the case… which brings us back to the problem of the case label from [5] not being compatible with recent versions of the board.

Which brings me to my take on adapting the design from [5] to match the header layout and labels of PSLab V5.1 as well as the laser-cut case design from [2]. It could probably be more accurate but having tried it out, it is close enough. Bluetooth and ICSP-programmer connectors near the centre of the board are not included because the current case design does not provide access to them and indeed, they haven’t even got headers soldered in. Licence and copyright: same as the original.


[1] pslab.io/

[2] github.com/fossasia/pslab-case

[3] github.com/fossasia/pslab-hardware/raw/master/docs/images/PSLab_v5_top.png

[4] github.com/fossasia/pslab-hardware/raw/master/docs/images/pslab_version_previews/PSLab_v4.png

[5] github.com/fossasia/pslab-hardware/raw/master/docs/images/pslabdesign.png

tl;dr: Here (PDF, 67 kB) is a case label for Pocket Science Lab version 5 that is compatible with the design for a laser-cut case published by FOSSAsia.


In case you haven’t heard about it, Pocket Science Lab [1] is a really nifty board developed by the FOSSAsia community which combines a multichannel, megahertz-range oscilloscope, a multimeter, a logic probe, several voltage sources and a current source, several wave generators, UART and I2C interfaces… and all of this in the form factor of an Arduino Mega, i.e. only somewhat larger than that of a credit card. Hook it up over USB to a PC or an Android device running the official (free and open source, of course) app and you are all set.

Well, not quite set yet. What you get for your 50-ish EUR is just the board itself. You will quite definitely need a set of probe cables (sadly, I have yet to find even an unofficial adaptor allowing one to equip PSLab with standard industry oscilloscope probes using BNC connectors) but if you expect to lug yours around anywhere you go, you will quite definitely want to invest in a case of some sort. While FOSSAsia does not to my knowledge sell PSLab cases, they provide a design for one [2]. It is meant to be laser-cut but I have successfully managed to 3D-print it as well, and for the more patient among us it shouldn’t be too difficult to hand-cut one with a jigsaw either.

Of course in addition to making sure your Pocket Science Lab is protected against accidental damage it would also be nice to have all the connectors clearly labelled. Documentation bundled with PSLab software does show not a few “how to connect instrument X” diagrams but unfortunately said diagrams picture a version 4 of the board and the current major version, V5, features radically different pinout (compare [3] with [4]/[5] and you will see immediately what I mean), not to mention that having to stare at a screen while wiring your circuit isn’t always optimal. Now, all versions of the board feature a complete set of header labels (along with LEDs showing the device is active) on the front side and at least the more recent ones additionally show more detailed descriptions on the back, clearly suggesting the optimal way to go is to make your case our of transparent material. But what if looking at the provided labels directly is not an option, for instance because you have gone eco-friendly and made your case out of wood? Probably stick a label to the front of the case… which brings us back to the problem of the case label from [5] not being compatible with recent versions of the board.

Which brings me to my take on adapting the design from [5] to match the header layout and labels of PSLab V5.1 as well as the laser-cut case design from [2]. It could probably be more accurate but having tried it out, it is close enough. Bluetooth and ICSP-programmer connectors near the centre of the board are not included because the current case design does not provide access to them and indeed, they haven’t even got headers soldered in. Licence and copyright: same as the original.


[1]
https://pslab.io/

[2]
https://github.com/fossasia/pslab-case

[3]
https://github.com/fossasia/pslab-hardware/raw/master/docs/images/PSLab_v5_top.png

[4]
https://github.com/fossasia/pslab-hardware/raw/master/docs/images/pslab_version_previews/PSLab_v4.png

[5]
https://github.com/fossasia/pslab-hardware/raw/master/docs/images/pslabdesign.png

VIEW

SCOPE

FILTER
  from
  to