• Patching
• Eclasses
• Dependencies
• Bad tests
• Test phase

The file lit.site.cfg has to be inspected for any incorrect calls to executables. For example see src_prepare function form dev-lang/boogie.

Because we will need to specify how many threads should lit run we need to inherit multiprocessing to detect how many parallel jobs the portage config sets.

inherit multiprocessing

Ensure that dev-python/lit is in BDEPEND, but also additional packages may be needed, for example dev-python/OutputCheck.

BDEPEND="
    ${RDEPEND}
    test? (
	dev-python/lit
	dev-python/OutputCheck
    )
"

To deal with bad test you can simply remove the files causing the failures.

local -a bad_tests=(
    civl/inductive-sequentialization/BroadcastConsensus.bpl
    civl/inductive-sequentialization/PingPong.bpl
    livevars/bla1.bpl
)
local bad_test
for bad_test in ${bad_tests[@]} ; do
    rm "${S}"/Test/${bad_test} || die
done

--threads $(makeopts_jobs) specifies how many parallel tests to run.

--verbose option will show output of failed tests.

Last lit argument specifies where lit should look for lit.site.cfg and tests.

src_test() {
    lit --threads $(makeopts_jobs) --verbose "${S}"/Test || die
}

♦

Do you want to learn more about Gentoo and contribute to your favourite free software project?! Once again, now for the 11th time, we have been accepted as a mentoring organization for this year’s Google Summer of Code!

The GSoC is an excellent opportunity for gaining real-world experience in software design and making oneself known in the broader open source community. It also looks great on a resume. Some initial project ideas can be found here, but new projects ideas are also welcome. For new projects time is of the essence: they have to be worked out, discussed with the mentors, and submitted before the April 4th deadline. It is strongly recommended that contributors refine new project ideas with a mentor before proposing the idea formally.

Potential GSoC contributors are encouraged to e-mail the GSoC admins with their name, IRC nickname, and the desired project, and discuss ideas in the #gentoo-soc IRC channel on Libera Chat. Further information can be found on the Gentoo GSoC 2023 wiki page. Those with unanswered questions should also not hesitate to contact the Summer of Code mentors via their mailing list.

♦ A quite late Happy New Year 2023 to all of you!

Once again with 2022 an eventful year has passed, and Gentoo is still alive and kicking! 2023 already started some time ago and some of us have even already been meeting up and networking at FOSDEM 2023. Still, we are happy to present once more a review of the Gentoo news of the past year 2022. Read on for new developers, distribution wide initiatives and improvements, up-to-date numbers on Gentoo development, tales from the infrastructure, and all the fresh new packages you can emerge now.

The number of commits to the main ::gentoo repository has remained at high level in 2022, from 126920 to 126682. This is also true for the number of commits by external contributors, 10492, now across an even increased 440 unique external authors compared to 435 last year.

GURU, our user-curated repository with a trusted user model, is clearly growing further. We have had 5761 commits in 2022, up by 12% from 5131 in 2021. The number of contributors to GURU has increased similarly, from 125 in 2021 to 144 in 2022. Please join us there and help packaging the latest and greatest software. That’s the ideal preparation for becoming a full Gentoo developer!

On the Gentoo bugtracker bugs.gentoo.org, both the number of reported and of resolved bugs has increased clearly. We’ve had 26362 bug reports created in 2022, compared to 24056 in 2021. The number of resolved bugs shows a similar trend, with 24499 in 2022 compared to 24076 in 2021.

In 2022 we have gained four new Gentoo developers. They are in chronological order:

1. Matthew Smith (matthew): ♦ Matthew joined us already in February from the North East of England. By trade embedded software developer, he helps with a diverse set of packages, from mold to erlang and from nasm to tree-sitter.

2. WANG Xuerui (xen0n): ♦ A long-time Gentoo user, Xuerui joined us as a developer in March from Shanghai, China. He jumped in right into the deep end, bringing LoongArch support to Gentoo as well as lots of toolchain and qemu expertise (as long as his cat lets him).

3. Kenton Groombridge (concord): ♦ Kenton comes from the US and from a real Gentoo family (yes, such a thing exists!); he joined up in May. His speciality is Gentoo Hardened and SELinux, and he has already collected quite some commits there!

4. Viorel Munteanu (ceamac): ♦ In November, Viorel joined us from Bucharest, Romania. He’s active in the virtualization and proxy maintainers teams, and takes care of the VirtualBox stack and, e.g., TigerVNC.

Let’s now look at the major improvements and news of 2022 in Gentoo.

• ♦ LiveGUI Gentoo ISO download: For an instant, full-fledged Gentoo experience we now have a weekly-built 3.7GByte amd64 LiveGUI ISO ready for download. It is suitable for booting from DVDs or USB sticks, and boots into a full KDE Plasma desktop based on stable Gentoo. A ton of ready-to-use software is included, from dozens of system utilities, LibreOffice, Inkscape, and TeXLive all the way to Firefox and Chromium. Also, all build dependencies are installed and you can emerge additional packages as you like!

• Modern C porting: This recent cross-distribution initiative has as its objective to port as much open source software as possible to modern C standards. Upcoming versions of GCC and Clang will eventually lose support for constructs that have been deprecated for decades, and we will have to be prepared for that. Together with Fedora we have taken the lead here, and a lot of effort has already gone into fixing and modernization.

• ♦ Clang / LLVM as primary system compiler: Closely related, support for using Clang as the primary system compiler in Gentoo has never been better than now. For the most popular architectures, we have LLVM stages available which replace the GNU toolchain as far as possible (also using libc++, compiler-rc, lld, …) While glibc at the moment still requires GCC to build, the LLVM/musl stages come fully without GNU toolchain.

• New binary package format gpkg: Gentoo’s package manager Portage now supports a new binary package format defined in GLEP 78. Besides many minor improvements, the most important new feature of the file format is that it fully supports cryptographic signing of packages. This was one of the most important roadblocks for more extensive binary package support in Gentoo.

• ♦ merged-usr profiles and systemd merged-usr stages: All systemd profiles have now gained a merged-usr subprofile, corresponding to a filesystem layout where, e.g., /bin is a symbolic link to /usr/bin. The migration procedure has been described in detail in a news item. With this, we prepare for the time when systemd will only support the merged-usr layout anymore, as already announced by the upstream developers. Across all architectures, we also now consistently offer in addition to openrc downloads systemd stages with and without merged-usr layout. Merged-usr openrc stages will follow for completeness.

• ♦ LoongArch64: In the meantime, LoongArch64, a Chinese development by Loongson Co. based in parts on MIPS and on RISC-V, has become a fully supported Gentoo architecture, with toolchain support, widespread keywording, and up-to-date stages for download. First server-type chipsets based on these chips are currently being sold. (Outside mainland China hardware is difficult to obtain though.)

• AArch64: An exotic variant of AArch64 (arm64) has been added to our download portfolio: Big-endian AArch64. Enjoy!

• ♦ PA-RISC: Weekly stage builds for the hppa architecture (PA-RISC) are back, including systemd images for both hppa-1.1 and hppa-2.0 and an installation CD.

• MIPS: The weekly builds for MIPS are back as well! Here, we can now offer downloads for the o32, n32, and n64 ABI plus multilib stages - and all that for both endianness variants and init systems. No matter what your hardware is, you should find a starting point.

• Hardened: With more and more hardening becoming de-facto standard, the compiler settings in the hardened profiles have been tightened again to include additional experimental switches. In particular, in Gentoo Hardened, gcc and clang both now default to _FORTIFY_SOURCE=3, C++ standard library assertions, and enabled stack-clash-protection.

• ♦ Modern Java: A huge amount of work was done by our Java project to revive the language ecosystem and in particular recent Java versions in Gentoo. Additionally, OpenJDK 11 and OpenJDK 17 were bootstrapped for big-endian ppc64, as well as for x86, riscv, and arm64 with musl as C library, enabling the usage of modern Java on those configurations.

• GNU Emacs: Emacs ebuild-mode has seen a flurry of activity in 2022. New features include a new ebuild-repo-mode, inserting of user’s name and date stamp in package.mask and friends, support for pkgdev and pkgcheck commands, support for colors in ebuild command output, and a major refactoring of the code for keyword highlighting. Additionally, there’s flycheck-pkgcheck for on-the-fly linting and company-ebuild for automatic completion.

• ♦ Mathematics: The sci-mathematics category has grown with the addition of theorem provers such as lean, yices2, cadabra, or picosat. Further, the Coq Proof Assistant ecosystem support has been improved with new Coq versions, Emacs support via company-coq, and packages such as coq-mathcomp, coq-serapi, flocq, gappalib-coq …

• Alternatives: Many base system utilities exist in different flavours that are more or less drop-in replacements. One example of this is the compressor bzip2, with lbzip2 and pbzip2 as parallelizing alternatives; another tar, which exists both as gtar (GNU tar) and as bsdtar in libarchive. With alternatives we now have a clean system in place to use either of these options as default program via a symlinked binary.

• ♦ Racket: An ongoing project aims to bring first-class support for Racket, a modern dialect of Lisp and a descendant of Scheme, and the Racket language ecosystem to Gentoo.

• Python: In the meantime the default Python version in Gentoo has reached Python 3.10. Additionally we have also Python 3.11 available stable, which means we’re fully up to date with upstream. Gentoo testing provides the alpha releases of Python 3.12, so we can easily prepare for what comes next.

• Hardware: Our infrastructure team has set up two beefy new servers as Ganeti nodes hosted at OSUOSL, with 2x AMD EPYC 7543, 1TiB RAM, 22TiB NVME, and 25Gbit networking each. These will provide virtual machines for various services in the future. A new 1/10/25Gbit switch was also added to better support new and existing servers.

• ♦ Gitlab: We are now running an experimental self-hosted Gitlab instance, gitlab.gentoo.org. It will slowly take over and serve more and more git repositories.

• Pkgcore: Building on existing coding efforts, an official Gentoo PkgCore project was created to improve this set of QA and commit tools for Gentoo developers. Repoman was deprecated and removed from the Portage code base, and pkgcheck, part of PkgCore, has become the official QA tool for commits to the main Gentoo repository. It is also the code running our automated continuous integration system.

• ♦ Tattoo: The new tattoo arch testing system now manages and automates large parts of the architecture testing process. This has simplified and streamlined the stabilization process, shortening developer response times and “saving” arch stabilization.

• Devmanual: The Gentoo Development Manual has seen major improvements in 2022. More documentation is good!

• ♦ Income: The Gentoo Foundation took in approximately $16,500 in fiscal year 2022; the majority (over 90%) were individual cash donations from the community.

• Expenses: Our expenses in 2022 were, as split into the usual three categories, operating expenses (for services, fees, …) $11,000, capital expenses (for bought assets) $55,000 (servers, networking gear, SSDs, …), and depreciation expenses (value loss of existing assets) $9,500.

• Balance: We have about $97,000 in the bank as of July 1, 2022 (which is when our fiscal year 2022 ends for accounting purposes). The draft finanical report for 2022 is available on the Gentoo Wiki.

Our end of year review of course cannot cover everything that happened in Gentoo in 2022 in detail, and if you look closely you will find much more. We would like to thank all Gentoo developers and all who have submitted contributions for their relentless everyday Gentoo work. As a volunteer project, Gentoo could not exist without them.

And now let’s look forward to the new year 2023, with hopefully less unpleasant surprises than the last one!

♦

Finally, after a long break, it’s FOSDEM time again! Join us at Université Libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2023 will be held on February 4th and 5th.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building H, level 1! Visit this year’s wiki page to see who’s coming.

• Prototype
  • Sandbox

Recently while browsing the Alpine git repo I noticed they have a function called snapshot, see: git.alpinelinux.org/aports/tree/testing/dart/APKBUILD#n45 I am not 100% sure about how that works but a wild guess is that the developers can run that function to fetch the sources and maybe later upload them to the Alpine repo or some sort of (cloud?) storage.

In Portage there exists a pkg_config function used to run miscellaneous configuration for packages. The only major difference between src_snapshot and that would of course be that users would never run snapshot.

Probably only the network sandbox would have to be lifted out… to fetch the sources of course.

But also a few (at least one?) special directories and variables would be useful.

• Portage
  • Emerge
• Standard
• Use-Package

Configure the following for Portage.

dev-util/pkgcheck emacs

Emerge the following packages:

• app-emacs/company-ebuild
• dev-util/pkgcheck

Company-Ebuild should pull in app-emacs/ebuild-mode, if that does not happen, then report a bug ;-D

Add the following to your user's Emacs initialization file. The initialization file is either ~/.emacs.d/init.el or ~/.config/emacs/init.el for newer versions of GNU Emacs.

(require 'ebuild-mode)
(require 'company-ebuild)
(require 'flycheck)
(require 'flycheck-pkgcheck)

(add-hook 'ebuild-mode-hook 'company-ebuild-setup)
(add-hook 'ebuild-mode-hook 'flycheck-mode)
(add-hook 'ebuild-mode-hook 'flycheck-pkgcheck-setup)

We can also configure our environment using a use-package macro that simplifies the setup a little bit.

To use the below configuration the app-emacs/use-package package will have to be installed.

(require 'use-package)

(use-package ebuild-mode
  :defer t
  :mode "\\.\\(ebuild\\|eclass\\)\\'"
  :hook
  ((ebuild-mode . company-ebuild-setup)
   (ebuild-mode . flycheck-mode)
   (ebuild-mode . flycheck-pkgcheck-setup)))

The :defer t and :mode "..." enable deferred loading which theoretically speeds up GNU Emacs initialization time at the cost of running the whole use-package block of ebuild-mode configuration when the :mode condition is met.

• Fosstodon

If you prefer to follow my posts on Fosstodon via RSS, then here is the link: fosstodon.org/users/xgqt.rss.

I added the patch indicated above to gentoo-sources behind the ‘experimental’ flag, for now. This is to remove a 20 year old workaround for newer AMD processors. I expect this to propagate to lower kernels via upstream directly. This will be in gentoo-sources-5.19.12 (USE=experimental)

See:

www.phoronix.com/news/Linux-AMD-Old-Chipset-WA

www.phoronix.com/news/Linux-6.0-AMD-Chipset-WA

lkml.org/lkml/2022/9/21/74

12 weeks quickly slips away, and I’m proud to say that the packaging quality of ROCm in Gentoo does gets improved in this project.

Two sets of major deliverables are achieved: New ebuilds of ROCm-5.1.3 tool-chain that purely depends on vanilla llvm/clang, and rocm.eclass along with ROCm-5.1.3 libraries utilizing them. Each brings one great QA improvement compare to the original ROCm packaging method.

Beyond these, I also maintained rocprofiler, rocm-opencl-runtimes, bumping their version with nontrivial changes. I discovered several bugs, and talked to upstream. I also wrote ROCm wiki pages, which starts my journey on Gentoo wiki.

By writing rocm.eclass, I learnt pretty much about eclass writing — how to design, how to balance needs and QA concerns, how to write comments and examples well, etc. I’m really grateful to those Gentoo developers who pointed out my mistakes and helped me polishing my eclass.

Since I’m working on top of Gentoo repo, my work is scattered around rather than having my own repo. My major products can be seen in [0], where all my PRs to ::gentoo located. My weekly report can be found on Gentoo GSoC blogs

[0] My finished PRs for gentoo during GSoC 2022

Details are as followed:

Originally, ROCm has its own llvm fork, which has some modifications not upstreamed yet. In the original Gentoo ROCm packaging roadmap, sys-devel/llvm-roc is introduced as the ROCm forked llvm/clang. This is the simple way, and worked well on ROCm-only packages [1]. But it brings troubles if a large project like blender pulls in dependencies using vanilla llvm, and results in symbol collision [2].

So, when I noticed [1] in week 1, I began my journey on porting ROCm on vanilla clang. I’m very lucky, because at that time clang-14.0.5 was just released, eliminating major obstacles for porting (previous versions more or less have bugs). After some quick hack I succeeded, which is recorded in the week 1 report [3]. In that week I successfully built blender with hip cycles (GPU-accelerated render code written in HIP), and rendered some example projects on a Radeon RX 6700XT.

While I was thrilled in porting ROCm tool-chain upon vanilla clang, my mentor pointed out that I have carelessly brought some serious bugs in ::gentoo. In week 2, I managed to fix bugs I created, and set up a reproducible test ground using docker, to make test more easy and clean and avoid such bugs from happening again. Details can be found in week 2’s report [4].

After that there weren’t non-trivial progresses in porting to vanilla clang, only bug fixes and ebuild polishing, until I met MIOpen in the last week.

In week 12 rocm.eclass is almost in its final shape, so I began to land ROCm libraries [1] including sci-libs/miopen. ROCm libraries are usually written in “high level” languages like HIP, while dev-util/hip is already ported to use vanilla clang in good shape, so there is no need to worry compilation problems. However, MIOpen have various hand-written assemblies for JIT, which causes several test failures [5]. It was frustrating because I’m unfamiliar with AMDGPU assemblies, so I was close to gave up (my mentor also suggest to give up working on it in GSoC). Thus, I reported my problem to upstream in [5], attached with my debugging attempts.

Thanks to my testing system mentioned previously, I have setup not only standard environments, but also one snapshot with full llvm/clang debug symbols. I quickly located the problem and reported to upstream via issue, but I still didn’t know why the error is happening.

In the second day, I decided to look at the assembly and debugging result once again. This time fortune is on my side, and I discovered the key issue is LLVM treating Y and N in metadata as boolean values, not strings (they should be kernel parameter names) [6]. I provided a fix in [7], and all tests passed on both Radeon VII and Radeon RX 6700XT. Amazing! I have also mentioned how excited I was in week 12’s report [8].

[1] For example, ROCm libraries in github.com/ROCmSoftwarePlatform
[2] bugs.gentoo.org/693200
[3] Week 1 Report for Refining ROCm Packages in Gentoo
[4] Week 4 Report for Refining ROCm Packages in Gentoo
[5] github.com/ROCmSoftwarePlatform/MIOpen/issues/1731
[6] github.com/ROCmSoftwarePlatform/MIOpen/issues/1731#issuecomment-1236913096
[7] github.com/littlewu2508/gentoo/commit/40eb81f151f43eb5d833dc7440b02f12dab04b89
[8] Week 12 Report for Refining ROCm Packages in Gentoo

The most challenging part for me, is to write rocm.eclass. I started writing it in week 4 [9], and finished my design in week 8 [10] (including 10 days of temporary leave). In week 9-12, I posted 7 revisions of rocm.eclass in gentoo-dev mailing list [10,11], and received many helpful comments. Also, on Github PR [12], I also got lots of suggestions from Gentoo developers.

Eventually, I finished rocm.eclass, providing amdgpu_targets USE_EXPAND, ROCM_REQUIRED_USE, and ROCM_USE_DEP to control which gpu targets to compile, and coherency among dependencies. The eclass provides get_amdgpu_flags for src_configure and check_amdgpu for ensuring AMDGPU device accessibility in src_test. Finally, rocm.eclass is merged into ::gentoo in [13].

[9] Week 9 Report for Refining ROCm Packages in Gentoo
[10] archives.gentoo.org/gentoo-dev/threads/2022-08/
[11] archives.gentoo.org/gentoo-dev/threads/2022-09/
[12] github.com/gentoo/gentoo/pull/26784
[13] gitweb.gentoo.org/repo/gentoo.git/commit/?id=cf8a6a845b68b578772f2ae0d2703f203c6dec33

I have bumped dev-util/rocprofiler and its dependencies to version 5.1.3, and fixed proprietary aql profiler lib loading, so ROCm stack on Gentoo stays fully open-sourced without losing most profiling functionalities [14].

[14] github.com/ROCm-Developer-Tools/rocprofiler/issues/38

Due to limited time and long testing period, ebuilds of ROCm-5.1.3 libraries (ones using rocm.eclass) does not get merged. They can be found in this PR.
dev-libs/rocm-opencl-runtime is a critical package because it provides opencl, and many users still use opencl for GPGPU since HIP is a new stuff. I bumped it to 5.1.3 to match the vanilla clang tool-chain, and enabled its src_test, so users can make sure that vanilla clang isn’t breaking anything. The PR is located here.

Existing bug fixing is also a part of my GSoC. I have created various PRs and closed corresponding bugs on Gentoo Bugzilla: #822828, #853718, #851795, #851792, #852236, #850937, #836248, #836274, #866839. Also, many bug fixing happens before new packages enter the gentoo main repo, or they are found by myself in the first place, so there is no record on Bugzilla.

I have created 3 pages [15-17], filling important information about ROCm. I also received a lot of help from the Gentoo community, mainly focused on refining my wiki to meet the standards.

[15] wiki.gentoo.org/wiki/ROCm
[16] wiki.gentoo.org/wiki/HIP
[17] wiki.gentoo.org/wiki/Rocprofiler

The original plan in proposal also contained rocm.eclass. But it only allocated the last week for “investigation on vanilla clang”. In week 1, my mentor and I added “porting ROCm on vanilla clang” to the plan, and this became the new major deliverable. Due to the time limit, packaging high level frameworks like pytorch and tensorflow is abandoned. I only worked to get CuPy worked [18], showing rocm.eclass functionality on packages that depend on ROCm libraries.

I think the change of plan and deliverables better annotated the project title “Refining”, because what I did greatly improves the quality of existing ebuilds, rather than introducing more ebuilds.

[18] github.com/littlewu2508/gentoo/commit/3d142fa4b4ada560c053c2fd3c8c1501c82aace2

Although this is the final week, I would like to say that it is as exciting as the first week.

I kept polishing rocm.eclass with the help of Michał and my mentor, and it is now in good shape [1]. I must admit that the time to write an eclass for a beginner like me is much more than what I expected. In my proposal, I leave 4 weeks to finish it, 2-week implementation and 2-week polishing. In reality, I implemented within 2 weeks, but polished it for 4 weeks. I made a lot of QA issues and was not aware, which increases the number of review-modify cycles. During this process, I leant a lot:

1. Always re-read the eclass, especially comments and examples thoroughly after modification. Many times I forgot there is an example far from the change that should be updated because one functions changes its behavior.

2. Read the bash manual carefully, because properly usage of features like bash array can greatly simplify code.

3. Consider the maintenance difficulty of the eclass. I wrote a oddly specific `src_test`, which can cover all the cases of ROCm packages. But it’s not worth it, because specialized code should be placed into ebuilds, not one eclass. So instead, I remain the most common part, `check_amdgpu`, and get rid of phase functions, which made the eclass much cleaner.

I also find some bugs and their solutions. As I mentioned in week 10’s report, I observed many test failures in sci-libs/miopen based on vanilla clang. In this week, I figured out that they have 3 different reasons, and I’ve provided the two fixes for two failures ([2, 3]). The third issue, I’ve found it’s root cause [4]. I believe there would be a simple solution to this.

For gcc-12 issues, I also come to a brutal workaround [5]: undef the __noinline__ macro before including stdc++ headers and def it afterwards. I also observed that clang-15 does not fix this issue as expected, and provided a MWE at [6].

I’m also writing wiki pages, filling installation and developing guide.

In this 12-week project, I proposed to deliver rocm.eclass, and packages like pytorch, tensorflow with rocm enabled. Instead, I delivered rocm.eclass as proposed, but migrated the ROCm toolchain to vanilla clang. I thought porting ROCm toolchain to vanilla clang is closer to my project title “Refining ROCm Packages” ♦

[1] github.com/gentoo/gentoo/pull/26784
[2] github.com/littlewu2508/gentoo/commit/2bfae2e26a23d78b634a87ef4a0b3f0cc242dbc4
[3] github.com/littlewu2508/gentoo/commit/cd11b542aec825338ec396bce5c63bbced534e27
[4] github.com/ROCmSoftwarePlatform/MIOpen/issues/1731
[5] github.com/littlewu2508/gentoo/commit/2a49b4db336b075f2ac1fdfbc907f828105ea7e1
[6] github.com/llvm/llvm-project/issues/57544