January 13 2023

FOSDEM 2023

Gentoo News (GentooNews) January 13, 2023, 6:00

Finally, after a long break, it’s FOSDEM time again! Join us at Université Libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2023 will be held on February 4th and 5th.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building H, level 1! Visit this year’s wiki page to see who’s coming.

FOSDEM logo

Finally, after a long break, it’s FOSDEM time again! Join us at Université Libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2023 will be held on February 4th and 5th.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building H, level 1! Visit this year’s wiki page to see who’s coming.

December 30 2022

Src_Snapshot

Maciej Barć (xgqt) December 30, 2022, 2:03
Table of Contents
  • Prototype
    • Sandbox
Prototype

Recently while browsing the Alpine git repo I noticed they have a function called snapshot, see: git.alpinelinux.org/aports/tree/testing/dart/APKBUILD#n45 I am not 100% sure about how that works but a wild guess is that the developers can run that function to fetch the sources and maybe later upload them to the Alpine repo or some sort of (cloud?) storage.

In Portage there exists a pkg_config function used to run miscellaneous configuration for packages. The only major difference between src_snapshot and that would of course be that users would never run snapshot.

Sandbox

Probably only the network sandbox would have to be lifted out… to fetch the sources of course.

But also a few (at least one?) special directories and variables would be useful.

xgqt (xgqt ) December 30, 2022, 2:03

Prototype

Recently while browsing the Alpine git repo I noticed they have a function called snapshot, see: https://git.alpinelinux.org/aports/tree/testing/dart/APKBUILD#n45 I am not 100% sure about how that works but a wild guess is that the developers can run that function to fetch the sources and maybe later upload them to the Alpine repo or some sort of (cloud?) storage.

In Portage there exists a pkg_config function used to run miscellaneous configuration for packages. The only major difference between src_snapshot and that would of course be that users would never run snapshot.

Sandbox

Probably only the network sandbox would have to be lifted out… to fetch the sources of course.

But also a few (at least one?) special directories and variables would be useful.

Ebuild-Mode and Stuff

Maciej Barć (xgqt) December 30, 2022, 0:27
Table of Contents
  • Portage
    • Emerge
  • Standard
  • Use-Package
Portage

Configure the following for Portage.

dev-util/pkgcheck emacs
Emerge

Emerge the following packages:

  • app-emacs/company-ebuild
  • dev-util/pkgcheck

Company-Ebuild should pull in app-emacs/ebuild-mode, if that does not happen, then report a bug ;-D

Standard

Add the following to your user's Emacs initialization file. The initialization file is either ~/.emacs.d/init.el or ~/.config/emacs/init.el for newer versions of GNU Emacs.

(require 'ebuild-mode)
(require 'company-ebuild)
(require 'flycheck)
(require 'flycheck-pkgcheck)

(add-hook 'ebuild-mode-hook 'company-ebuild-setup)
(add-hook 'ebuild-mode-hook 'flycheck-mode)
(add-hook 'ebuild-mode-hook 'flycheck-pkgcheck-setup)
Use-Package

We can also configure our environment using a use-package macro that simplifies the setup a little bit.

To use the below configuration the app-emacs/use-package package will have to be installed.

(require 'use-package)

(use-package ebuild-mode
  :defer t
  :mode "\\.\\(ebuild\\|eclass\\)\\'"
  :hook
  ((ebuild-mode . company-ebuild-setup)
   (ebuild-mode . flycheck-mode)
   (ebuild-mode . flycheck-pkgcheck-setup)))

The :defer t and :mode "..." enable deferred loading which theoretically speeds up GNU Emacs initialization time at the cost of running the whole use-package block of ebuild-mode configuration when the :mode condition is met.

xgqt (xgqt ) December 30, 2022, 0:27

Portage

Configure the following for Portage.

dev-util/pkgcheck emacs

Emerge

Emerge the following packages:

  • app-emacs/company-ebuild
  • dev-util/pkgcheck

Company-Ebuild should pull in app-emacs/ebuild-mode, if that does not happen, then report a bug ;-D

Standard

Add the following to your user's Emacs initialization file. The initialization file is either ~/.emacs.d/init.el or ~/.config/emacs/init.el for newer versions of GNU Emacs.

(require 'ebuild-mode)
(require 'company-ebuild)
(require 'flycheck)
(require 'flycheck-pkgcheck)

(add-hook 'ebuild-mode-hook 'company-ebuild-setup)
(add-hook 'ebuild-mode-hook 'flycheck-mode)
(add-hook 'ebuild-mode-hook 'flycheck-pkgcheck-setup)

Use-Package

We can also configure our environment using a use-package macro that simplifies the setup a little bit.

To use the below configuration the app-emacs/use-package package will have to be installed.

(require 'use-package)

(use-package ebuild-mode
  :defer t
  :mode "\\.\\(ebuild\\|eclass\\)\\'"
  :hook
  ((ebuild-mode . company-ebuild-setup)
   (ebuild-mode . flycheck-mode)
   (ebuild-mode . flycheck-pkgcheck-setup)))

The :defer t and :mode "..." enable deferred loading which theoretically speeds up GNU Emacs initialization time at the cost of running the whole use-package block of ebuild-mode configuration when the :mode condition is met.

November 18 2022

Fediverse RSS

Maciej Barć (xgqt) November 18, 2022, 21:20
Table of Contents
  • Fosstodon
Fosstodon

If you prefer to follow my posts on Fosstodon via RSS, then here is the link: fosstodon.org/users/xgqt.rss.

xgqt (xgqt ) November 18, 2022, 21:20

Fosstodon

If you prefer to follow my posts on Fosstodon via RSS, then here is the link: fosstodon.org/users/xgqt.rss.

.tar sorting vs .xz compression ratio

Michał Górny (mgorny) November 18, 2022, 7:09

It is a pretty common knowledge that ordering of members within archive can affect the compression ratio. I’ve done some quick testing and the results somewhat surprised me. Firstly, it turned out that the simplest lexical sorting by name (path) gave the best result. Secondly, because it turned out that the difference between that and sorting by size was as large as 8%.

Note that this is a pretty specific source archive, so results may vary. Test details and commands in the remainder of the post.

Compression results per sort order Sort order Size in bytes Compared to best name 108 011 756 100.00% suffix 108 573 612 100.52% size (smallest first) 116 797 440 108.13% size (largest first) 116 645 940 108.00% suffix + size 111 709 128 103.42%

The conclusion? Sorting can affect compression ratio more than I have anticipated. However, all the “obvious” optimizations have made the result worse than plain lexical sorting. Perhaps it’s just the matter of well-organized source code keeping similar files in the same directories. Perhaps there is a way to optimize it even more (and beat sorting by name). One interesting option would be to group files by bucket sizes, and then sort by name.

Special thanks to Adrien Nader and Lasse Collin from #tukaani for inspiring me to do this.

Testing details

Testing was done on data from llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef.tar.gz snapshot. The archive was unpacked, then repacked using sorted file lists. Directory entries were not included in the archive.

Sorting by name was done using plain lexical sort, using the following command:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d | sort  | tar -cvvf /var/tmp/llvm.byname.tar -T -

Sorting by size involved numeric sort by file size, then by name:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d -printf '%s %p\n' | sort -n | cut -d' ' -f2- | tar -cf /var/tmp/llvm.bysize.tar -T -
find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d -printf '%s %p\n' | sort -k 1nr,2 | cut -d' ' -f2- | tar -cvvf /var/tmp/llvm.bysize.rev.tar -T -

Sorting by suffix involved extracting the final file suffix (if any) and sorting by it, then by full name:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d | sed -r -e 's:^:_ :' -e 's:.*\.([^/]*):\1&:' | sort | cut -d' ' -f2- | tar -cf /var/tmp/llvm.bysuffix.tar -T -

Sorting by suffix + size involved sorting by suffix first (so that similar files are grouped together), then by size, then by full path. Took some find(1) hackery:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d -printf '%s %p\n' | sed -r -e 's:^:_ :' -e 's:.*\.([^/]*):\1&:' | sort -n -k2 | sort -k 1,1 -s | cut -d' ' -f3- | tar -cf /var/tmp/llvm.bysuffix.size.tar -T -

It is a pretty common knowledge that ordering of members within archive can affect the compression ratio. I’ve done some quick testing and the results somewhat surprised me. Firstly, it turned out that the simplest lexical sorting by name (path) gave the best result. Secondly, because it turned out that the difference between that and sorting by size was as large as 8%.

Note that this is a pretty specific source archive, so results may vary. Test details and commands in the remainder of the post.

Compression results per sort order
Sort order Size in bytes Compared to best
name 108 011 756 100.00%
suffix 108 573 612 100.52%
size (smallest first) 116 797 440 108.13%
size (largest first) 116 645 940 108.00%
suffix + size 111 709 128 103.42%

The conclusion? Sorting can affect compression ratio more than I have anticipated. However, all the “obvious” optimizations have made the result worse than plain lexical sorting. Perhaps it’s just the matter of well-organized source code keeping similar files in the same directories. Perhaps there is a way to optimize it even more (and beat sorting by name). One interesting option would be to group files by bucket sizes, and then sort by name.

Special thanks to Adrien Nader and Lasse Collin from #tukaani for inspiring me to do this.

Testing details

Testing was done on data from llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef.tar.gz snapshot. The archive was unpacked, then repacked using sorted file lists. Directory entries were not included in the archive.

Sorting by name was done using plain lexical sort, using the following command:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d | sort  | tar -cvvf /var/tmp/llvm.byname.tar -T -

Sorting by size involved numeric sort by file size, then by name:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d -printf '%s %p\n' | sort -n | cut -d' ' -f2- | tar -cf /var/tmp/llvm.bysize.tar -T -
find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d -printf '%s %p\n' | sort -k 1nr,2 | cut -d' ' -f2- | tar -cvvf /var/tmp/llvm.bysize.rev.tar -T -

Sorting by suffix involved extracting the final file suffix (if any) and sorting by it, then by full name:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d | sed -r -e 's:^:_ :' -e 's:.*\.([^/]*):\1&:' | sort | cut -d' ' -f2- | tar -cf /var/tmp/llvm.bysuffix.tar -T -

Sorting by suffix + size involved sorting by suffix first (so that similar files are grouped together), then by size, then by full path. Took some find(1) hackery:

find llvm-project-f6f1fd443f48f417de9dfe23353055f1b20d87ef '!' -type d -printf '%s %p\n' | sed -r -e 's:^:_ :' -e 's:.*\.([^/]*):\1&:' | sort -n -k2 | sort -k 1,1 -s | cut -d' ' -f3- | tar -cf /var/tmp/llvm.bysuffix.size.tar -T -

October 07 2022

Clang in Gentoo now sets default runtimes via config file

Michał Górny (mgorny) October 07, 2022, 5:27

The upcoming clang 16 release features substantial improvements to configuration file support. Notably, it adds support for specifying multiple files and better default locations. This enabled Gentoo to finally replace the default-* flags used on sys-devel/clang, effectively empowering our users with the ability to change defaults without rebuilding whole clang.

This change has also been partially backported to clang 15.0.2 in Gentoo, and (unless major problems are reported) will be part of the stable clang 15.x release (currently planned for upcoming 15.0.3).

In this post, I’d like to shortly describe the new configuration file features, how much of them have been backported to 15.x in Gentoo and how defaults are going to be selected from now on.

Configuration file support in clang 16.x

Configuration files were supported at least for a few clang releases now but they weren’t very useful for us before. With clang 16, I have taken the opportunity to finally change that.

In clang 16, configuration files can be both specified explicitly and loaded from default locations. Default configuration files are loaded first (unless explicitly disabled by --no-default-config or a non-empty CLANG_NO_DEFAULT_CONFIG envvar — the latter intended to be used in clang’s test suites), and configuration files specified via --config= options are loaded afterwards. This permits explicit files to override the options specified in default configs. However, it should be noted that some values are appended rather than overrode, and there is no way to “reset” them right now.

As built in Gentoo, clang looks for configuration files in two locations: /etc/clang and the executable directory. Technically, the build system also permits specifying “user” configuration directory but it’s not practically useful, as it provides no way of referencing the user’s home directory. Effectively, we only use /etc/clang. The sys-devel/clang-common package installs a default set of configuration files there.

The default config lookup algorithm looks for <triple>-<driver>.cfg first. If this file is not found, it looks for separate <triple>.cfg and <driver>.cfg files and loads both. This enables the first location to be used as an override, without suffering from the appending problem. Triple is the effective target triple (i.e. accounting for options such as --target= and -m32) and driver is the string corresponding to driver mode, e.g. clang, clang++ or clang-cpp (but it does not account for -x c++ option!).

So for example, on a typical amd64 system clang will first try:

x86_64-pc-linux-gnu-clang.cfg

with fallback to loading both of:

x86_64-pc-linux-gnu.cfg
clang.cfg

If -m32 is used, this will be i386-pc-linux-gnu* instead. If clang++ is called, this will be *clang++.cfg, etc.

Explicit configuration files are specified using the --config=<file> option. They are loaded after the default configs, in order of being listed. They can either be specified by full path, or by bare filename. In the latter case, clang looks for them in the directories listed earlier.

Configuration files use response file syntax, i.e. you specify command-line options inside them as you would pass them on the command-line. They also support including additional files via @<filename> syntax. An example configuration file could specify:

-I/opt/mystuffs/include
-L/opt/mystuffs/lib
@gentoo-runtimes.cfg

Full documentation: Clang Compiler User’s Manual: configuration files.

The backport to clang 15.0.2

There are notable differences in configuration file support in clang 15.x:

  • the new --config=<file> spelling is not supported, you have to use --config <file> (yes, compilers don’t follow getopt_long rules)
  • only one configuration file can be loaded
  • --config disables loading default configuration files
  • there is no explicit --no-default-config option

The rules for loading configuration files were also different (and not very reliable). I have mostly backported the new lookup rules. However, since this version does not support loading multiple configuration files (and I did not want to diverge too far from vanilla), only <triple>-<driver>.cfg and <driver>.cfg names are supported (i.e. pure-triple variant is not).

I think this divergence is acceptable because Gentoo did not enable configuration file support before (i.e. did not specify the system configuration directory), so there is no reason to assume that any of regular Gentoo users would have relied on the prior logic.

Use of configuration files in Gentoo

We currently install two “base” configuration files: gentoo-gcc-install.cfg and gentoo-runtimes.cfg.

gentoo-gcc-install.cfg is used to provide the path to the GCC installation. Its initial contents are e.g.:

# This file is maintained by gcc-config.
# It is used to specify the selected GCC installation.
--gcc-install-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0

The intent is that gcc-config would update this file when switching between versions and clang would not have to include logic for reading from Gentoo-specific files anymore. However, gcc-config part is not yet implemented, and it is not clear if this solution will be workable long-term, as it probably breaks support for sysroots.

gentoo-runtimes.cfg is used to control the default runtimes and link editor used. Its initial contents are controlled by USE flags on the clang-common package and are e.g.:

# This file is initially generated by sys-devel/clang-runtime.
# It is used to control the default runtimes using by clang.

--rtlib=libgcc
--unwindlib=libgcc
--stdlib=libstdc++
-fuse-ld=bfd

On top of these two files, we install the actual configuration files for the three driver modes relevant to Gentoo compilations to use: clang.cfg, clang++.cfg and clang-cpp.cfg. All of them have the following contents:

# This configuration file is used by clang driver.
@gentoo-runtimes.cfg
@gentoo-gcc-install.cfg

Effectively, they defer into loading the two base files.

Future use of config files

Why now?, you might ask. After all, configuration files were there for a while now, and I haven’t taken the effort to fix them to be usable before. Well, it all started with the total mayhem caused by clang 15.x becoming more strict. While we managed to convince upstream to revert that change and defer it into 16.x, it became important for us to be easily able to test for the breakage involved in clang changing its behavior.

One part of this effort was starting to package clang 16 snapshots. If you’d like to help testing it, you can unmask them using the following package.accept_keywords snippet:

=dev-ml/llvm-ocaml-16.0.0_pre* **
=dev-python/lit-16.0.0_pre* **
=dev-util/lldb-16.0.0_pre* **
=dev-python/clang-python-16.0.0_pre* **
=sys-devel/clang-16.0.0_pre* **
=sys-devel/clang-common-16.0.0_pre* **
=sys-devel/clang-runtime-16.0.0_pre* **
=sys-devel/lld-16.0.0_pre* **
=sys-devel/llvm-16.0.0_pre* **
=sys-devel/llvm-common-16.0.0_pre* **
=sys-libs/compiler-rt-16.0.0_pre* **
=sys-libs/compiler-rt-sanitizers-16.0.0_pre* **
=sys-libs/libcxx-16.0.0_pre* **
=sys-libs/libcxxabi-16.0.0_pre* **
=sys-libs/libcxxrt-16.0.0_pre* **
=sys-libs/libomp-16.0.0_pre* **
=sys-libs/llvm-libunwind-16.0.0_pre* **
=dev-libs/libclc-16.0.0_pre* **

~sys-devel/llvmgold-16 **
~sys-devel/clang-toolchain-symlinks-16 **
~sys-devel/lld-toolchain-symlinks-16 **
~sys-devel/llvm-toolchain-symlinks-16 **

The other part is providing support for configuration files that can be used to easily pass -Werror= and -Wno-error= flags reliably to all clang invocations, and therefore adjust its behavior while testing.

With this, we should be able to assess the damage beforehand earlier. However, between the size of clang 16 tracker and upstream continuing to make the behavior more strict, I’m starting to have doubts whether clang will continue being a general-purpose that could be reliably used as a replacement for GCC, and not just a corporation-driven tool used to compile a few big projects.

The upcoming clang 16 release features substantial improvements to configuration file support. Notably, it adds support for specifying multiple files and better default locations. This enabled Gentoo to finally replace the default-* flags used on sys-devel/clang, effectively empowering our users with the ability to change defaults without rebuilding whole clang.

This change has also been partially backported to clang 15.0.2 in Gentoo, and (unless major problems are reported) will be part of the stable clang 15.x release (currently planned for upcoming 15.0.3).

In this post, I’d like to shortly describe the new configuration file features, how much of them have been backported to 15.x in Gentoo and how defaults are going to be selected from now on.

Configuration file support in clang 16.x

Configuration files were supported at least for a few clang releases now but they weren’t very useful for us before. With clang 16, I have taken the opportunity to finally change that.

In clang 16, configuration files can be both specified explicitly and loaded from default locations. Default configuration files are loaded first (unless explicitly disabled by --no-default-config or a non-empty CLANG_NO_DEFAULT_CONFIG envvar — the latter intended to be used in clang’s test suites), and configuration files specified via --config= options are loaded afterwards. This permits explicit files to override the options specified in default configs. However, it should be noted that some values are appended rather than overrode, and there is no way to “reset” them right now.

As built in Gentoo, clang looks for configuration files in two locations: /etc/clang and the executable directory. Technically, the build system also permits specifying “user” configuration directory but it’s not practically useful, as it provides no way of referencing the user’s home directory. Effectively, we only use /etc/clang. The sys-devel/clang-common package installs a default set of configuration files there.

The default config lookup algorithm looks for <triple>-<driver>.cfg first. If this file is not found, it looks for separate <triple>.cfg and <driver>.cfg files and loads both. This enables the first location to be used as an override, without suffering from the appending problem. Triple is the effective target triple (i.e. accounting for options such as --target= and -m32) and driver is the string corresponding to driver mode, e.g. clang, clang++ or clang-cpp (but it does not account for -x c++ option!).

So for example, on a typical amd64 system clang will first try:

x86_64-pc-linux-gnu-clang.cfg

with fallback to loading both of:

x86_64-pc-linux-gnu.cfg
clang.cfg

If -m32 is used, this will be i386-pc-linux-gnu* instead. If clang++ is called, this will be *clang++.cfg, etc.

Explicit configuration files are specified using the --config=<file> option. They are loaded after the default configs, in order of being listed. They can either be specified by full path, or by bare filename. In the latter case, clang looks for them in the directories listed earlier.

Configuration files use response file syntax, i.e. you specify command-line options inside them as you would pass them on the command-line. They also support including additional files via @<filename> syntax. An example configuration file could specify:

-I/opt/mystuffs/include
-L/opt/mystuffs/lib
@gentoo-runtimes.cfg

Full documentation: Clang Compiler User’s Manual: configuration files.

The backport to clang 15.0.2

There are notable differences in configuration file support in clang 15.x:

  • the new --config=<file> spelling is not supported, you have to use --config <file> (yes, compilers don’t follow getopt_long rules)
  • only one configuration file can be loaded
  • --config disables loading default configuration files
  • there is no explicit --no-default-config option

The rules for loading configuration files were also different (and not very reliable). I have mostly backported the new lookup rules. However, since this version does not support loading multiple configuration files (and I did not want to diverge too far from vanilla), only <triple>-<driver>.cfg and <driver>.cfg names are supported (i.e. pure-triple variant is not).

I think this divergence is acceptable because Gentoo did not enable configuration file support before (i.e. did not specify the system configuration directory), so there is no reason to assume that any of regular Gentoo users would have relied on the prior logic.

Use of configuration files in Gentoo

We currently install two “base” configuration files: gentoo-gcc-install.cfg and gentoo-runtimes.cfg.

gentoo-gcc-install.cfg is used to provide the path to the GCC installation. Its initial contents are e.g.:

# This file is maintained by gcc-config.
# It is used to specify the selected GCC installation.
--gcc-install-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0

The intent is that gcc-config would update this file when switching between versions and clang would not have to include logic for reading from Gentoo-specific files anymore. However, gcc-config part is not yet implemented, and it is not clear if this solution will be workable long-term, as it probably breaks support for sysroots.

gentoo-runtimes.cfg is used to control the default runtimes and link editor used. Its initial contents are controlled by USE flags on the clang-common package and are e.g.:

# This file is initially generated by sys-devel/clang-runtime.
# It is used to control the default runtimes using by clang.

--rtlib=libgcc
--unwindlib=libgcc
--stdlib=libstdc++
-fuse-ld=bfd

On top of these two files, we install the actual configuration files for the three driver modes relevant to Gentoo compilations to use: clang.cfg, clang++.cfg and clang-cpp.cfg. All of them have the following contents:

# This configuration file is used by clang driver.
@gentoo-runtimes.cfg
@gentoo-gcc-install.cfg

Effectively, they defer into loading the two base files.

Future use of config files

Why now?, you might ask. After all, configuration files were there for a while now, and I haven’t taken the effort to fix them to be usable before. Well, it all started with the total mayhem caused by clang 15.x becoming more strict. While we managed to convince upstream to revert that change and defer it into 16.x, it became important for us to be easily able to test for the breakage involved in clang changing its behavior.

One part of this effort was starting to package clang 16 snapshots. If you’d like to help testing it, you can unmask them using the following package.accept_keywords snippet:

=dev-ml/llvm-ocaml-16.0.0_pre* **
=dev-python/lit-16.0.0_pre* **
=dev-util/lldb-16.0.0_pre* **
=dev-python/clang-python-16.0.0_pre* **
=sys-devel/clang-16.0.0_pre* **
=sys-devel/clang-common-16.0.0_pre* **
=sys-devel/clang-runtime-16.0.0_pre* **
=sys-devel/lld-16.0.0_pre* **
=sys-devel/llvm-16.0.0_pre* **
=sys-devel/llvm-common-16.0.0_pre* **
=sys-libs/compiler-rt-16.0.0_pre* **
=sys-libs/compiler-rt-sanitizers-16.0.0_pre* **
=sys-libs/libcxx-16.0.0_pre* **
=sys-libs/libcxxabi-16.0.0_pre* **
=sys-libs/libcxxrt-16.0.0_pre* **
=sys-libs/libomp-16.0.0_pre* **
=sys-libs/llvm-libunwind-16.0.0_pre* **
=dev-libs/libclc-16.0.0_pre* **

~sys-devel/llvmgold-16 **
~sys-devel/clang-toolchain-symlinks-16 **
~sys-devel/lld-toolchain-symlinks-16 **
~sys-devel/llvm-toolchain-symlinks-16 **

The other part is providing support for configuration files that can be used to easily pass -Werror= and -Wno-error= flags reliably to all clang invocations, and therefore adjust its behavior while testing.

With this, we should be able to assess the damage beforehand earlier. However, between the size of clang 16 tracker and upstream continuing to make the behavior more strict, I’m starting to have doubts whether clang will continue being a general-purpose that could be reliably used as a replacement for GCC, and not just a corporation-driven tool used to compile a few big projects.

September 12 2022

Refining ROCm Packages in Gentoo — project summary

Gentoo Google Summer of Code (GSoC) September 12, 2022, 13:07

12 weeks quickly slips away, and I’m proud to say that the packaging quality of ROCm in Gentoo does gets improved in this project.

Two sets of major deliverables are achieved: New ebuilds of ROCm-5.1.3 tool-chain that purely depends on vanilla llvm/clang, and rocm.eclass along with ROCm-5.1.3 libraries utilizing them. Each brings one great QA improvement compare to the original ROCm packaging method.

Beyond these, I also maintained rocprofiler, rocm-opencl-runtimes, bumping their version with nontrivial changes. I discovered several bugs, and talked to upstream. I also wrote ROCm wiki pages, which starts my journey on Gentoo wiki.

By writing rocm.eclass, I learnt pretty much about eclass writing — how to design, how to balance needs and QA concerns, how to write comments and examples well, etc. I’m really grateful to those Gentoo developers who pointed out my mistakes and helped me polishing my eclass.

Since I’m working on top of Gentoo repo, my work is scattered around rather than having my own repo. My major products can be seen in [0], where all my PRs to ::gentoo located. My weekly report can be found on Gentoo GSoC blogs

[0] My finished PRs for gentoo during GSoC 2022

Details are as followed:

First, it’s about ROCm on vanilla llvm/clang

Originally, ROCm has its own llvm fork, which has some modifications not upstreamed yet. In the original Gentoo ROCm packaging roadmap, sys-devel/llvm-roc is introduced as the ROCm forked llvm/clang. This is the simple way, and worked well on ROCm-only packages [1]. But it brings troubles if a large project like blender pulls in dependencies using vanilla llvm, and results in symbol collision [2].

So, when I noticed [1] in week 1, I began my journey on porting ROCm on vanilla clang. I’m very lucky, because at that time clang-14.0.5 was just released, eliminating major obstacles for porting (previous versions more or less have bugs). After some quick hack I succeeded, which is recorded in the week 1 report [3]. In that week I successfully built blender with hip cycles (GPU-accelerated render code written in HIP), and rendered some example projects on a Radeon RX 6700XT.

While I was thrilled in porting ROCm tool-chain upon vanilla clang, my mentor pointed out that I have carelessly brought some serious bugs in ::gentoo. In week 2, I managed to fix bugs I created, and set up a reproducible test ground using docker, to make test more easy and clean and avoid such bugs from happening again. Details can be found in week 2’s report [4].

After that there weren’t non-trivial progresses in porting to vanilla clang, only bug fixes and ebuild polishing, until I met MIOpen in the last week.

The story of debugging MIOpen assemblies

In week 12 rocm.eclass is almost in its final shape, so I began to land ROCm libraries [1] including sci-libs/miopen. ROCm libraries are usually written in “high level” languages like HIP, while dev-util/hip is already ported to use vanilla clang in good shape, so there is no need to worry compilation problems. However, MIOpen have various hand-written assemblies for JIT, which causes several test failures [5]. It was frustrating because I’m unfamiliar with AMDGPU assemblies, so I was close to gave up (my mentor also suggest to give up working on it in GSoC). Thus, I reported my problem to upstream in [5], attached with my debugging attempts.

Thanks to my testing system mentioned previously, I have setup not only standard environments, but also one snapshot with full llvm/clang debug symbols. I quickly located the problem and reported to upstream via issue, but I still didn’t know why the error is happening.

In the second day, I decided to look at the assembly and debugging result once again. This time fortune is on my side, and I discovered the key issue is LLVM treating Y and N in metadata as boolean values, not strings (they should be kernel parameter names) [6]. I provided a fix in [7], and all tests passed on both Radeon VII and Radeon RX 6700XT. Amazing! I have also mentioned how excited I was in week 12’s report [8].

[1] For example, ROCm libraries in github.com/ROCmSoftwarePlatform
[2] bugs.gentoo.org/693200
[3] Week 1 Report for Refining ROCm Packages in Gentoo
[4] Week 4 Report for Refining ROCm Packages in Gentoo
[5] github.com/ROCmSoftwarePlatform/MIOpen/issues/1731
[6] github.com/ROCmSoftwarePlatform/MIOpen/issues/1731#issuecomment-1236913096
[7] github.com/littlewu2508/gentoo/commit/40eb81f151f43eb5d833dc7440b02f12dab04b89
[8] Week 12 Report for Refining ROCm Packages in Gentoo

The second deliverable is rocm.eclass

The most challenging part for me, is to write rocm.eclass. I started writing it in week 4 [9], and finished my design in week 8 [10] (including 10 days of temporary leave). In week 9-12, I posted 7 revisions of rocm.eclass in gentoo-dev mailing list [10,11], and received many helpful comments. Also, on Github PR [12], I also got lots of suggestions from Gentoo developers.

Eventually, I finished rocm.eclass, providing amdgpu_targets USE_EXPAND, ROCM_REQUIRED_USE, and ROCM_USE_DEP to control which gpu targets to compile, and coherency among dependencies. The eclass provides get_amdgpu_flags for src_configure and check_amdgpu for ensuring AMDGPU device accessibility in src_test. Finally, rocm.eclass is merged into ::gentoo in [13].

[9] Week 9 Report for Refining ROCm Packages in Gentoo
[10] archives.gentoo.org/gentoo-dev/threads/2022-08/
[11] archives.gentoo.org/gentoo-dev/threads/2022-09/
[12] github.com/gentoo/gentoo/pull/26784
[13] gitweb.gentoo.org/repo/gentoo.git/commit/?id=cf8a6a845b68b578772f2ae0d2703f203c6dec33

Other coding products Merged ebuilds rocprofiler

I have bumped dev-util/rocprofiler and its dependencies to version 5.1.3, and fixed proprietary aql profiler lib loading, so ROCm stack on Gentoo stays fully open-sourced without losing most profiling functionalities [14].

[14] github.com/ROCm-Developer-Tools/rocprofiler/issues/38

Unmerged ebuilds

Due to limited time and long testing period, ebuilds of ROCm-5.1.3 libraries (ones using rocm.eclass) does not get merged. They can be found in this PR.
dev-libs/rocm-opencl-runtime is a critical package because it provides opencl, and many users still use opencl for GPGPU since HIP is a new stuff. I bumped it to 5.1.3 to match the vanilla clang tool-chain, and enabled its src_test, so users can make sure that vanilla clang isn’t breaking anything. The PR is located here.

Bug fixes

Existing bug fixing is also a part of my GSoC. I have created various PRs and closed corresponding bugs on Gentoo Bugzilla: #822828, #853718, #851795, #851792, #852236, #850937, #836248, #836274, #866839. Also, many bug fixing happens before new packages enter the gentoo main repo, or they are found by myself in the first place, so there is no record on Bugzilla.

Last but not least, the wiki page

I have created 3 pages [15-17], filling important information about ROCm. I also received a lot of help from the Gentoo community, mainly focused on refining my wiki to meet the standards.

[15] wiki.gentoo.org/wiki/ROCm
[16] wiki.gentoo.org/wiki/HIP
[17] wiki.gentoo.org/wiki/Rocprofiler

Comparison with original plan

The original plan in proposal also contained rocm.eclass. But it only allocated the last week for “investigation on vanilla clang”. In week 1, my mentor and I added “porting ROCm on vanilla clang” to the plan, and this became the new major deliverable. Due to the time limit, packaging high level frameworks like pytorch and tensorflow is abandoned. I only worked to get CuPy worked [18], showing rocm.eclass functionality on packages that depend on ROCm libraries.

I think the change of plan and deliverables better annotated the project title “Refining”, because what I did greatly improves the quality of existing ebuilds, rather than introducing more ebuilds.

[18] github.com/littlewu2508/gentoo/commit/3d142fa4b4ada560c053c2fd3c8c1501c82aace2

12 weeks quickly slips away, and I’m proud to say that the packaging quality of ROCm in Gentoo does gets improved in this project.

Two sets of major deliverables are achieved: New ebuilds of ROCm-5.1.3 tool-chain that purely depends on vanilla llvm/clang, and rocm.eclass along with ROCm-5.1.3 libraries utilizing them. Each brings one great QA improvement compare to the original ROCm packaging method.

Beyond these, I also maintained rocprofiler, rocm-opencl-runtimes, bumping their version with nontrivial changes. I discovered several bugs, and talked to upstream. I also wrote ROCm wiki pages, which starts my journey on Gentoo wiki.

By writing rocm.eclass, I learnt pretty much about eclass writing — how to design, how to balance needs and QA concerns, how to write comments and examples well, etc. I’m really grateful to those Gentoo developers who pointed out my mistakes and helped me polishing my eclass.

Since I’m working on top of Gentoo repo, my work is scattered around rather than having my own repo. My major products can be seen in [0], where all my PRs to ::gentoo located. My weekly report can be found on Gentoo GSoC blogs

[0] My finished PRs for gentoo during GSoC 2022

Details are as followed:

First, it’s about ROCm on vanilla llvm/clang

Originally, ROCm has its own llvm fork, which has some modifications not upstreamed yet. In the original Gentoo ROCm packaging roadmap, sys-devel/llvm-roc is introduced as the ROCm forked llvm/clang. This is the simple way, and worked well on ROCm-only packages [1]. But it brings troubles if a large project like blender pulls in dependencies using vanilla llvm, and results in symbol collision [2].

So, when I noticed [1] in week 1, I began my journey on porting ROCm on vanilla clang. I’m very lucky, because at that time clang-14.0.5 was just released, eliminating major obstacles for porting (previous versions more or less have bugs). After some quick hack I succeeded, which is recorded in the week 1 report [3]. In that week I successfully built blender with hip cycles (GPU-accelerated render code written in HIP), and rendered some example projects on a Radeon RX 6700XT.

While I was thrilled in porting ROCm tool-chain upon vanilla clang, my mentor pointed out that I have carelessly brought some serious bugs in ::gentoo. In week 2, I managed to fix bugs I created, and set up a reproducible test ground using docker, to make test more easy and clean and avoid such bugs from happening again. Details can be found in week 2’s report [4].

After that there weren’t non-trivial progresses in porting to vanilla clang, only bug fixes and ebuild polishing, until I met MIOpen in the last week.

The story of debugging MIOpen assemblies

In week 12 rocm.eclass is almost in its final shape, so I began to land ROCm libraries [1] including sci-libs/miopen. ROCm libraries are usually written in “high level” languages like HIP, while dev-util/hip is already ported to use vanilla clang in good shape, so there is no need to worry compilation problems. However, MIOpen have various hand-written assemblies for JIT, which causes several test failures [5]. It was frustrating because I’m unfamiliar with AMDGPU assemblies, so I was close to gave up (my mentor also suggest to give up working on it in GSoC). Thus, I reported my problem to upstream in [5], attached with my debugging attempts.

Thanks to my testing system mentioned previously, I have setup not only standard environments, but also one snapshot with full llvm/clang debug symbols. I quickly located the problem and reported to upstream via issue, but I still didn’t know why the error is happening.

In the second day, I decided to look at the assembly and debugging result once again. This time fortune is on my side, and I discovered the key issue is LLVM treating Y and N in metadata as boolean values, not strings (they should be kernel parameter names) [6]. I provided a fix in [7], and all tests passed on both Radeon VII and Radeon RX 6700XT. Amazing! I have also mentioned how excited I was in week 12’s report [8].

[1] For example, ROCm libraries in https://github.com/ROCmSoftwarePlatform
[2] https://bugs.gentoo.org/693200
[3] Week 1 Report for Refining ROCm Packages in Gentoo
[4] Week 4 Report for Refining ROCm Packages in Gentoo
[5] https://github.com/ROCmSoftwarePlatform/MIOpen/issues/1731
[6] https://github.com/ROCmSoftwarePlatform/MIOpen/issues/1731#issuecomment-1236913096
[7] https://github.com/littlewu2508/gentoo/commit/40eb81f151f43eb5d833dc7440b02f12dab04b89
[8] Week 12 Report for Refining ROCm Packages in Gentoo

The second deliverable is rocm.eclass

The most challenging part for me, is to write rocm.eclass. I started writing it in week 4 [9], and finished my design in week 8 [10] (including 10 days of temporary leave). In week 9-12, I posted 7 revisions of rocm.eclass in gentoo-dev mailing list [10,11], and received many helpful comments. Also, on Github PR [12], I also got lots of suggestions from Gentoo developers.

Eventually, I finished rocm.eclass, providing amdgpu_targets USE_EXPAND, ROCM_REQUIRED_USE, and ROCM_USE_DEP to control which gpu targets to compile, and coherency among dependencies. The eclass provides get_amdgpu_flags for src_configure and check_amdgpu for ensuring AMDGPU device accessibility in src_test. Finally, rocm.eclass is merged into ::gentoo in [13].

[9] Week 9 Report for Refining ROCm Packages in Gentoo
[10] https://archives.gentoo.org/gentoo-dev/threads/2022-08/
[11] https://archives.gentoo.org/gentoo-dev/threads/2022-09/
[12] https://github.com/gentoo/gentoo/pull/26784
[13] https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=cf8a6a845b68b578772f2ae0d2703f203c6dec33

Other coding products

Merged ebuilds

rocprofiler

I have bumped dev-util/rocprofiler and its dependencies to version 5.1.3, and fixed proprietary aql profiler lib loading, so ROCm stack on Gentoo stays fully open-sourced without losing most profiling functionalities [14].

[14] https://github.com/ROCm-Developer-Tools/rocprofiler/issues/38

Unmerged ebuilds

Due to limited time and long testing period, ebuilds of ROCm-5.1.3 libraries (ones using rocm.eclass) does not get merged. They can be found in this PR.
dev-libs/rocm-opencl-runtime is a critical package because it provides opencl, and many users still use opencl for GPGPU since HIP is a new stuff. I bumped it to 5.1.3 to match the vanilla clang tool-chain, and enabled its src_test, so users can make sure that vanilla clang isn’t breaking anything. The PR is located here.

Bug fixes

Existing bug fixing is also a part of my GSoC. I have created various PRs and closed corresponding bugs on Gentoo Bugzilla: #822828, #853718, #851795, #851792, #852236, #850937, #836248, #836274, #866839. Also, many bug fixing happens before new packages enter the gentoo main repo, or they are found by myself in the first place, so there is no record on Bugzilla.

Last but not least, the wiki page

I have created 3 pages [15-17], filling important information about ROCm. I also received a lot of help from the Gentoo community, mainly focused on refining my wiki to meet the standards.

[15] https://wiki.gentoo.org/wiki/ROCm
[16] https://wiki.gentoo.org/wiki/HIP
[17] https://wiki.gentoo.org/wiki/Rocprofiler

Comparison with original plan

The original plan in proposal also contained rocm.eclass. But it only allocated the last week for “investigation on vanilla clang”. In week 1, my mentor and I added “porting ROCm on vanilla clang” to the plan, and this became the new major deliverable. Due to the time limit, packaging high level frameworks like pytorch and tensorflow is abandoned. I only worked to get CuPy worked [18], showing rocm.eclass functionality on packages that depend on ROCm libraries.

I think the change of plan and deliverables better annotated the project title “Refining”, because what I did greatly improves the quality of existing ebuilds, rather than introducing more ebuilds.

[18] https://github.com/littlewu2508/gentoo/commit/3d142fa4b4ada560c053c2fd3c8c1501c82aace2

September 11 2022

Week 12 Report for Refining ROCm Packages in Gentoo

Gentoo Google Summer of Code (GSoC) September 11, 2022, 10:16

Although this is the final week, I would like to say that it is as exciting as the first week.

I kept polishing rocm.eclass with the help of Michał and my mentor, and it is now in good shape [1]. I must admit that the time to write an eclass for a beginner like me is much more than what I expected. In my proposal, I leave 4 weeks to finish it, 2-week implementation and 2-week polishing. In reality, I implemented within 2 weeks, but polished it for 4 weeks. I made a lot of QA issues and was not aware, which increases the number of review-modify cycles. During this process, I leant a lot:

1. Always re-read the eclass, especially comments and examples thoroughly after modification. Many times I forgot there is an example far from the change that should be updated because one functions changes its behavior.

2. Read the bash manual carefully, because properly usage of features like bash array can greatly simplify code.

3. Consider the maintenance difficulty of the eclass. I wrote a oddly specific `src_test`, which can cover all the cases of ROCm packages. But it’s not worth it, because specialized code should be placed into ebuilds, not one eclass. So instead, I remain the most common part, `check_amdgpu`, and get rid of phase functions, which made the eclass much cleaner.

I also find some bugs and their solutions. As I mentioned in week 10’s report, I observed many test failures in sci-libs/miopen based on vanilla clang. In this week, I figured out that they have 3 different reasons, and I’ve provided the two fixes for two failures ([2, 3]). The third issue, I’ve found it’s root cause [4]. I believe there would be a simple solution to this.

For gcc-12 issues, I also come to a brutal workaround [5]: undef the __noinline__ macro before including stdc++ headers and def it afterwards. I also observed that clang-15 does not fix this issue as expected, and provided a MWE at [6].

I’m also writing wiki pages, filling installation and developing guide.

In this 12-week project, I proposed to deliver rocm.eclass, and packages like pytorch, tensorflow with rocm enabled. Instead, I delivered rocm.eclass as proposed, but migrated the ROCm toolchain to vanilla clang. I thought porting ROCm toolchain to vanilla clang is closer to my project title “Refining ROCm Packages” ♦

[1] github.com/gentoo/gentoo/pull/26784
[2] github.com/littlewu2508/gentoo/commit/2bfae2e26a23d78b634a87ef4a0b3f0cc242dbc4
[3] github.com/littlewu2508/gentoo/commit/cd11b542aec825338ec396bce5c63bbced534e27
[4] github.com/ROCmSoftwarePlatform/MIOpen/issues/1731
[5] github.com/littlewu2508/gentoo/commit/2a49b4db336b075f2ac1fdfbc907f828105ea7e1
[6] github.com/llvm/llvm-project/issues/57544

Although this is the final week, I would like to say that it is as exciting as the first week.

I kept polishing rocm.eclass with the help of Michał and my mentor, and it is now in good shape [1]. I must admit that the time to write an eclass for a beginner like me is much more than what I expected. In my proposal, I leave 4 weeks to finish it, 2-week implementation and 2-week polishing. In reality, I implemented within 2 weeks, but polished it for 4 weeks. I made a lot of QA issues and was not aware, which increases the number of review-modify cycles. During this process, I leant a lot:

1. Always re-read the eclass, especially comments and examples thoroughly after modification. Many times I forgot there is an example far from the change that should be updated because one functions changes its behavior.

2. Read the bash manual carefully, because properly usage of features like bash array can greatly simplify code.

3. Consider the maintenance difficulty of the eclass. I wrote a oddly specific `src_test`, which can cover all the cases of ROCm packages. But it’s not worth it, because specialized code should be placed into ebuilds, not one eclass. So instead, I remain the most common part, `check_amdgpu`, and get rid of phase functions, which made the eclass much cleaner.

I also find some bugs and their solutions. As I mentioned in week 10’s report, I observed many test failures in sci-libs/miopen based on vanilla clang. In this week, I figured out that they have 3 different reasons, and I’ve provided the two fixes for two failures ([2, 3]). The third issue, I’ve found it’s root cause [4]. I believe there would be a simple solution to this.

For gcc-12 issues, I also come to a brutal workaround [5]: undef the __noinline__ macro before including stdc++ headers and def it afterwards. I also observed that clang-15 does not fix this issue as expected, and provided a MWE at [6].

I’m also writing wiki pages, filling installation and developing guide.

In this 12-week project, I proposed to deliver rocm.eclass, and packages like pytorch, tensorflow with rocm enabled. Instead, I delivered rocm.eclass as proposed, but migrated the ROCm toolchain to vanilla clang. I thought porting ROCm toolchain to vanilla clang is closer to my project title “Refining ROCm Packages” 🙂

[1] https://github.com/gentoo/gentoo/pull/26784
[2] https://github.com/littlewu2508/gentoo/commit/2bfae2e26a23d78b634a87ef4a0b3f0cc242dbc4
[3] https://github.com/littlewu2508/gentoo/commit/cd11b542aec825338ec396bce5c63bbced534e27
[4] https://github.com/ROCmSoftwarePlatform/MIOpen/issues/1731
[5] https://github.com/littlewu2508/gentoo/commit/2a49b4db336b075f2ac1fdfbc907f828105ea7e1
[6] https://github.com/llvm/llvm-project/issues/57544

Week 11 Report for Refining ROCm Packages in Gentoo

Gentoo Google Summer of Code (GSoC) September 11, 2022, 10:14

My progress this week is mainly writing wiki and refining rocm.eclass.

Although the current eclass can work with my new ebuilds [1], Michał Górny has pointed out various flaws on the Github PR [2]. He also pointed out the necessity about rocm.eclass, because it seems like a combination of two eclasses. In my opinion, rocm.eclass has its value, mainly for handling USE_EXPANDS and common phase functions. The ugly part is mainly in rocm_src_test: due to the inconsistency of test methods of packages in [3], I have to detect which method is using and do it accordingly. So my plan is to split the one-size-fits-all rocm_src_test into two functions, corresponding to two scenarios (cmake test or standalone binary), and let each ebuild decide which to use. This can avoid detailed detection code that make rocm_src_test bloated.

Wiki writing: I think the main part of ROCm wiki[1] and HIP[2] is nearly finished. But due to the delay of rocm.eclass, the related information is not appended (ROCm#Developing guide). There is also a section a reserved: ROCm#Installation guide. I have little clue on how to write this part, because ROCm is a wide collection of packages. Maybe a meta package (there are users working on this) would be helpful.

To be honest I’m a bit anxious, because there is only one week left, but there are still a lot to be determined and tested on rocm.eclass along with the sci-libs/roc* ebuilds. I hope I can resolve these core issues in the last week.

[1] github.com/littlewu2508/gentoo/tree/rocm-5.1.3-scilibs
[2] github.com/gentoo/gentoo/pull/26784
[3] github.com/ROCmSoftwarePlatform
[4] wiki.gentoo.org/wiki/ROCm
[5] wiki.gentoo.org/wiki/HIP

My progress this week is mainly writing wiki and refining rocm.eclass.

Although the current eclass can work with my new ebuilds [1], Michał Górny has pointed out various flaws on the Github PR [2]. He also pointed out the necessity about rocm.eclass, because it seems like a combination of two eclasses. In my opinion, rocm.eclass has its value, mainly for handling USE_EXPANDS and common phase functions. The ugly part is mainly in rocm_src_test: due to the inconsistency of test methods of packages in [3], I have to detect which method is using and do it accordingly. So my plan is to split the one-size-fits-all rocm_src_test into two functions, corresponding to two scenarios (cmake test or standalone binary), and let each ebuild decide which to use. This can avoid detailed detection code that make rocm_src_test bloated.

Wiki writing: I think the main part of ROCm wiki[1] and HIP[2] is nearly finished. But due to the delay of rocm.eclass, the related information is not appended (ROCm#Developing guide). There is also a section a reserved: ROCm#Installation guide. I have little clue on how to write this part, because ROCm is a wide collection of packages. Maybe a meta package (there are users working on this) would be helpful.

To be honest I’m a bit anxious, because there is only one week left, but there are still a lot to be determined and tested on rocm.eclass along with the sci-libs/roc* ebuilds. I hope I can resolve these core issues in the last week.

[1] https://github.com/littlewu2508/gentoo/tree/rocm-5.1.3-scilibs
[2] https://github.com/gentoo/gentoo/pull/26784
[3] https://github.com/ROCmSoftwarePlatform
[4] https://wiki.gentoo.org/wiki/ROCm
[5] https://wiki.gentoo.org/wiki/HIP

Week 10 Report for Refining ROCm Packages in Gentoo

Gentoo Google Summer of Code (GSoC) September 11, 2022, 10:10

This week I have leant a lot from Ulrich’s comments on rocm.eclass. I polished the eclass to v3 and send to gentoo-dev mailing list. However, I observed another error introduced in v3, and I’ll include a fix for it in the v4 in the following days.

Another half of my time is spent on testing sci-libs/roc-* packages on various platforms, utilizing rocm.eclass. I can say that rocm.eclass did its job as expected, so I believe after v4 it can be merged.

With src_test enabled, I have found various test failures. rocBLAS-5.1.3 fails 3 tests on Radeon RX 6700XT, slightly exceeding tolerance, which seems not a big issue; rocFFT-5.1.3 fails 16 suites on Radeon VII [1], which is serious and confirmed by upstream, so I suggest masking <code>amdgpu_targets_gfx906</code> USE flag for rocFFT-5.1.3; just today I observe MIOpen is failing many tests, probably due to vanilla clang. I’ll open issues and report those test failures to upstream. Running tests suite takes a lot of time, and often drain the GPU. It may takes more than 15 hours testing rocBLAS, even on performant CPU like Ryzen 5950X. If I use the GPU to render graphics (run a desktop environment) and do test simultaneously, it often result in amdgpu driver failure. I hope one day we can have a testing farm for ROCm packages, but that would be expensive because there are a lot of GPU architectures, and the compilation takes a lot of time.

I planned to finish the draft of wiki pages [2,3], but turns out I’m running out of time. I’ll catch up in week 11. My mentor is also busy in week 10, so my PR about rocm-opencl-runtime is still pending for review. Now we are working on solving the dependency issue of ROCm packages — gcc-12 and gcc-11.3.0 incompatibilities. Due to two bugs, the current stable gcc, gcc-11.3.0 cannot compile some ROCm packages [4], and the current unstable gcc, gcc-12, is unable to compile nearly all ROCm packages [5].

I’ll continue to do what’s postponed in week 10 — landing rocm.eclass and sci-libs packages, preparing cupy, fixing bugs, and writing the wiki pages. I’ll investigate MIOpen’s situation as well.

[1] github.com/ROCmSoftwarePlatform/rocFFT/issues/369
[2] wiki.gentoo.org/wiki/ROCm
[3] wiki.gentoo.org/wiki/HIP
[4] bugs.gentoo.org/842405
[5] bugs.gentoo.org/857660

This week I have leant a lot from Ulrich’s comments on rocm.eclass. I polished the eclass to v3 and send to gentoo-dev mailing list. However, I observed another error introduced in v3, and I’ll include a fix for it in the v4 in the following days.

Another half of my time is spent on testing sci-libs/roc-* packages on various platforms, utilizing rocm.eclass. I can say that rocm.eclass did its job as expected, so I believe after v4 it can be merged.

With src_test enabled, I have found various test failures. rocBLAS-5.1.3 fails 3 tests on Radeon RX 6700XT, slightly exceeding tolerance, which seems not a big issue; rocFFT-5.1.3 fails 16 suites on Radeon VII [1], which is serious and confirmed by upstream, so I suggest masking <code>amdgpu_targets_gfx906</code> USE flag for rocFFT-5.1.3; just today I observe MIOpen is failing many tests, probably due to vanilla clang. I’ll open issues and report those test failures to upstream. Running tests suite takes a lot of time, and often drain the GPU. It may takes more than 15 hours testing rocBLAS, even on performant CPU like Ryzen 5950X. If I use the GPU to render graphics (run a desktop environment) and do test simultaneously, it often result in amdgpu driver failure. I hope one day we can have a testing farm for ROCm packages, but that would be expensive because there are a lot of GPU architectures, and the compilation takes a lot of time.

I planned to finish the draft of wiki pages [2,3], but turns out I’m running out of time. I’ll catch up in week 11. My mentor is also busy in week 10, so my PR about rocm-opencl-runtime is still pending for review. Now we are working on solving the dependency issue of ROCm packages — gcc-12 and gcc-11.3.0 incompatibilities. Due to two bugs, the current stable gcc, gcc-11.3.0 cannot compile some ROCm packages [4], and the current unstable gcc, gcc-12, is unable to compile nearly all ROCm packages [5].

I’ll continue to do what’s postponed in week 10 — landing rocm.eclass and sci-libs packages, preparing cupy, fixing bugs, and writing the wiki pages. I’ll investigate MIOpen’s situation as well.

[1] https://github.com/ROCmSoftwarePlatform/rocFFT/issues/369
[2] https://wiki.gentoo.org/wiki/ROCm
[3] https://wiki.gentoo.org/wiki/HIP
[4] https://bugs.gentoo.org/842405
[5] https://bugs.gentoo.org/857660

Week 9 Report for Refining ROCm Packages in Gentoo

Gentoo Google Summer of Code (GSoC) September 11, 2022, 10:07

This week I mainly focused on dev-libs/rocm-opencl-runtime.

I bumped dev-libs/rocm-opencl-runtime to 5.1.3. That’s relatively easy. The difficult part is enabling its tests. I came across a major problem, which is oclgl test requiring X server. I compiled using debug options and use gdb to dive into the code, but found there is no simple solution. Currently the test needs a X server where OpenGL vender is AMD. Xvfb only provides llvmpipe, not meeting the requirements. I consulted some friends, they said NVIDIA recommends using EGL when there is no X [1], but apparently ROCm can only get OpenGL from X [2]. So my workaround is to let user passing an X display into the ebuild, by reading the environment variable OCLGL_DISPLAY (DISPLAY variable will be wiped when calling emerge, while this can survive). If no display is detected, or glxinfo shows the OpenGL vendor is not AMD, then src_test dies, throwing indications about running an X server using amdgpu driver.

I was also trapped by CRLF problem in src_test of dev-libs/rocm-opencl-runtime. Tests in oclperf.exclude should be skipped for oclperf test, but it did not. After numerous trials, I finally found that this file is using CRLF, not LF, which causes the exclusion failed ♦

Nevertheless, rocm-opencl-runtime tests passed on Radeon RX 6700XT! A good thing, because I know many user in Gentoo rely on this package to provide opencl in their computation, and the correctness is vital. Before we does not have src_test enabled. The PR is now in [6].

Other works including starting wiki writing [3,4], refine rocm.eclass according to feedback (not much, see gentoo-dev mailing list), and found a bug of dev-util/hipFindHIP.cmake module is not in the correct place. Fix can be found in [5] but I need to further polish the patch before PR.

If no further suggestions on rocm.eclass, I’ll land rocm.eclass in ::gentoo next week, and start bumping the sci-libs version already done locally.

[1] developer.nvidia.com/blog/egl-eye-opengl-visualization-without-x-server/
[2] github.com/RadeonOpenCompute/ROCm-OpenCL-Runtime/blob/bbdc87e08b322d349f82bdd7575c8ce94d31d276/tests/ocltst/module/common/OCLGLCommonLinux.cpp
[3] wiki.gentoo.org/wiki/ROCm
[4] wiki.gentoo.org/wiki/HIP
[5] github.com/littlewu2508/gentoo/tree/hip-correct-cmake
[6] github.com/gentoo/gentoo/pull/26870

This week I mainly focused on dev-libs/rocm-opencl-runtime.

I bumped dev-libs/rocm-opencl-runtime to 5.1.3. That’s relatively easy. The difficult part is enabling its tests. I came across a major problem, which is oclgl test requiring X server. I compiled using debug options and use gdb to dive into the code, but found there is no simple solution. Currently the test needs a X server where OpenGL vender is AMD. Xvfb only provides llvmpipe, not meeting the requirements. I consulted some friends, they said NVIDIA recommends using EGL when there is no X [1], but apparently ROCm can only get OpenGL from X [2]. So my workaround is to let user passing an X display into the ebuild, by reading the environment variable OCLGL_DISPLAY (DISPLAY variable will be wiped when calling emerge, while this can survive). If no display is detected, or glxinfo shows the OpenGL vendor is not AMD, then src_test dies, throwing indications about running an X server using amdgpu driver.

I was also trapped by CRLF problem in src_test of dev-libs/rocm-opencl-runtime. Tests in oclperf.exclude should be skipped for oclperf test, but it did not. After numerous trials, I finally found that this file is using CRLF, not LF, which causes the exclusion failed 🙁

Nevertheless, rocm-opencl-runtime tests passed on Radeon RX 6700XT! A good thing, because I know many user in Gentoo rely on this package to provide opencl in their computation, and the correctness is vital. Before we does not have src_test enabled. The PR is now in [6].

Other works including starting wiki writing [3,4], refine rocm.eclass according to feedback (not much, see gentoo-dev mailing list), and found a bug of dev-util/hipFindHIP.cmake module is not in the correct place. Fix can be found in [5] but I need to further polish the patch before PR.

If no further suggestions on rocm.eclass, I’ll land rocm.eclass in ::gentoo next week, and start bumping the sci-libs version already done locally.

[1] https://developer.nvidia.com/blog/egl-eye-opengl-visualization-without-x-server/
[2] https://github.com/RadeonOpenCompute/ROCm-OpenCL-Runtime/blob/bbdc87e08b322d349f82bdd7575c8ce94d31d276/tests/ocltst/module/common/OCLGLCommonLinux.cpp
[3] https://wiki.gentoo.org/wiki/ROCm
[4] https://wiki.gentoo.org/wiki/HIP
[5] https://github.com/littlewu2508/gentoo/tree/hip-correct-cmake
[6] https://github.com/gentoo/gentoo/pull/26870

Week 8 Report for Refining ROCm Packages in Gentoo

Gentoo Google Summer of Code (GSoC) September 11, 2022, 10:04

This week there are two major progress: dev-util/rocprofiler and rocm.eclass.

I have implemented all the functions I think necessary for rocm.eclass. It was just send to rocm.eclass draft to gentoo-dev mailing list (also with a Github PR at [1]), please have a review. In the following weeks, I will collect feedbacks and continue to polish it.

In summary, I have implemented those functions which is listed in my proposal:
USE_EXPNAD of amdgpu_targets_, and ROCM_USEDEP to make the use flag coherent among dependencies;
rocm_src_configure contains common arguments in src_prepare;
rocm_src_test which checks the permission on /dev/kfd and /dev/dri/render*

There are also something listed in proposal but I decided not to implement now:
rocm_src_prepare: although there are some similarities among ebuilds, src_prepare are highly customized to each ROCm components. Unifying would take extra work.
SRC_URI: currently all SRC_URI is already specified in each ebuilds. It does not hurt to keep the status quo.

Moreover, during implementation I found another feature necessary
rocm_src_test: correctly handles different scenarios. ROCm packages may have cmake test, which can be run using cmake_src_test, or only compiled some testing binaries which requires execution from command-line. I made rocm_src_test automatically detect the method, so ROCm packages just have to call this function directly without doing anything.

Actually I have never imagined rocm.eclass could be in this shape eventually. Initially I just thought it would provide some utilities, mainly src_test and USE_EXPAND. But when implementing I found all these feature requires careful treatment. The comments (mainly examples) also takes half of the length. It ends up in 278 lines, which is a middle-sized among current eclasses. Maybe it can be further trimmed down after polishing, because there could be awkward implementations or re-inventions in it.

Based on my draft rocm.eclass, I have prepared sci-libs/roc*=5.1.3, sci-lib/hip-*-5.1.3 and dev-python/cupy making use of it. It feels great to simplify the ebuilds, and portage can handles the USE_EXPAND and dependencies just as expected. Once the rocm.eclass get in tree, I’ll push those ROCm-5.1.3 ebuilds.

Anther thing to mention is that ROCm-5.1.3 toolchains finally get merged [5], with the fixed dev-util/rocprofiler-{4.3.0,5.0.2,5.1.3}. rocprofiler is actually buggy before, because I thought I committed the patch which stripped the libhsa-amd-aqlprofile.so loading (I even claimed it in the commit message), but it was not committed and lost in history. So I reproduced the patch. Also, I did some research about this proprietary lib. By default, not loading it means tracing hsa/hip is not possible — you only get basic information like name and time of each GPU kernel execution, but do not know the pipeline of kernel execution (which one has spawned which kernel). AQL should be HSA architected queuing language (HSA AQL), where llvm.org/docs/AMDGPUUsage.html#hsa-aql-queue documented. It did sound related to the pipeline of kernel dispatching. By the description, libhsa-amd-aqlprofile.so is an extension API of AQL Profile. But actually, patching the source code to let rocprofiler not loading libhsa-amd-aqlprofile.so does not breaks the tracing of hsa/hip. So, I’m not sure why libhsa-amd-aqlprofile.so is needed, and raised a question at [2]. So I complete the fix in [3,4].

According to the renewed proposal (I have been leaving for two weeks, so there are changes in plan), I should collect feedback and refine rocm.eclass, and prepare dev-python/cupy and sci-libs/rocWMMA. I’ll investigate ROCgdb, too. Also, rocm-device-libs is a major package because many users relies on it to provide opencl. I’ll work on bumping its version, too. What’s more, with hip-5.1.3 against vanilla clang, rocm for blender can land in ::gentoo.

[1] github.com/gentoo/gentoo/pull/26784
[2] github.com/RadeonOpenCompute/ROCm/issues/1781
[3] github.com/gentoo/gentoo/pull/26755
[4] github.com/gentoo/gentoo/pull/26771
[5] github.com/gentoo/gentoo/pull/26441

This week there are two major progress: dev-util/rocprofiler and rocm.eclass.

I have implemented all the functions I think necessary for rocm.eclass. It was just send to rocm.eclass draft to gentoo-dev mailing list (also with a Github PR at [1]), please have a review. In the following weeks, I will collect feedbacks and continue to polish it.

In summary, I have implemented those functions which is listed in my proposal:
USE_EXPNAD of amdgpu_targets_, and ROCM_USEDEP to make the use flag coherent among dependencies;
rocm_src_configure contains common arguments in src_prepare;
rocm_src_test which checks the permission on /dev/kfd and /dev/dri/render*

There are also something listed in proposal but I decided not to implement now:
rocm_src_prepare: although there are some similarities among ebuilds, src_prepare are highly customized to each ROCm components. Unifying would take extra work.
SRC_URI: currently all SRC_URI is already specified in each ebuilds. It does not hurt to keep the status quo.

Moreover, during implementation I found another feature necessary
rocm_src_test: correctly handles different scenarios. ROCm packages may have cmake test, which can be run using cmake_src_test, or only compiled some testing binaries which requires execution from command-line. I made rocm_src_test automatically detect the method, so ROCm packages just have to call this function directly without doing anything.

Actually I have never imagined rocm.eclass could be in this shape eventually. Initially I just thought it would provide some utilities, mainly src_test and USE_EXPAND. But when implementing I found all these feature requires careful treatment. The comments (mainly examples) also takes half of the length. It ends up in 278 lines, which is a middle-sized among current eclasses. Maybe it can be further trimmed down after polishing, because there could be awkward implementations or re-inventions in it.

Based on my draft rocm.eclass, I have prepared sci-libs/roc*=5.1.3, sci-lib/hip-*-5.1.3 and dev-python/cupy making use of it. It feels great to simplify the ebuilds, and portage can handles the USE_EXPAND and dependencies just as expected. Once the rocm.eclass get in tree, I’ll push those ROCm-5.1.3 ebuilds.

Anther thing to mention is that ROCm-5.1.3 toolchains finally get merged [5], with the fixed dev-util/rocprofiler-{4.3.0,5.0.2,5.1.3}. rocprofiler is actually buggy before, because I thought I committed the patch which stripped the libhsa-amd-aqlprofile.so loading (I even claimed it in the commit message), but it was not committed and lost in history. So I reproduced the patch. Also, I did some research about this proprietary lib. By default, not loading it means tracing hsa/hip is not possible — you only get basic information like name and time of each GPU kernel execution, but do not know the pipeline of kernel execution (which one has spawned which kernel). AQL should be HSA architected queuing language (HSA AQL), where https://llvm.org/docs/AMDGPUUsage.html#hsa-aql-queue documented. It did sound related to the pipeline of kernel dispatching. By the description, libhsa-amd-aqlprofile.so is an extension API of AQL Profile. But actually, patching the source code to let rocprofiler not loading libhsa-amd-aqlprofile.so does not breaks the tracing of hsa/hip. So, I’m not sure why libhsa-amd-aqlprofile.so is needed, and raised a question at [2]. So I complete the fix in [3,4].

According to the renewed proposal (I have been leaving for two weeks, so there are changes in plan), I should collect feedback and refine rocm.eclass, and prepare dev-python/cupy and sci-libs/rocWMMA. I’ll investigate ROCgdb, too. Also, rocm-device-libs is a major package because many users relies on it to provide opencl. I’ll work on bumping its version, too. What’s more, with hip-5.1.3 against vanilla clang, rocm for blender can land in ::gentoo.

[1] https://github.com/gentoo/gentoo/pull/26784
[2] https://github.com/RadeonOpenCompute/ROCm/issues/1781
[3] https://github.com/gentoo/gentoo/pull/26755
[4] https://github.com/gentoo/gentoo/pull/26771
[5] https://github.com/gentoo/gentoo/pull/26441

September 05 2022

Week 12 Report for RISC-V Support for Gentoo Prefix

Gentoo Google Summer of Code (GSoC) September 05, 2022, 9:34

Hello all,
Hope you all are doing good, this is my report for 12th week of my Google Summer of Code project.

I got documentation on Porting Prefix reviewed and I have added the suggested changes.

My GSoC delieverables have been completed, so I played around with the compatibility layer and ansible. Synced the latest changes to the bootstrap script from upstream and used it for installing prefix. Working on updating the main.yml[1] accordingly. The process has been smooth so far, within next few weeks we might have a working compatibility layer for RISC-V.

Will start working on the final report and update the blogs on Gentoo Blog site. Although the official period is over I will continue working on compatibility layer and there are also few other things like pkgcraft in my bucket list which I will get my hands on.

The 12 weeks of GSoC have been super fun, thanks to mentors and the community.

[1] github.com/EESSI/compatibility-layer/blob/main/ansible/playbooks/roles/compatibility_layer/defaults/main.yml

Regards,
wiredhikari

Hello all,
Hope you all are doing good, this is my report for 12th week of my Google Summer of Code project.

I got documentation on Porting Prefix reviewed and I have added the suggested changes.

My GSoC delieverables have been completed, so I played around with the compatibility layer and ansible. Synced the latest changes to the bootstrap script from upstream and used it for installing prefix. Working on updating the main.yml[1] accordingly. The process has been smooth so far, within next few weeks we might have a working compatibility layer for RISC-V.

Will start working on the final report and update the blogs on Gentoo Blog site. Although the official period is over I will continue working on compatibility layer and there are also few other things like pkgcraft in my bucket list which I will get my hands on.

The 12 weeks of GSoC have been super fun, thanks to mentors and the community.

[1] https://github.com/EESSI/compatibility-layer/blob/main/ansible/playbooks/roles/compatibility_layer/defaults/main.yml

Regards,
wiredhikari

September 04 2022

Gentoo musl Support Expansion for Qt/KDE Week 12

Gentoo Google Summer of Code (GSoC) September 04, 2022, 22:41

This week has been mostly been spent on writing documentation and fixing up some left over things.

I started with looking over the *-standalone libraries. It turns out that tree.h is provided by libbsd and because libbsd works just fine on musl I removed the standalone. The second thing I did was removing error.h because it caused issues with some builds, and we suspect it works on Void Linux because they build packages inside a clean chroot (without error.h). The only one left is now cdefs.h. This header is an internal glibc header, and using it is basically a bug, so upstreaming fixes should be very easy. Therefore I feel like this doesn’t need to be added either, so I closed the pull request for now.

Next I rewrote Sam’s musl porting notes, moving it from his personal page to a “real” wiki page (wiki.gentoo.org/wiki/Musl_porting_notes). It’s now more like a wiki page and less like a list of errors with attached fixes. I’ve also added several things myself into it.

Another wiki I’ve added stuff to is Chroot (wiki.gentoo.org/wiki/Chroot#Sound_and_graphics). In my GSoC planning I wanted to write documentation about using Gentoo musl. There I wanted information about how to work around using glibc programs that do not work on musl, ex proprietary programs. Instead of doing that I wrote documentation about how running graphical applications with sound into the Chroot documentation, as it helps every Gentoo user. I don’t think Gentoo musl users should have any issues finding the Chroot wikipage. ♦

I have also tested gettext-tiny on Gentoo musl. This is a smaller implementation of gettext with some functionality stubbed out. gettext-tiny is built for musl, and it makes use of the libintl provided by musl. For users that only want English this makes a lot of sense because it is much smaller than gettext but still allows most packages to be built. When replacing gettext Portage complained about two packages using uninstalled libraries from GNU gettext, those being bison and poxml. When reemerging bison it errored out and I was sure it was because of gettext, but after debugging bison I found out it was caused by error-standalone. After unmerging error-standalone bison detected that the library was not installed and it compiled correctly. Poxml on the other hand hard depends on libgettextpo, a library not provided by gettext-tiny. Running “equery d -a poxml” however we can see that nothing important actually depends on poxml, so gettext-tiny should for the most part be fine.

$ equery d -a poxml
* These packages depend on poxml:
kde-apps/kdesdk-meta-22.04.3-r1 (>=kde-apps/poxml-22.04.3:5)
kde-apps/kdesdk-meta-22.08.0 (>=kde-apps/poxml-22.08.0:5)

Next week I will write my final evaluation and then I am done with GSoC! I will however continue working with some things like ebuildshell and crossdev when I have time.

This week has been mostly been spent on writing documentation and fixing up some left over things.

I started with looking over the *-standalone libraries. It turns out that tree.h is provided by libbsd and because libbsd works just fine on musl I removed the standalone. The second thing I did was removing error.h because it caused issues with some builds, and we suspect it works on Void Linux because they build packages inside a clean chroot (without error.h). The only one left is now cdefs.h. This header is an internal glibc header, and using it is basically a bug, so upstreaming fixes should be very easy. Therefore I feel like this doesn’t need to be added either, so I closed the pull request for now.

Next I rewrote Sam’s musl porting notes, moving it from his personal page to a “real” wiki page (https://wiki.gentoo.org/wiki/Musl_porting_notes). It’s now more like a wiki page and less like a list of errors with attached fixes. I’ve also added several things myself into it.

Another wiki I’ve added stuff to is Chroot (https://wiki.gentoo.org/wiki/Chroot#Sound_and_graphics). In my GSoC planning I wanted to write documentation about using Gentoo musl. There I wanted information about how to work around using glibc programs that do not work on musl, ex proprietary programs. Instead of doing that I wrote documentation about how running graphical applications with sound into the Chroot documentation, as it helps every Gentoo user. I don’t think Gentoo musl users should have any issues finding the Chroot wikipage. 🙂

I have also tested gettext-tiny on Gentoo musl. This is a smaller implementation of gettext with some functionality stubbed out. gettext-tiny is built for musl, and it makes use of the libintl provided by musl. For users that only want English this makes a lot of sense because it is much smaller than gettext but still allows most packages to be built. When replacing gettext Portage complained about two packages using uninstalled libraries from GNU gettext, those being bison and poxml. When reemerging bison it errored out and I was sure it was because of gettext, but after debugging bison I found out it was caused by error-standalone. After unmerging error-standalone bison detected that the library was not installed and it compiled correctly. Poxml on the other hand hard depends on libgettextpo, a library not provided by gettext-tiny. Running “equery d -a poxml” however we can see that nothing important actually depends on poxml, so gettext-tiny should for the most part be fine.

$ equery d -a poxml
* These packages depend on poxml:
kde-apps/kdesdk-meta-22.04.3-r1 (>=kde-apps/poxml-22.04.3:5)
kde-apps/kdesdk-meta-22.08.0 (>=kde-apps/poxml-22.08.0:5)

Next week I will write my final evaluation and then I am done with GSoC! I will however continue working with some things like ebuildshell and crossdev when I have time.

September 02 2022

Chromium Flags

Maciej Barć (xgqt) September 02, 2022, 4:37
Table of Contents
  • Dark interface
  • Hardware acceleration
  • Reader mode
  • Runtime cache
  • Window size
  • Full config
Dark interface

Force web UI dark mode

--enable-features=WebUIDarkMode --force-dark-mode
Hardware acceleration

Accelerated video decoding and GPU support

--enable-accelerated-video-decode --enable-gpu
Reader mode
--enable-reader-mode
Runtime cache

Use the cache directory /run/user/1000/chrome/cache

--disk-cache-dir=${XDG_RUNTIME_DIR}/chrome/cache
Window size

Sawn a window of size 1200x900

--window-size=1200,900
Full config

The file /etc/chromium/default is sourced by the Chromium launcher, that's why we can sue bash syntax here.

#!/usr/bin/env bash

# Dark interface
# Force web UI dark mode
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --enable-features=WebUIDarkMode --force-dark-mode"

# Hardware acceleration
# Accelerated video decoding and GPU support
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --enable-accelerated-video-decode --enable-gpu"

# Reader mode
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --enable-reader-mode"

# Runtime cache
# Use the cache directory /run/user/1000/chrome/cache
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --disk-cache-dir=${XDG_RUNTIME_DIR}/chrome/cache"

# Window size
# Sawn a window of size 1200x900
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --window-size=1200,900"
xgqt (xgqt ) September 02, 2022, 4:37

Dark interface

Force web UI dark mode

--enable-features=WebUIDarkMode --force-dark-mode

Hardware acceleration

Accelerated video decoding and GPU support

--enable-accelerated-video-decode --enable-gpu

Reader mode

--enable-reader-mode

Runtime cache

Use the cache directory /run/user/1000/chrome/cache

--disk-cache-dir=${XDG_RUNTIME_DIR}/chrome/cache

Window size

Sawn a window of size 1200x900

--window-size=1200,900

Full config

The file /etc/chromium/default is sourced by the Chromium launcher, that's why we can sue bash syntax here.

#!/usr/bin/env bash

# Dark interface
# Force web UI dark mode
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --enable-features=WebUIDarkMode --force-dark-mode"

# Hardware acceleration
# Accelerated video decoding and GPU support
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --enable-accelerated-video-decode --enable-gpu"

# Reader mode
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --enable-reader-mode"

# Runtime cache
# Use the cache directory /run/user/1000/chrome/cache
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --disk-cache-dir=${XDG_RUNTIME_DIR}/chrome/cache"

# Window size
# Sawn a window of size 1200x900
CHROMIUM_FLAGS="${CHROMIUM_FLAGS} --window-size=1200,900"

August 29 2022

Gentoo musl Support Expansion for Qt/KDE Week 11

Gentoo Google Summer of Code (GSoC) August 29, 2022, 21:32

This week has mostly been dedicated to fixing old, and harder problems that I had previously put off. I spent a whole lot of time learning about the AccountsService codebase and setting up systems with LDAP authentication, but it turned out it didn’t need a rewrite after reading a couple of issues on the GitLab page, more on that later.

To start with I added a CMAKE_SKIP_TESTS variable to cmake.eclass. Currently you need to specify skipped tests by doing myctestargs=( -E ‘(test1|test2|test3)’ ). This works fine for the most part, but if you need to specify skipped tests multiple times it gets really messy, because ctest does not allow you to pass -E multiple times. Personally I ran into this when fixing tests for kde-apps/okular. Most tests for Okular only pass when it’s installed (#653618), but the ebuild already skips some tests for other reasons. So I needed to first unconditionally disable some tests, and then conditionally with “has_version ${P} || append tests”. To solve it I introduced an array and then parsed it with myctestargs+=( -E '('$( IFS='|'; echo "${CMAKE_SKIP_TESTS[*]}")')' ), but as this was useful for a lot more ebuilds than just Okular I decided to implement it in the eclass.

The second thing I worked on was AccountsService, it’s a daemon that retrieves a list of users on the system and presents them with a DBus interface. It’s used for showing users in things like login managers and accounts settings panels. I actually worked on this a long time ago but I put it off for a while because it required a bigger rewrite, and I had more important things to do back then.
AccountsService has two issues on musl. It uses the glibc function fgetspent_r, and wtmp which is just implemented as stubs in musl (wiki.musl-libc.org/faq.html#Q:-Why-is-the-utmp/wtmp-functionality-only-implemented-as-stubs?). I asked in #musl to figure out a fgetspent_r replacement, but we then discussed why it was bad to enumerate /etc/passwd to get a list of users, for example it does not respect non-local (LDAP/NIS users), so AS needed a bigger rewrite, we thought :).
So I started with setting up two virtual machines, one LDAP client, and one server. Having never used LDAP before this was a little hard but I got it working. I also needed to set up QEMU networking so that my VMs could connect to each other, and I also set up an LDAP webui called ldap-ui so I could easily get an overview of my LDAP instance. Because AS works by providing a DBus interface I also learned using the qdbusviewer and dbus-send tools. Before taking a deep dive into the AS source code I wrote some small test programs to get comfortable with the DBus C API, passwd+shadow database functions, and GLib.
I then started reading the AccountsService source code to understand it better, its main.c just sets up a daemon that’s mostly defined in daemon.c, the rest of the source files are mostly just helpers and wrappers. When the daemon initializes it sets up user enumerators using the entry_generator_* functions. The main one is entry_generator_fgetpwent, this generator uses fgetspent_r to enumerate /etc/passwd, and my idea was to replace it with getpwent + getspnam. But there are two other generators, requested_users and cachedir. requested_users takes a requested user (ex. when manually entering username+password in the login manager), and adds it into /var/lib/AccountsService/users. cachedir looks at that directory and adds these entries into the daemon. It turns out that requesting a non-local LDAP user with the requested_users generator is completely fine, and the login information will be cached in the dir so that the cachedir generator can expose it for future logins. I then looked at some issues in the AccountsService GitLab, and it turns out that enumerating /etc/passwd was intentional to not blow up the login screen with thousands of users on a big LDAP domain for example. So, the rewrite was sadly not needed, but I learned a lot! Still, fixing fgetspent_r and wtmp needs to get done, but I already have a fix for that.

Another thing I spent a lot of time on this week was poxml. This is also an old issue that I put off, mostly because it was too hard at the time. The build crashes because it can’t find the function gl_get_setlocale_null_lock in libgettextpo.so. This shared object belongs to GNU Gettext, so I something was wrong with that. Looking at the so with nm --dynamic /usr/lib/libgettextpo.so I could see that the function was undefined, bad! We reported this issue to upstream and got into a long conversation. Apparently Bruno (GNU) used Alpine Linux which packages GNU libintl, while Gentoo uses the musl libintl implementation. GNU libintl actually provides gl_get_setlocale_null_lock which explains why it worked on Alpine without issue. After grepping for gl_get_setlocale_null_lock I found this:
/* When it is known that the gl_get_setlocale_null_lock function is defined by a dependency library, it should not be defined here. */
#if OMIT_SETLOCALE_LOCK
*do nothing*
#else
*define gl_get_setlocale_null_lock*
#fi

So I tried just forcing the check to false, and it worked! I then looked at the build system and expected something like AC_SEARCH_LIBS([gl_get_setlocale_null_lock], [intl], ...) *set OMIT_SETLOCALE_LOCK*, but it turns out that autotools just forces OMIT_SETLOCALE_LOCK to 1. This is clearly wrong so I sent another comment upstream and temporarily fixed it in the Gentoo tree. Instead of doing it properly I made an ugly hack to not get complacent (sams idea) and hopefully we can get it resolved upstream instead :D.

To summarize I feel like this week has gone pretty good. I’ve solved everything that was left and now I’m ready to start writing a lot of documentation. A lot of the accountsservice setup and work was ultimately unnecessary but I still learned a lot.

This week has mostly been dedicated to fixing old, and harder problems that I had previously put off. I spent a whole lot of time learning about the AccountsService codebase and setting up systems with LDAP authentication, but it turned out it didn’t need a rewrite after reading a couple of issues on the GitLab page, more on that later.

To start with I added a CMAKE_SKIP_TESTS variable to cmake.eclass. Currently you need to specify skipped tests by doing myctestargs=( -E ‘(test1|test2|test3)’ ). This works fine for the most part, but if you need to specify skipped tests multiple times it gets really messy, because ctest does not allow you to pass -E multiple times. Personally I ran into this when fixing tests for kde-apps/okular. Most tests for Okular only pass when it’s installed (#653618), but the ebuild already skips some tests for other reasons. So I needed to first unconditionally disable some tests, and then conditionally with “has_version ${P} || append tests”. To solve it I introduced an array and then parsed it with myctestargs+=( -E '('$( IFS='|'; echo "${CMAKE_SKIP_TESTS[*]}")')' ), but as this was useful for a lot more ebuilds than just Okular I decided to implement it in the eclass.

The second thing I worked on was AccountsService, it’s a daemon that retrieves a list of users on the system and presents them with a DBus interface. It’s used for showing users in things like login managers and accounts settings panels. I actually worked on this a long time ago but I put it off for a while because it required a bigger rewrite, and I had more important things to do back then.
AccountsService has two issues on musl. It uses the glibc function fgetspent_r, and wtmp which is just implemented as stubs in musl (https://wiki.musl-libc.org/faq.html#Q:-Why-is-the-utmp/wtmp-functionality-only-implemented-as-stubs?). I asked in #musl to figure out a fgetspent_r replacement, but we then discussed why it was bad to enumerate /etc/passwd to get a list of users, for example it does not respect non-local (LDAP/NIS users), so AS needed a bigger rewrite, we thought :).
So I started with setting up two virtual machines, one LDAP client, and one server. Having never used LDAP before this was a little hard but I got it working. I also needed to set up QEMU networking so that my VMs could connect to each other, and I also set up an LDAP webui called ldap-ui so I could easily get an overview of my LDAP instance. Because AS works by providing a DBus interface I also learned using the qdbusviewer and dbus-send tools. Before taking a deep dive into the AS source code I wrote some small test programs to get comfortable with the DBus C API, passwd+shadow database functions, and GLib.
I then started reading the AccountsService source code to understand it better, its main.c just sets up a daemon that’s mostly defined in daemon.c, the rest of the source files are mostly just helpers and wrappers. When the daemon initializes it sets up user enumerators using the entry_generator_* functions. The main one is entry_generator_fgetpwent, this generator uses fgetspent_r to enumerate /etc/passwd, and my idea was to replace it with getpwent + getspnam. But there are two other generators, requested_users and cachedir. requested_users takes a requested user (ex. when manually entering username+password in the login manager), and adds it into /var/lib/AccountsService/users. cachedir looks at that directory and adds these entries into the daemon. It turns out that requesting a non-local LDAP user with the requested_users generator is completely fine, and the login information will be cached in the dir so that the cachedir generator can expose it for future logins. I then looked at some issues in the AccountsService GitLab, and it turns out that enumerating /etc/passwd was intentional to not blow up the login screen with thousands of users on a big LDAP domain for example. So, the rewrite was sadly not needed, but I learned a lot! Still, fixing fgetspent_r and wtmp needs to get done, but I already have a fix for that.

Another thing I spent a lot of time on this week was poxml. This is also an old issue that I put off, mostly because it was too hard at the time. The build crashes because it can’t find the function gl_get_setlocale_null_lock in libgettextpo.so. This shared object belongs to GNU Gettext, so I something was wrong with that. Looking at the so with nm --dynamic /usr/lib/libgettextpo.so I could see that the function was undefined, bad! We reported this issue to upstream and got into a long conversation. Apparently Bruno (GNU) used Alpine Linux which packages GNU libintl, while Gentoo uses the musl libintl implementation. GNU libintl actually provides gl_get_setlocale_null_lock which explains why it worked on Alpine without issue. After grepping for gl_get_setlocale_null_lock I found this:
/* When it is known that the gl_get_setlocale_null_lock function is defined by a dependency library, it should not be defined here. */
#if OMIT_SETLOCALE_LOCK
*do nothing*
#else
*define gl_get_setlocale_null_lock*
#fi

So I tried just forcing the check to false, and it worked! I then looked at the build system and expected something like AC_SEARCH_LIBS([gl_get_setlocale_null_lock], [intl], ...) *set OMIT_SETLOCALE_LOCK*, but it turns out that autotools just forces OMIT_SETLOCALE_LOCK to 1. This is clearly wrong so I sent another comment upstream and temporarily fixed it in the Gentoo tree. Instead of doing it properly I made an ugly hack to not get complacent (sams idea) and hopefully we can get it resolved upstream instead :D.

To summarize I feel like this week has gone pretty good. I’ve solved everything that was left and now I’m ready to start writing a lot of documentation. A lot of the accountsservice setup and work was ultimately unnecessary but I still learned a lot.

August 28 2022

Week 11 Report for RISC-V Support for Gentoo Prefix

Gentoo Google Summer of Code (GSoC) August 28, 2022, 9:32

Hello all,

Hope everyone is fine. This is my report for the 11th week of my GSoC project. This week I worked on documentation, closing dangling pr’s and looked into bootstrapping the EESSI compat layer for RISC-V. I spent some of my time learning Ansible as a part of the process.

The documentation[1] is almost complete, I will work on feedbacks of mentors and pass it through some review softwares and fix accordingly. In the upcoming week I will look into EESSI compat layer for RISC-V and a blog for end-term evaluations.

[1] github.com/wiredhikari/prefix_on_riscv/blob/main/docs/porting.md

Regards,

wiredhikari

Hello all,

Hope everyone is fine. This is my report for the 11th week of my GSoC project. This week I worked on documentation, closing dangling pr’s and looked into bootstrapping the EESSI compat layer for RISC-V. I spent some of my time learning Ansible as a part of the process.

The documentation[1] is almost complete, I will work on feedbacks of mentors and pass it through some review softwares and fix accordingly. In the upcoming week I will look into EESSI compat layer for RISC-V and a blog for end-term evaluations.

[1] https://github.com/wiredhikari/prefix_on_riscv/blob/main/docs/porting.md

Regards,

wiredhikari

August 26 2022

Tighter integration of KDE with GNU Emacs

Maciej Barć (xgqt) August 26, 2022, 20:23
Table of Contents
  • Proper window size
  • Opening files from Dolphin in one Emacs instance
  • Breeze theme
    • Xresources
  • Dbus integration
  • KDE development
  • Opening files in different applications
  • Outside Emacs, inside Plasma
Proper window size

Sometimes while the Emacs GUI window is tiled to a side or maximized small gaps may appear around the window. This "bug" can be worked around by:

  • right-click on the title bar,
  • "More Actions",
  • "Configure Special Window Settings…",
  • "Add Property",
  • "Obey Geometry Restrictions",
  • Select "Force" form the combo box,
  • Select "No" from the radio buttons.
Opening files from Dolphin in one Emacs instance

Emacs daemon can help with that. But before you run emacs --daemon, I need You to know that there might be a better way:

(unless (or noninteractive (server-running-p))
  (server-start))

Adding the above to Your Emacs config will cause Emacs to start a daemon after it is opened (and no other Emacs servers are running), this also does not require --daemon flag.

After the daemon is started You can open files by right-clicking on them and selecting to open them in "Emacsclient".

Furthermore: You also utilize --iconic and add emacs --iconic to your Plasma startup. This is way better than using emacs --daemon because you can just click on your taskbar to open the minimized Emacs window. Also, Emacs will load all Your graphical libraries and configurations so Your theme will look properly and not as if Emacs was being used on the console.

Breeze theme

Sadly I have not found any theme that would look like Plasma. I use the spacemacs theme which looks a little bit similar, especially the background color comes close to Breeze's dark background color.

Note that the theme which You load with the function load-theme is a different thing that the GTK theme Emacs uses.

The GTK theme should be enabled if Your Emacs version is built with GTK support. On Gentoo this setting is controlled with the gtk USE flag. Also the flag toolkit-scroll-bars can be enabled for a look of scroll-bars consistent with the selected toolkit.

Xresources

There is a different approach to theming Your Emacs that loading a theme defined in ELisp - You can use a ~/.Xresource file.

If you do not load any theme in your configuration Emacs will by default read the .Xresources file, unless the --no-x-resources flag is used.

Here are a few Xresources config files that come close to the default Breeze theme:

  • github.com/kuriot/xresources-breeze-theme/blob/master/.Xresources
  • gitcode.net/mirrors/mbadolato/iTerm2-Color-Schemes/-/blob/master/Xresources/Breeze
Dbus integration

Emacs can be built with FreeDesktop's D-Bus support to communicate over the dbus protocol. This can come handy when using ERC as it has a setting to enable desktop notifications on mentions (erc-desktop-notifications.el).

The dbus interface can also be utilized to query desktop-oriented daemons, for example this library talks to the Bluetooth daemon.

KDE development

Those are some ELisp libraries that I found while browsing GitHub, they might be useful for somebody who delves into KDE app development.

Opening files in different applications

In addition to async-shell-command and start-process-shell-command I wrote this small library that may come handy.

Outside Emacs, inside Plasma

Sadly the KDE team did not add support to emulate Emacs-like keys in Plasma itself, but some applications like, for example Kate have configuration options to customize the key bindings. This is a repository explaining how to setup Kate's bindings.

Proper window size

Sometimes while the Emacs GUI window is tiled to a side or maximized small gaps may appear around the window. This "bug" can be worked around by:

  • right-click on the title bar,
  • "More Actions",
  • "Configure Special Window Settings…",
  • "Add Property",
  • "Obey Geometry Restrictions",
  • Select "Force" form the combo box,
  • Select "No" from the radio buttons.

Opening files from Dolphin in one Emacs instance

Emacs daemon can help with that. But before you run emacs --daemon, I need You to know that there might be a better way:

(unless (or noninteractive (server-running-p))
  (server-start))

Adding the above to Your Emacs config will cause Emacs to start a daemon after it is opened (and no other Emacs servers are running), this also does not require --daemon flag.

After the daemon is started You can open files by right-clicking on them and selecting to open them in "Emacsclient".

Furthermore: You also utilize --iconic and add emacs --iconic to your Plasma startup. This is way better than using emacs --daemon because you can just click on your taskbar to open the minimized Emacs window. Also, Emacs will load all Your graphical libraries and configurations so Your theme will look properly and not as if Emacs was being used on the console.

Breeze theme

Sadly I have not found any theme that would look like Plasma. I use the spacemacs theme which looks a little bit similar, especially the background color comes close to Breeze's dark background color.

Note that the theme which You load with the function load-theme is a different thing that the GTK theme Emacs uses.

The GTK theme should be enabled if Your Emacs version is built with GTK support. On Gentoo this setting is controlled with the gtk USE flag. Also the flag toolkit-scroll-bars can be enabled for a look of scroll-bars consistent with the selected toolkit.

Xresources

There is a different approach to theming Your Emacs that loading a theme defined in ELisp - You can use a ~/.Xresource file.

If you do not load any theme in your configuration Emacs will by default read the .Xresources file, unless the --no-x-resources flag is used.

Here are a few Xresources config files that come close to the default Breeze theme:

Dbus integration

Emacs can be built with FreeDesktop's D-Bus support to communicate over the dbus protocol. This can come handy when using ERC as it has a setting to enable desktop notifications on mentions (erc-desktop-notifications.el).

The dbus interface can also be utilized to query desktop-oriented daemons, for example this library talks to the Bluetooth daemon.

KDE development

Those are some ELisp libraries that I found while browsing GitHub, they might be useful for somebody who delves into KDE app development.

Opening files in different applications

In addition to async-shell-command and start-process-shell-command I wrote this small library that may come handy.

Outside Emacs, inside Plasma

Sadly the KDE team did not add support to emulate Emacs-like keys in Plasma itself, but some applications like, for example Kate have configuration options to customize the key bindings. This is a repository explaining how to setup Kate's bindings.

VIEW

SCOPE

FILTER
  from
  to