Gentoo Logo
Gentoo Logo Side
Gentoo Spaceship

Contributors:
. Aaron W. Swenson
. Agostino Sarubbo
. Alexey Shvetsov
. Alexis Ballier
. Alexys Jacob
. Alice Ferrazzi
. Alice Ferrazzi
. Andreas K. Hüttel
. Anthony Basile
. Arun Raghavan
. Bernard Cafarelli
. Brian Harring
. Christian Ruppert
. Chí-Thanh Christopher Nguyễn
. Denis Dupeyron
. Detlev Casanova
. Diego E. Pettenò
. Domen Kožar
. Doug Goldstein
. Eray Aslan
. Fabio Erculiani
. Gentoo Haskell Herd
. Gentoo Miniconf 2016
. Gentoo Monthly Newsletter
. Gentoo News
. Gilles Dartiguelongue
. Greg KH
. Göktürk Yüksek
. Hanno Böck
. Hans de Graaff
. Ian Whyman
. Jan Kundrát
. Jason A. Donenfeld
. Jeffrey Gardner
. Joachim Bartosik
. Johannes Huber
. Jonathan Callen
. Jorge Manuel B. S. Vicetto
. Kristian Fiskerstrand
. Lance Albertson
. Liam McLoughlin
. Luca Barbato
. Marek Szuba
. Mart Raudsepp
. Matt Turner
. Matthew Thode
. Michael Palimaka
. Michal Hrusecky
. Michał Górny
. Mike Doty
. Mike Gilbert
. Mike Pagano
. Nathan Zachary
. Pacho Ramos
. Patrick Kursawe
. Patrick Lauer
. Patrick McLean
. Paweł Hajdan, Jr.
. Petteri Räty
. Piotr Jaroszyński
. Rafael G. Martins
. Remi Cardona
. Richard Freeman
. Robin Johnson
. Ryan Hill
. Sean Amoss
. Sebastian Pipping
. Steev Klimaszewski
. Stratos Psomadakis
. Sven Vermeulen
. Sven Wegener
. Tom Wijsman
. Tomáš Chvátal
. Yury German
. Zack Medico

Last updated:
August 21, 2017, 02:06 UTC

Disclaimer:
Views expressed in the content published here do not necessarily represent the views of Gentoo Linux or the Gentoo Foundation.


Bugs? Comments? Suggestions? Contact us!

Powered by:
Planet Venus

Welcome to Gentoo Universe, an aggregation of weblog articles on all topics written by Gentoo developers. For a more refined aggregation of Gentoo-related topics only, you might be interested in Planet Gentoo.

August 19, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)

This post is part of a series of free ideas that I’m posting on my blog in the hope that someone with more time can implement. It’s effectively a very sketched proposal that comes with no design attached, but if you have time you would like to spend learning something new, but no idea what to do, it may be a good fit for you.

Going back to a previous topic I wrote about, and the fact that I’m trying to set up a secure WordPress instance, I would like to throw out another idea I won’t have time to implement myself any time soon.

When running complex web applications, such as WordPress, defense-in-depth is a good security practice. This means that in addition to locking down what the code itself can do on to the state of the local machine, it also makes sense to limit what it can do to the external state and the Internet at large. Indeed, even if you cannot drop a shell on a remote server, there is value (negative for the world, positive for the attacker) to at least being able to use it form DDoS (e.g. through an amplification attack).

With that in mind, if your app does not require network at all, or the network dependency can be sacrificed (like I did for Typo), just blocking the user from making outgoing connection with iptables would be enough. The --uid-owner option makes it very easy to figure out who’s trying to open new connections, and thus stop a single user transmitting unwanted traffic. Unfortunately, this does not always work because sometimes the application really needs network support. In the case of WordPress, there is a definite need to contact the WordPress servers, both to install plugins and to check if it should self-update.

You could try to limit access to what the user can access by hosts. But that’s not easy to implement right either. Take WordPress as an example still: if you wanted to limit access to the WordPress infrastructure, you would effectively have to allow it accessing *.wordpress.org, and this can’t really be done in iptables, at far as I know, since those connections go to IP literal addresses. You could rely on FcRDNS to verify the connections, but that can be slow, and if you happen to have access to poison the DNS cache of the server, you’re effectively in control of this kind of ACL. I ignored the option of just using “standard” reverse DNS resolution, because in that case you don’t even need to poison DNS, you can just decide what your IP will reverse-resolve to.

So what you need to do is actually filter at the connection-request level, which is what proxies are designed for. I’ll be assuming we want to have a non-terminating proxy (because terminating proxies are hard), but even in that case you can now know what (forward) address you want to connect to, and in that case *.wordpress.org becomes a valid ACL to use. And this is something you can actually do relatively easily with Squid, for instance. Indeed, this is the whole point of tools such as ufdbguard (which I used to maintain for Gentoo), and the ICP protocol. But Squid is particularly designed as a caching proxy, it’s not lightweight at all, and it can easily become a liability to have it in your server stack.

Up to now, what I have used to reduce the surface of attacks of my webapps is set them behind a tinyproxy, which does not really allow for per-connection ACLs. This only provides isolation against random non-proxied connections, but it’s a starting point. And here is where I want to provide a free idea for anyone who has the time and would like to provide better security tools for srver-side defense-in-depth.

A server-side proxy for this kind of security usage would have to be able to provide ACLs, with both positive and negative lists. You may want to provide all access to *.wordpress.org, but at the same time block all non-TLS-encrypted traffic, to avoid the possibility of downgrade (given that WordPress has a silent downgrade for requests to api.wordpress.org, that I talked about before).

Even better, such a proxy should have the ability to distinguish the ACLs based on which user (i.e. which webapp) is making the request. The obvious way would be to provide separate usernames to authenticate to the proxy — which again Squid can do, but it’s designed for clients for which the validation of username and password is actually important. Indeed, for this target usage, I would ignore the password altogether, and just use the user at face value, since the connection should always only be local. I would be even happier if instead of pseudo-authenticating to the proxy, the proxy could figure out which (local) user the connection came from, by inspecting the TCP socket connection, kind of like querying the ident protocol used to work for IRC.

So to summarise, what I would like to have is an HTTP(S) proxy that focuses on securing server-side web applications. Does not have to support TLS transport (because it should only accept local connections), nor it should be a terminating proxy. It should support ACLs that allow/deny access to a subset of hosts, possibly per-user, without needing a user database of any sort, and even better if it can tell by itself which user the connection came from. I’m more than happy if someone tells me this already exists, or if not, someone starts writing this… thank you!

Hardened Linux kernel sources removal (August 19, 2017, 00:00 UTC)

As you may know the core of sys-kernel/hardened-sources has been the grsecurity patches. Recently the grsecurity developers have decided to limit access to these patches. As a result, the Gentoo Hardened team is unable to ensure a regular patching schedule and therefore the security of the users of these kernel sources. Thus, we will be masking hardened-sources on the 27th of August and will proceed to remove them from the package repository by the end of September. We recommend to use sys-kernel/gentoo-sources instead. Userspace hardening and support for SELinux will of course remain in the Gentoo package tree. Please see the full news item for additional information and links.

August 18, 2017

FroSCon logo

Upcoming weekend, 19-20th August 2017, there will be a Gentoo booth again at the FrOSCon “Free and Open Source Conference” 12, in St. Augustin near Bonn! Visitors can see Gentoo live in action, get Gentoo swag, and prepare, configure, and compile their own Gentoo buttons. See you there!

August 16, 2017

Description:
openjpeg is an open-source JPEG 2000 library.

The complete ASan output of the issue:

# opj_compress -I -cinema4K -n 1 -i $FILE -o null.jp2
==133214==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61100000012b at pc 0x7f221efde81a bp 0x7ffd4c1d9ad0 sp 0x7ffd4c1d9ac8           
WRITE of size 1 at 0x61100000012b thread T0                                                                                                          
    #0 0x7f221efde819 in opj_write_bytes_LE /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/cio.c:67:23               
    #1 0x7f221f0261b8 in opj_j2k_write_sot /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:4237:5               
    #2 0x7f221f0261b8 in opj_j2k_write_all_tile_parts /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11604     
    #3 0x7f221f0261b8 in opj_j2k_post_write_tile /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11273          
    #4 0x7f221f0240fd in opj_j2k_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11014:15                
    #5 0x7f221f06edf8 in opj_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/openjpeg.c:775:20                 
    #6 0x50b9a2 in main /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/opj_compress.c:1990:36                            
    #7 0x7f221da06680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289                       
    #8 0x41bc78 in _start (/usr/bin/opj_compress+0x41bc78)                                                                                           
                                                                                                                                                     
0x61100000012b is located 0 bytes to the right of 235-byte region [0x611000000040,0x61100000012b)                                                    
allocated by thread T0 here:                                                                                                                         
    #0 0x4d1628 in malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:66         
    #1 0x7f221f11a8a9 in opj_malloc /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/opj_malloc.c:196:12               
    #2 0x7f221f051260 in opj_j2k_update_rates /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:5156:22
    #3 0x7f221f027f8c in opj_j2k_exec /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:7940:33
    #4 0x7f221f027f8c in opj_j2k_start_compress /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11089
    #5 0x7f221f059260 in opj_jp2_start_compress /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/jp2.c:2474:12
    #6 0x7f221f06ec9c in opj_start_compress /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/openjpeg.c:758:20
    #7 0x50b96f in main /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/opj_compress.c:1967:20
    #8 0x7f221da06680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/cio.c:67:23 in opj_write_bytes_LE
Shadow bytes around the buggy address:
  0x0c227fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c227fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c227fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c227fff8020: 00 00 00 00 00[03]fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==133214==ABORTING
CINEMA 4K profile activated
Other options specified could be overridden
[WARNING] JPEG 2000 Profile-4 (4k dc profile) requires:
Number of decomposition levels >= 1 &&  Number of decomposition levels forced to 1 (rather than 2)
[WARNING] JPEG 2000 Profile-3 and 4 (2k/4k dc profile) requires:
Maximum 1302083 compressed bytes @ 24fps
As no rate has been given, this limit will be used.
[WARNING] JPEG 2000 Profile-3 and 4 (2k/4k dc profile) requires:
Maximum 1041666 compressed bytes @ 24fps
As no rate has been given, this limit will be used.
[WARNING] JPEG 2000 Profile-3 (2k dc profile) requires:
Precision of each component shall be 12 bits unsigned-> At least component 0 of input image (8 bits, unsigned) is not compliant
-> Non-profile-3 codestream will be generated
[INFO] tile number 1 / 1

Affected version:
2.2.0

Fixed version:
N/A

Commit fix:
https://github.com/uclouvain/openjpeg/commit/4241ae6fbbf1de9658764a80944dc8108f2b4154

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:

Reproducer:
https://github.com/asarubbo/poc/blob/master/00317-openjpeg-heapoverflow-opj_write_bytes_LE

Timeline:
2017-08-15: bug discovered and reported to upstream
2017-08-15: upstream released a fix
2017-08-16: blog post about the issue

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

openjpeg: heap-based buffer overflow in opj_write_bytes_LE (cio.c)

Description:
openjpeg is an open-source JPEG 2000 library.

The complete ASan output of the issue:

# opj_compress -n 1 -i $FILE -o null.j2c
==81142==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b6 at pc 0x7fc39ca4a189 bp 0x7fff91c10aa0 sp 0x7fff91c10a98
WRITE of size 1 at 0x6020000000b6 thread T0
    #0 0x7fc39ca4a188 in opj_mqc_flush /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/mqc.c
    #1 0x7fc39ca7db6a in opj_t1_encode_cblk /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/t1.c:2213:21
    #2 0x7fc39ca7db6a in opj_t1_encode_cblks /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/t1.c:2061
    #3 0x7fc39cae8689 in opj_tcd_t1_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/tcd.c:2184:11
    #4 0x7fc39cae8689 in opj_tcd_encode_tile /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/tcd.c:1362
    #5 0x7fc39ca05527 in opj_j2k_write_sod /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:4661:11
    #6 0x7fc39ca05527 in opj_j2k_write_first_tile_part /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11507
    #7 0x7fc39ca05527 in opj_j2k_post_write_tile /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11265
    #8 0x7fc39ca040fd in opj_j2k_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11014:15
    #9 0x7fc39ca4edf8 in opj_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/openjpeg.c:775:20
    #10 0x50b9a2 in main /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/opj_compress.c:1990:36
    #11 0x7fc39b3e6680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #12 0x41bc78 in _start (/usr/bin/opj_compress+0x41bc78)

0x6020000000b6 is located 0 bytes to the right of 6-byte region [0x6020000000b0,0x6020000000b6)
allocated by thread T0 here:
    #0 0x4d1628 in malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:66
    #1 0x7fc39cafa8a9 in opj_malloc /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/opj_malloc.c:196:12
    #2 0x7fc39cae3522 in opj_tcd_code_block_enc_allocate_data /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/tcd.c:1196:42
    #3 0x7fc39cae3522 in opj_tcd_init_tile /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/tcd.c:1113
    #4 0x7fc39c9ff364 in opj_j2k_pre_write_tile /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:11115:11
    #5 0x7fc39c9ff364 in opj_j2k_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/j2k.c:10958
    #6 0x7fc39ca4edf8 in opj_encode /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/openjpeg.c:775:20
    #7 0x50b9a2 in main /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/opj_compress.c:1990:36
    #8 0x7fc39b3e6680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/mqc.c in opj_mqc_flush
Shadow bytes around the buggy address:
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff8000: fa fa fd fa fa fa 00 00 fa fa 00 00 fa fa 00 00
=>0x0c047fff8010: fa fa 00 fa fa fa[06]fa fa fa 06 fa fa fa 06 fa
  0x0c047fff8020: fa fa 06 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==81142==ABORTING
[INFO] tile number 1 / 1

Affected version:
2.2.0

Fixed version:
N/A

Commit fix:
https://github.com/uclouvain/openjpeg/commit/afb308b9ccbe129608c9205cf3bb39bbefad90b9

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:

Reproducer:
https://github.com/asarubbo/poc/blob/master/00314-openjpeg-heapoverflow-opj_mqc_flush

Timeline:
2017-08-14: bug discovered and reported to upstream
2017-08-14: upstream releases a fix
2017-08-16: blog post about the issue

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

openjpeg: heap-based buffer overflow in opj_mqc_flush (mqc.c)

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Modern feed readers (August 16, 2017, 05:04 UTC)

Four years ago, I wrote a musing about the negative effects of the Google Reader shutdown on content publishers. Today I can definitely confirm that some of the problems I foretold materialized. Indeed, thanks to the fortuitous fact that people have started posting my blog articles to reddit and hacker news (neither of which I’m fond of, but let’s leave that aside), I can declare that the vast majority of the bandwidth used by my blog is consumed by bots and in particular by feed readers. But let’s start from the start.

This blog is syndicated over a feed, the URL and format of which changed a number of times before, mostly with the software, or with the update of the software. The most recent change was due to switching from Typo to Hugo, and the feed name changing. I could have kept the original feed name, but it made little sense at the time, so instead I set up permanent redirects from the old URLs to the new URLs, as I always do. I say I always do because I keep working even the original URLs from when I ran the blog off my home DSL.

Some services and feed reading software know how to deal with permanent redirects correctly, and will (eventually) replace the old feed URL with the new one. For instance NewsBlur will replace URLs after ten fetches replied with a permanent redirect (which is sensible, to avoid accepting a redirection that was set up by mistake and soon rolled back, and to avoid data poisoning attacks). Unfortunately, it seems like this behaviour is extremely rare, and so on August 14th I received over three thousands requests for the old Typo feed URL (admittedly, that was the most persistent URL I used). In addition to that, I also received over 300 requests for the very old Typo /xml/ feeds, of which 122 still pointing at my old dynamic domain, which is now pointing at the normal domain for my blog. This has been the case now for almost ten years, and yet some people still have subscription to those URLs! At least one Liferea and one Akregator pointing at those URLs.

But while NewsBlur implements sane semantics for handling permanent redirects, it is far from a perfect implementation. In particular even though I have brought this up many times, Newsblur is not actually sending If-Modified-Since or If-None-Match headers, which means it will take a copy of the feed at every request. Even though it does support compressed responses (non fetch of the feed is allowed without compressed responses), NewsBlur is requesting the same URL more than twice an hour, because it seems to have two “sites” described by the same URL. At 50KiB per request, that makes up about 1% of the total bandwidth usage of the blog. To be fair, this is not bad at all, but one has to wonder why they can’t be saving the last modified or etag values — I guess I could install my own instance of NewsBlur and figure out how to do that myself, but who knows when I would find the time for that.

Update (2017-08-16): Turns out that, as Samuel pointed out in the comments and on Twitter, I wrote something untrue. NewsBlur does send the headers, and supports this correctly. The problem is an Apache bug that causes 304 never to be issued when using If-None-Match and mod_deflate.

To be fair, even rawdog, which I use for Planet Multimedia, does not appear to support these properly. Oh and speaking of Planet Multimedia, would someone be interested in providing a more modern template so that Monty’s pictures don’t take over the page, that would be awesome!

There actually are a few other readers that do support these values correctly, and indeed receive 304 (Not Modified) status code most of the time. These include Lighting (somebody appears to be still using it!) and at least yet-another-reader-service Willreadit.com — this latter appears to be in beta and being invite only; it’s probably the best HTTP implementation I’ve seen for a service with such a rough website. Indeed the bot landing page points out how it supports If-Modified-Since and gzip-compressed responses. Alas it does not appear to learn from persistent redirects though, so it’s currently fetching my blog’s feed twice, probably because there are at least two subscribers for it.

Also note that supporting If-Modified-Since is a prerequisite for supporting delta feeds which is an interesting way to save even more bandwidth (although I don’t think this is feasible to do with a static website at all).

At the very least it looks like we won the battle for supporting compressed responses. The only 406 (Not Acceptable) responses for the feed URL are for Fever, which is no longer developed or supported. Even Gwene, which I pointed out was hammering my blog last time I wrote about this, is now content to get the compressed version. Unfortunately it does not appear like my pull request was ever merged, which means it’s likely the repository itself is completely out of sync with what is being run.

So in 2017, what is the current state of the art feed reader support? NewsBlur has recently added support for JSON Feed which is not particularly exciting – when I read the post I was reminded, by the screenshot of choice there, where I heard of Brent Simmons before: Vesper, which is an interesting connection, but I should not go into that now – but at least shows that Samuel Clay is actually paying attention to the development of the format — even though that development right now appears to just avoiding XML. Which to be honest is not that bad of an idea: since HTML (even HTML5) does not have to be well-formatted XML, you need to provide it as cdata in an XML feed. And the way you do that makes it very easy to implement it incorrectly.

Also, as I wrote this post I realized what else I would like from NewsBlur: the ability to subscribe to an OPML feed as a folder. I still subscribe to lots of Planets, even though they seem to have lost their charm, but a few people are aggregated in multiple planets and it would make sense to be able to avoid duplicate posts. If I could tell NewsBlur «I want to subscribe to this Planet, aggregate it into a folder», it would be able to tell the duplicated feeds, and mark the posts as read on all of them at the same time. Note that what I’d like is something different from just importing an OPML description of the planet! I would like for the folder to be kept in sync with the OPML feed, so that if new feeds are added, they also get added to the folder, and same for removed feeds. I should probably file that on GetSatisfaction at some point.

Europe and USA: my personal market comparison (August 16, 2017, 05:04 UTC)

While I have already announced I’m moving to London, I don’t want to give the idea that I don’t trust Europe. One of my acquaintances, an eurosceptic, thought it was apt o congratulate me for dropping out of Europe when I announced my move, but that couldn’t be farthest from my intention. As I said already repeatedly now, my decision is all to be found in my lack of a social circle in Dublin, and the feelings of loneliness that really need to be taken care of.

Indeed, I’m more than an Europeist, I’m a Globalist, in so far as I don’t see any reason why we should have borders, or limitations on travel. So my hope is not just for Europe to become a bigger, more common block. Having run a business for a number of years in Italy, where business rules are overly complicated, and the tax system assumes that you’re a cheater by default, and fines you if you don’t invoice enough, I would have seriously loved the option to have an “European business” rather than an “Italian business” — since a good chunk of my customers were based outside of Italy anyway.

This concept of “European business”, unfortunately, does not exist. Even VAT handling in Europe is not unified, and even though we should have at least a common VAT ID registration, back when I set up my business, it required an explicit registration at the Finance Ministry to be able to make use of the ID outside of Italy. At the time, at least, I think Spain also opted out to registering their VAT IDs on the European system by default. Indeed that was the reason why Amazon used to the run separate processes for most European business customers, and for Italian and Spanish customers.

Speaking of Amazon, those of you reading me from outside Europe may be surprised to know that there is no such thing as “Amazon Europe”, – heck, we don’t even have Amazon Ireland! – at least as a consumer website. Each country has its own Amazon website, with similar, but not identical listings, prices and “rules of engagement” (what can be shipped where and for which prices). For the customers this has quite a few detrimental effects: the prices may be lower in a country that they may not usually look at the store of, or they may have to weight the options based on price, shipping restrictions and shipping costs.

Since, as I said, there is no Amazon Ireland, living in Dublin also is an interesting exercise with Amazon: you may want to order things from Amazon UK, either because of language reasons, or simply because it requires a power plug and Ireland has the same British plug as the UK. And most of the shipping costs are lower, either by themselves, or because there are re-mailers from Northern Ireland to Dublin, if you are okay with waiting an extra day. But at the same time, you’re forced to pay in GBP rather than Euro (well, okay not forced, but at least strongly advised to — Amazon currency conversion has a significantly worse exchange rate than any of my cards, especially Revolut) and some of the sellers will actually refuse to send to Ireland, for no specific reason. Sometimes, you can actually buy the same device from Amazon Germany, which will then ship from a UK-based storehouse anyway, despite the item not being available to send to Ireland from Amazon UK. And sometimes Amazon Italy may be a good 15% cheaper (on a multiple-hundreds euro item) than Amazon UK.

So why does Amazon not run a global European website? Or why doesn’t an European-native alternative appears? It looks to me like the European Union and its various bodies and people keep hoping to find European-native alternatives to the big American names all the times, at least on the papers, probably in the hope of not being tied to the destiny of American with what comes down in the future, particularly given how things have gone with the current politics on all sides. But in all their words, there does not appear to be any option of opening up opportunities for creating cross-Europe collaboration on corporations.

The current situation of the countries that make up Europe and the States that make up the USA, is that you are just not allowed to do certain types, or levels of business in all the countries without registering and operating as a company in that country. That is the case for instance of phone operators, that get licenses per country, and so each operates independent units. This becomes sometimes ludicrous because you then have Vodafone providing services in about half of Europe, but with such independent units that their level of competence for instance on security and privacy is extremely variable. In particular it looks like Vodafone Italy still has not learnt how to set up HTTPS correctly, and despite logging you in a TLS-encrypted connection, it does not set the cookie as secure, so a downgrade is enough to steal authentication cookies. In 2017.

If you remember, when I complained about the half-baked roaming directive results, I have suggested that one of my favourite options would be to have a “European number” — just give me a special “country code” that can be replaced by any one member’s code, and the phone number is still valid, and appears local. This is important because, despite the roaming directive allowing me to keep my regular Irish (for now) SIM card on my phone while travelling to either UK or Finland, it prevents me from getting a local phone number. And since signing up for some local services, including sometimes free WiFi hotspots from various cafes and establishment, relies on being able to receive a local SMS, it is sometimes more of an hindrance than a favour.

Both Revolut and Transferwise, as well as other similar “FinTech” companies have started providing users with what they call “borderless” accounts: Euro, Sterling and Dollar accounts all into one system. Unfortunately this is only half of the battle. Indeed, while I welcome in particular Revolut’s option of using a single balance that can provide all the currencies in a single card is a great option. But this only works to a point, because these accounts are “special” — in particular the Revolut Euro account is provided with a Lithuanian IBAN, but a UK BIC code, which makes a few system that still expect both throw up. And this is not even going into how SEPA Direct Debit just does not work: my Italian services can only debit an Italian bank, my Irish services can only charge an Irish bank, and my one French service can only charge a French bank. Using credit cards via VISA has actually better success rate for me, even though at least Vodafone Italy can only charge a specific one of my credit cards, rather than any of them. Oh yeah and let’s not forget the fact that you just can’t get your salary paid into a non-Irish bank account in Ireland.

Banks in Europe end up operating as country-wide silos, to the point that even Ulster Bank Republic of Ireland cannot (at least, can no longer) provide me with an Ulster Bank Northern Ireland bank account — or to be precise, cannot act on my already-existing foreigner bank account that is open in Northern Ireland. And because of all these things happening, the moment I will actually move to London I’ll have to figure out how to get a proper account there. I’m having trouble right now opening an account there already not because I don’t have the local tax ID but because they need proof of employment from a UK company, while I’m still employed by the Irish company. Of the same multinational. Oh my.

You could say that banks and telcos are special cases. They are partial monopolies and there are good reasons why they should be administered on a country-by-country basis. But the reality is that in the United States, these things are mostly solved — plenty of telco stuff is still pretty much local, but that’s because of network access and antitrust requirements, as well, to a point, the need of building and servicing local infrastructure (a solution to this is effectively splitting the operation of the telco from the provider of physical infrastructure, but that comes with its own problems). But at the very least, banking in the US is not something that people have to deal with when changing State, or having to work with companies of other states.

These silos are also visible to consumers in other forms, that may not be quite obvious. TV, movie and similar rights are also a problem the same way. Netflix for instance will only show a subset of the programming they have access to depending on the country you’re currently located in. This is because, except for the content they produce themselves, they have to acquire rights from different companies holding them in different countries, because different TV networks would already have secured rights and not want to let them broadcast in their place.

I brought up this part last, despite being probably the one most consumers know or even care about, because it shows the other problem that companies trying to build up support for Europe, or even to be started as Europe-native companies, have to deal with. TV networks are significantly more fragmented than in the USA. There is no HBO, despite Sky being present in a number of different countries. There is nothing akin to CNN. There are a number of 24-hours news channels that are reachable over more-or-less freeview means, but the truth is that if you want to watch TV in Europe, you need a local company to provide you with it. And the reason is not one that is easy to solve: different countries just speak different languages, sometimes more than once.

It’s not just a matter of providing a second channel in a different language: content needs to be translated, sometimes adapted. This is very clear in video games, where some countries (cough Germany cough) require cutting content explicitly, to avoid upsetting something or someone. Indeed, video games releases for many platforms, in the past at least including PC, but luckily it appears not the case nowadays, end up distributing games only in a subset of European languages at a time. Which is why I loathed playing Skyrim on the PlayStation 3, as the disk only includes Italian, French and German, but no English, which would be my default option (okay, nowadays I would probably play it in French to freshen up my understanding of it).

For American start-ups – but this is true also for open source project, and authors of media such as books, TV series or movies – internationalization or localization are problems that can be easily shelved for the “after we’re famous” pile. First make the fame, or the money, then export and care about other languages. In Europe that cannot possibly be the case. Even for English, that in the computer world is still for now the lingua franca (pun intended), I wouldn’t expect there would be a majority of users happy to use a non-localized software, particularly when you consider as part of that localization the differences in date handling. I mean, I started using “English (UK)” rather than the default American for my Google account years ago because I wanted a sane date format in Gmail!

All of this makes the fragmented European market harder for most projects, companies, and even ideas to move as fast as the American or (I presume, but have not enough detail about it) the Chinese market, in which a much wider audience can be gained without spending so much effort to deal with cross-border bureaucracy and cross-culture porting. But let me be clear, I do not think that the solution is to normalize Europe onto a single language. We can’t even do that for countries, and I don’t think it would be fair to anyone to even consider this. What we need is to remove as many other roadblocks as it’s feasible to remove, and then try to come up with an easier way to fund translation and localization processes, or an easier way to access rights at a Union level rather than on a country-by-country basis.

Unfortunately, I do not expect that this is going to happen in my lifetime. I still wish we’ll end up with a United Federation of Planets, at the end of the day, though.

August 14, 2017

Description:
openjpeg is an open-source JPEG 2000 library.

The complete ASan output of the issue:

# opj_compress -n 1 -i $FILE -o null.j2c
==78690==ERROR: AddressSanitizer failed to allocate 0x5ea7983000 (406538694656) bytes of LargeMmapAllocator (error code: 12)
==78690==Process memory map follows:
        0x000000400000-0x0000005a6000   /usr/bin/opj_compress
        0x0000007a5000-0x0000007a6000   /usr/bin/opj_compress
        0x0000007a6000-0x0000007b0000   /usr/bin/opj_compress
        0x0000007b0000-0x000001425000
        0x00007fff7000-0x00008fff7000
        0x00008fff7000-0x02008fff7000
        0x02008fff7000-0x10007fff8000
        0x600000000000-0x602000000000
        0x602000000000-0x602000010000
        0x602000010000-0x602e00000000
        0x602e00000000-0x602e00010000
        0x602e00010000-0x604000000000
        0x604000000000-0x604000010000
        0x604000010000-0x604e00000000
        0x604e00000000-0x604e00010000
        0x604e00010000-0x606000000000
        0x606000000000-0x606000010000
        0x606000010000-0x606e00000000
        0x606e00000000-0x606e00010000
        0x606e00010000-0x610000000000
        0x610000000000-0x610000010000
        0x610000010000-0x610e00000000
        0x610e00000000-0x610e00010000
        0x610e00010000-0x616000000000
        0x616000000000-0x616000010000
        0x616000010000-0x616e00000000
        0x616e00000000-0x616e00010000
        0x616e00010000-0x621000000000
        0x621000000000-0x621000010000
        0x621000010000-0x621e00000000
        0x621e00000000-0x621e00010000
        0x621e00010000-0x640000000000
        0x640000000000-0x640000003000
        0x7f2622bf7000-0x7f2623800000
        0x7f2623900000-0x7f2623a00000
        0x7f2623a5c000-0x7f2625dae000
        0x7f2625dae000-0x7f2625e16000   /usr/lib64/libjpeg.so.62.2.0
        0x7f2625e16000-0x7f2626016000   /usr/lib64/libjpeg.so.62.2.0
        0x7f2626016000-0x7f2626017000   /usr/lib64/libjpeg.so.62.2.0
        0x7f2626017000-0x7f2626018000   /usr/lib64/libjpeg.so.62.2.0
        0x7f2626018000-0x7f2626021000   /usr/lib64/libjbig.so
        0x7f2626021000-0x7f2626220000   /usr/lib64/libjbig.so
        0x7f2626220000-0x7f2626221000   /usr/lib64/libjbig.so
        0x7f2626221000-0x7f2626224000   /usr/lib64/libjbig.so
        0x7f2626224000-0x7f2626248000   /lib64/liblzma.so.5.2.3
        0x7f2626248000-0x7f2626448000   /lib64/liblzma.so.5.2.3
        0x7f2626448000-0x7f2626449000   /lib64/liblzma.so.5.2.3
        0x7f2626449000-0x7f262644a000   /lib64/liblzma.so.5.2.3
        0x7f262644a000-0x7f2626460000   /lib64/libz.so.1.2.11
        0x7f2626460000-0x7f262665f000   /lib64/libz.so.1.2.11
        0x7f262665f000-0x7f2626660000   /lib64/libz.so.1.2.11
        0x7f2626660000-0x7f2626661000   /lib64/libz.so.1.2.11
        0x7f2626661000-0x7f26267f0000   /lib64/libc-2.23.so
        0x7f26267f0000-0x7f26269f0000   /lib64/libc-2.23.so
        0x7f26269f0000-0x7f26269f4000   /lib64/libc-2.23.so
        0x7f26269f4000-0x7f26269f6000   /lib64/libc-2.23.so
        0x7f26269f6000-0x7f26269fa000
        0x7f26269fa000-0x7f2626a10000   /usr/lib64/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1
        0x7f2626a10000-0x7f2626c0f000   /usr/lib64/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1
        0x7f2626c0f000-0x7f2626c10000   /usr/lib64/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1
        0x7f2626c10000-0x7f2626c11000   /usr/lib64/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1
        0x7f2626c11000-0x7f2626c13000   /lib64/libdl-2.23.so
        0x7f2626c13000-0x7f2626e13000   /lib64/libdl-2.23.so
        0x7f2626e13000-0x7f2626e14000   /lib64/libdl-2.23.so
        0x7f2626e14000-0x7f2626e15000   /lib64/libdl-2.23.so
        0x7f2626e15000-0x7f2626e2c000   /lib64/libpthread-2.23.so
        0x7f2626e2c000-0x7f262702b000   /lib64/libpthread-2.23.so
        0x7f262702b000-0x7f262702c000   /lib64/libpthread-2.23.so
        0x7f262702c000-0x7f262702d000   /lib64/libpthread-2.23.so
        0x7f262702d000-0x7f2627031000
        0x7f2627031000-0x7f2627037000   /lib64/librt-2.23.so
        0x7f2627037000-0x7f2627237000   /lib64/librt-2.23.so
        0x7f2627237000-0x7f2627238000   /lib64/librt-2.23.so
        0x7f2627238000-0x7f2627239000   /lib64/librt-2.23.so
        0x7f2627239000-0x7f262733b000   /lib64/libm-2.23.so
        0x7f262733b000-0x7f262753a000   /lib64/libm-2.23.so
        0x7f262753a000-0x7f262753b000   /lib64/libm-2.23.so
        0x7f262753b000-0x7f262753c000   /lib64/libm-2.23.so
        0x7f262753c000-0x7f2627591000   /usr/lib64/liblcms2.so.2.0.8
        0x7f2627591000-0x7f2627790000   /usr/lib64/liblcms2.so.2.0.8
        0x7f2627790000-0x7f2627791000   /usr/lib64/liblcms2.so.2.0.8
        0x7f2627791000-0x7f2627796000   /usr/lib64/liblcms2.so.2.0.8
        0x7f2627796000-0x7f2627809000   /usr/lib64/libtiff.so.5.2.6
        0x7f2627809000-0x7f2627a08000   /usr/lib64/libtiff.so.5.2.6
        0x7f2627a08000-0x7f2627a0c000   /usr/lib64/libtiff.so.5.2.6
        0x7f2627a0c000-0x7f2627a0d000   /usr/lib64/libtiff.so.5.2.6
        0x7f2627a0d000-0x7f2627a3f000   /usr/lib64/libpng16.so.16.29.0
        0x7f2627a3f000-0x7f2627c3e000   /usr/lib64/libpng16.so.16.29.0
        0x7f2627c3e000-0x7f2627c3f000   /usr/lib64/libpng16.so.16.29.0
        0x7f2627c3f000-0x7f2627c40000   /usr/lib64/libpng16.so.16.29.0
        0x7f2627c40000-0x7f2627da7000   /usr/lib64/libopenjp2.so.2.2.0
        0x7f2627da7000-0x7f2627fa6000   /usr/lib64/libopenjp2.so.2.2.0
        0x7f2627fa6000-0x7f2627fa9000   /usr/lib64/libopenjp2.so.2.2.0
        0x7f2627fa9000-0x7f2627fb1000   /usr/lib64/libopenjp2.so.2.2.0
        0x7f2627fb1000-0x7f2627fd5000   /lib64/ld-2.23.so
        0x7f262804a000-0x7f26281c6000
        0x7f26281c6000-0x7f26281d4000
        0x7f26281d4000-0x7f26281d5000   /lib64/ld-2.23.so
        0x7f26281d5000-0x7f26281d6000   /lib64/ld-2.23.so
        0x7f26281d6000-0x7f26281d7000
        0x7ffeff1e8000-0x7ffeff209000   [stack]
        0x7ffeff28f000-0x7ffeff291000   [vdso]
        0x7ffeff291000-0x7ffeff293000   [vvar]
        0xffffffffff600000-0xffffffffff601000   [vsyscall]
==78690==End of process memory map.
==78690==AddressSanitizer CHECK failed: /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_common.cc:120 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
    #0 0x4db60f in AsanCheckFailed /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_rtl.cc:69
    #1 0x4f6375 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_termination.cc:79
    #2 0x4e59a2 in __sanitizer::ReportMmapFailureAndDie(unsigned long, char const*, char const*, int, bool) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_common.cc:120
    #3 0x4ef2a5 in __sanitizer::MmapOrDie(unsigned long, char const*, bool) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_posix.cc:132
    #4 0x426caa in __sanitizer::LargeMmapAllocator::Allocate(__sanitizer::AllocatorStats*, unsigned long, unsigned long) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/../sanitizer_common/sanitizer_allocator_secondary.h:41
    #5 0x426caa in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64 >, __sanitizer::LargeMmapAllocator >::Allocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64 >*, unsigned long, unsigned long, bool, bool) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/../sanitizer_common/sanitizer_allocator_combined.h:70
    #6 0x426caa in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, bool) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_allocator.cc:407
    #7 0x42138d in __asan::asan_posix_memalign(void**, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*) /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_allocator.cc:815
    #8 0x4d206d in __interceptor_posix_memalign /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:144
    #9 0x7f2627d95aa4 in opj_aligned_alloc_n /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/opj_malloc.c:61:9
    #10 0x7f2627d95aa4 in opj_aligned_malloc /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/opj_malloc.c:209
    #11 0x7f2627c79d09 in opj_image_create /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/image.c:77:39
    #12 0x53437b in bmptoimage /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/convertbmp.c:768:13
    #13 0x50b635 in main /var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/opj_compress.c:1844:21
    #14 0x7f2626681680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #15 0x41bc78 in _start (/usr/bin/opj_compress+0x41bc78)

Affected version:
2.2.0

Fixed version:
N/A

Commit fix:
https://github.com/uclouvain/openjpeg/commit/baf0c1ad4572daa89caa3b12985bdd93530f0dd7

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:

Reproducer:
https://github.com/asarubbo/poc/blob/master/00315-openjpeg-memallocfailure-opj_aligned_alloc_n

Timeline:
2017-08-14: bug discovered and reported to upstream
2017-08-14: blog post about the issue

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

openjpeg: memory allocation failure in opj_aligned_alloc_n (opj_malloc.c)

August 12, 2017
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Optimizing rust (August 12, 2017, 19:16 UTC)

After the post about optimization, Kostya and many commenters (me included) discussed a bit about if there are better ways to optimize that loop without using unsafe code.

Kostya provided me with a test function and multiple implementations from him and I polished and benchmarked the whole thing.

The code

I put the code in a simple project, initially it was a simple main.rs and then it grew a little.

All it started with this function:

pub fn recombine_plane_reference(
    src: &[i16],
    sstride: usize,
    dst: &mut [u8],
    dstride: usize,
    w: usize,
    h: usize,
) {
    let mut idx0 = 0;
    let mut idx1 = w / 2;
    let mut idx2 = (h / 2) * sstride;
    let mut idx3 = idx2 + idx1;
    let mut oidx0 = 0;
    let mut oidx1 = dstride;

    for _ in 0..(h / 2) {
        for x in 0..(w / 2) {
            let p0 = src[idx0 + x];
            let p1 = src[idx1 + x];
            let p2 = src[idx2 + x];
            let p3 = src[idx3 + x];
            let s0 = p0.wrapping_add(p2);
            let d0 = p0.wrapping_sub(p2);
            let s1 = p1.wrapping_add(p3);
            let d1 = p1.wrapping_sub(p3);
            let o0 = s0.wrapping_add(s1).wrapping_add(2);
            let o1 = d0.wrapping_add(d1).wrapping_add(2);
            let o2 = s0.wrapping_sub(s1).wrapping_add(2);
            let o3 = d0.wrapping_sub(d1).wrapping_add(2);
            dst[oidx0 + x * 2 + 0] = clip8(o0.wrapping_shr(2).wrapping_add(128));
            dst[oidx0 + x * 2 + 1] = clip8(o1.wrapping_shr(2).wrapping_add(128));
            dst[oidx1 + x * 2 + 0] = clip8(o2.wrapping_shr(2).wrapping_add(128));
            dst[oidx1 + x * 2 + 1] = clip8(o3.wrapping_shr(2).wrapping_add(128));
        }
        idx0 += sstride;
        idx1 += sstride;
        idx2 += sstride;
        idx3 += sstride;
        oidx0 += dstride * 2;
        oidx1 += dstride * 2;
    }
}

Benchmark

Kostya used perf to measure the number of samples it takes over a large number of iterations, I wanted to make the benchmark a little more portable so I used the time::PreciseTime Rust provides to measure something a little more coarse, but good enough for our purposes.

We want to see if rewriting the loop using unsafe pointers or using high level iterators provides a decent speedup, no need to be overly precise.

NB: I decided to not use the bencher utility provided with nightly rust to make the code even easier to use.

+fn benchme<F>(name: &str, n: usize, mut f: F)
+    where F : FnMut() {
+    let start = PreciseTime::now();
+    for _ in 0..n {
+        f();
+    }
+    let end = PreciseTime::now();
+    println!("Runtime {} {}", name, start.to(end));
+}
# cargo run --release

Unsafe code

Both me and Kostya have a C background so for him (and for me), was sort of natural embracing unsafe {} and use the raw pointers like we are used to.

pub fn recombine_plane_unsafe(
    src: &[i16],
    sstride: usize,
    dst: &mut [u8],
    dstride: usize,
    w: usize,
    h: usize,
) {
    unsafe {
        let hw = (w / 2) as isize;
        let mut band0 = src.as_ptr();
        let mut band1 = band0.offset(hw);
        let mut band2 = band0.offset(((h / 2) * sstride) as isize);
        let mut band3 = band2.offset(hw);
        let mut dst0 = dst.as_mut_ptr();
        let mut dst1 = dst0.offset(dstride as isize);
        let hh = (h / 2) as isize;
        for _ in 0..hh {
            let mut b0_ptr = band0;
            let mut b1_ptr = band1;
            let mut b2_ptr = band2;
            let mut b3_ptr = band3;
            let mut d0_ptr = dst0;
            let mut d1_ptr = dst1;
            for _ in 0..hw {
                let p0 = *b0_ptr;
                let p1 = *b1_ptr;
                let p2 = *b2_ptr;
                let p3 = *b3_ptr;
                let s0 = p0.wrapping_add(p2);
                let s1 = p1.wrapping_add(p3);
                let d0 = p0.wrapping_sub(p2);
                let d1 = p1.wrapping_sub(p3);
                let o0 = s0.wrapping_add(s1).wrapping_add(2);
                let o1 = d0.wrapping_add(d1).wrapping_add(2);
                let o2 = s0.wrapping_sub(s1).wrapping_add(2);
                let o3 = d0.wrapping_sub(d1).wrapping_add(2);
                *d0_ptr.offset(0) = clip8((o0 >> 2).wrapping_add(128));
                *d0_ptr.offset(1) = clip8((o1 >> 2).wrapping_add(128));
                *d1_ptr.offset(0) = clip8((o2 >> 2).wrapping_add(128));
                *d1_ptr.offset(1) = clip8((o3 >> 2).wrapping_add(128));
                b0_ptr = b0_ptr.offset(1);
                b1_ptr = b1_ptr.offset(1);
                b2_ptr = b2_ptr.offset(1);
                b3_ptr = b3_ptr.offset(1);
                d0_ptr = d0_ptr.offset(2);
                d1_ptr = d1_ptr.offset(2);
            }
            band0 = band0.offset(sstride as isize);
            band1 = band1.offset(sstride as isize);
            band2 = band2.offset(sstride as isize);
            band3 = band3.offset(sstride as isize);
            dst0 = dst0.offset((dstride * 2) as isize);
            dst1 = dst1.offset((dstride * 2) as isize);
        }
    }
}

The function is faster than baseline:

    Runtime reference   PT1.598052169S
    Runtime unsafe      PT1.222646190S

Explicit upcasts

Kostya noticed that telling rust to use i32 instead of i16 gave some performance boost.

    Runtime reference       PT1.601846926S
    Runtime reference 32bit PT1.371876242S
    Runtime unsafe          PT1.223115917S
    Runtime unsafe 32bit    PT1.124667021S

I’ll keep variants between i16 and i32 to see when it is important and when it is not.

Note: Making code generic over primitive types is currently pretty painful and hopefully will be fixed in the future.

High level abstractions

Most of the comments to Kostya’s original post were about leveraging the high level abstractions to make the compiler understand the code better.

Use Iterators

Rust is able to omit the bound checks if there is a warranty that the code cannot go out of the array boundaries. Using Iterators instead of for loops over an external variables should do the trick.

Use Chunks

chunks and chunks_mut take a slice and provides a nice iterator that gets you at-most-N-sized pieces of the input slice.

Since that the code works by line it is sort of natural to use it.

Use split_at

split_at and split_at_mut get you independent slices, even mutable. The code is writing two lines at time so having the ability to access mutably two regions of the frame is a boon.

The (read-only) input is divided in bands and the output produced is 2 lines at time. split_at is much better than using hand-made slicing and
split_at_mut is perfect to write at the same time the even and the odd line.

All together

pub fn recombine_plane_chunks_32(
    src: &[i16],
    sstride: usize,
    dst: &mut [u8],
    dstride: usize,
    w: usize,
    h: usize,
) {
    let hw = w / 2;
    let hh = h / 2;
    let (src1, src2) = src.split_at(sstride * hh);
    let mut src1i = src1.chunks(sstride);
    let mut src2i = src2.chunks(sstride);
    let mut dstch = dst.chunks_mut(dstride * 2);
    for _ in 0..hh {
        let s1 = src1i.next().unwrap();
        let s2 = src2i.next().unwrap();
        let mut d = dstch.next().unwrap();
        let (mut d0, mut d1) = d.split_at_mut(dstride);
        let (b0, b1) = s1.split_at(hw);
        let (b2, b3) = s2.split_at(hw);
        let mut di0 = d0.iter_mut();
        let mut di1 = d1.iter_mut();
        let mut bi0 = b0.iter();
        let mut bi1 = b1.iter();
        let mut bi2 = b2.iter();
        let mut bi3 = b3.iter();
        for _ in 0..hw {
            let p0 = bi0.next().unwrap();
            let p1 = bi1.next().unwrap();
            let p2 = bi2.next().unwrap();
            let p3 = bi3.next().unwrap();
            recombine_core_32(*p0, *p1, *p2, *p3, &mut di0, &mut di1);
        }
    }
}

It is a good improvement over the reference baseline, but still not as fast as unsafe.

    Runtime reference       PT1.621158410S
    Runtime reference 32bit PT1.467441931S
    Runtime unsafe          PT1.226046003S
    Runtime unsafe 32bit    PT1.126615305S
    Runtime chunks          PT1.349947181S
    Runtime chunks 32bit    PT1.350027322S

Use of zip or izip

Using next().unwrap() feels clumsy and force the iterator to be explicitly mutable. The loop can be written in a nicer way using the system provided zip and the itertools-provided izip.

zip works fine for 2 iterators, then you start piling up (so, (many, (tuples, (that, (feels, lisp))))) (or (feels (lisp, '(so, many, tuples))) according to a reader). izip flattens the result so it is sort of nicers.

pub fn recombine_plane_zip_16(
    src: &[i16],
    sstride: usize,
    dst: &mut [u8],
    dstride: usize,
    w: usize,
    h: usize,
) {
    let hw = w / 2;
    let hh = h / 2;
    let (src1, src2) = src.split_at(sstride * hh);
    let src1i = src1.chunks(sstride);
    let src2i = src2.chunks(sstride);
    let mut dstch = dst.chunks_mut(dstride * 2);
    for (s1, s2) in src1i.zip(src2i) {
        let mut d = dstch.next().unwrap();
        let (mut d0, mut d1) = d.split_at_mut(dstride);
        let (b0, b1) = s1.split_at(hw);
        let (b2, b3) = s2.split_at(hw);
        let mut di0 = d0.iter_mut();
        let mut di1 = d1.iter_mut();
        let iterband = b0.iter().zip(b1.iter().zip(b2.iter().zip(b3.iter())));
        for (p0, (p1, (p2, p3))) in iterband {
            recombine_core_16(*p0, *p1, *p2, *p3, &mut di0, &mut di1);
        }
    }
}

How they would fare?

    Runtime reference        PT1.614962959S
    Runtime reference 32bit  PT1.369636641S
    Runtime unsafe           PT1.223157417S
    Runtime unsafe 32bit     PT1.125534521S
    Runtime chunks           PT1.350069795S
    Runtime chunks 32bit     PT1.381841742S
    Runtime zip              PT1.249227707S
    Runtime zip 32bit        PT1.094282423S
    Runtime izip             PT1.366320546S
    Runtime izip 32bit       PT1.208708213S

Pretty well.

Looks like izip is a little more wasteful than zip currently, so looks like we have a winner 🙂

Conclusions

  • Compared to common imperative programming patterns, using the high level abstractions does lead to a nice speedup: use iterators when you can!
  • Not all the abstractions cost zero, zip made the overall code faster while izip lead to a speed regression.
  • Do benchmark your time critical code. nightly has some facility for it BUT it is not great for micro-benchmarks.

Overall I’m enjoying a lot writing code in Rust.

August 11, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Ten Years Ago (August 11, 2017, 21:04 UTC)

At the time this entry is going to post, it would have been ten years since that fateful night, when I asked to be brought to the hospital, afraid of an ulcer. Instead of an ulcer (which I thought of because I vomited red — it turned out to be ketchup, rather than blood), they found I was nearly dying from pancreatitis.

It’s ten years, and I would say I’m still surprised I made it up to this point. As I already recounted years ago I spent the following year almost sure that I would not survive it – and indeed, an year later I ended up at the hospital again. I have been surprised two years ago that I turned 30, and decided to spend that day in Paris with my best friend. And now I’m surprised that I already survived 10 years, and thinking back to it, I would not expect I’ll be risking my life ever again, just as long as I keep away from alcohol.

It does not mean that those events ten years ago didn’t leave any sign. I don’t have a gallbladder, and my pancreas is connected to my stomach rather than intestine, that makes for an interesting acid reflux situation when I eat too much or eat something that disagrees with me. And the not-quite-clear-cut diabetes that sometimes confuses my doctors just as much as me. But all of these are now survivable problems, and I don’t live in fear of dying as much as I did.

I still haven’t managed to save up much money — I no longer spend everything I get and live day-to-day as I used to, but that’s mostly because I’m paid much better, rather than me acting like the proverbial ant. I just don’t want to plan for a future I’m not going to have, though I don’t want to go into the future with nothing.

The last ten years have been a different life for me. Before, and for the first couple of years after that, I could easily be classified as one of those “nerds in their mother’s basement” — although living in Italy my home office was in the upper floor instead. But soon after, friends who’ve been with me during the ordeal, or that I met afterwards, effectively helped me out of my shell, and out of my office.

A short, and obviously non-comprehensive list of things I did since leaving the hospital:

  • I legally changed my name — which is why you see me referenced earlier as “Diego Pettenò” rather than “Diego Elio Pettenò”. The name change had little to do with the hospital, and more to do with the the fact that three other people shared my old name just in my city.
  • I attended a number of different gaming cons in Italy, thanks to a friend who’s been trying to sell his game, including the one in Parma where I finally managed to meet Enrico.
  • I started (and closed), my own company. This may or may not have been a mistake, given the amount of taxes I ended up paying for it, but it was definitely an experience.
  • I started flying. Oh my if I did. And that was a huge change! I now live outside of Italy, attend conferences, meet my fellow developers. I spend a third of the year on the road, and can actually learn even more from people!

I have definitely not let my second chance go to waste, at least for what concerns professional opportunities and my career. From this point of view, it has also been very important, as it allowed for some other changes in my family life, but those don’t relate to me and so I don’t want to go much down that road.

My social life instead, well, that has not really changed much since those days. It clearly improved, as I’m no longer spending all my time in my home office, but the last few years while in Dublin it also shrunk down again. Which is why I’ll be moving to London in just a couple of months, hoping to find more people to share time with, whether it is playing board games or watching movies or anime at home.

The feeling I get now is realizing that I wasted a significant amount of my adult life without a social life, and without caring for my health enough to avoid these situations. So if I were to give a suggestion to other geeks who may now be in their teens, and think that it’s cool to be a shut-in, is to try to find some friends. You don’t have to be an extrovert for that. You can still be an introvert and like some people. And the people who you get around yourself are the people who will sustain you when you fall. And everybody will fall at some point.

As for the future. As I said previously, I’m moving to London in a couple of months. I hope there, between a number of people I already know, and the different scale that the city has, compare to Dublin, I will manage to build another group of friends to go around with. I would be a hypocrite to not admit that I also would like to meet someone special to share life with — but I’m also not counting on that to happen, as I probably am still too socially awkward for that to be possible.

August 10, 2017

Description:
imagemagick is a software suite to create, edit, compose, or convert bitmap images.

The complete ASan output of the issue:

# convert $FILE null
==109188==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000049f8 at pc 0x7f81ecd9b4c2 bp 0x7ffe3c52f850 sp 0x7ffe3c52f848                                                                        
READ of size 8 at 0x6020000049f8 thread T0                                                                                                                                                                        
    #0 0x7f81ecd9b4c1 in .omp_outlined..32 /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/enhance.c:1248:13                                                                   
    #1 0x7f81eb8395c2 in __kmp_invoke_microtask /var/tmp/portage/sys-libs/libomp-4.0.1/work/openmp-4.0.1.src/runtime/src/z_Linux_asm.s:1399                                                                       
    #2 0x7f81eb7e125a in __kmp_fork_call /var/tmp/portage/sys-libs/libomp-4.0.1/work/openmp-4.0.1.src/runtime/src/kmp_runtime.cpp:1858                                                                            
    #3 0x7f81eb7cd74f in __kmpc_fork_call /var/tmp/portage/sys-libs/libomp-4.0.1/work/openmp-4.0.1.src/runtime/src/kmp_csupport.cpp:337                                                                           
    #4 0x7f81ecd999b9 in ContrastStretchImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/enhance.c:1213:11                                                                
    #5 0x7f81ecbd2280 in SetImageType /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/attribute.c:1262:18                                                                      
    #6 0x7f81e5acd5bd in WriteTIFFImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/coders/tiff.c:3245:16                                                                             
    #7 0x7f81eccc4026 in WriteImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:1114:14                                                                       
    #8 0x7f81eccc55a9 in WriteImages /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:1333:13                                                                      
    #9 0x7f81ec50f456 in ConvertImageCommand /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/convert.c:3280:11                                                                 
    #10 0x7f81ec62e225 in MagickCommandGenesis /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/mogrify.c:183:14                                                                
    #11 0x5093e9 in MagickMain /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:149:10                                                                                  
    #12 0x5093e9 in main /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:180                                                                                           
    #13 0x7f81eb206680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289                                                                                   
    #14 0x41a1f8 in _init (/usr/bin/magick+0x41a1f8)                                                                                                                                                              
                                                                                                                                                                                                                  
0x6020000049f8 is located 0 bytes to the right of 8-byte region [0x6020000049f0,0x6020000049f8)                                                                                                                   
allocated by thread T0 here:                                                                                                                                                                                      
    #0 0x4cfba8 in malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:66                                                                      
    #1 0x7f81ecef8df7 in AcquireMagickMemory /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/memory.c:464:10                                                                   
    #2 0x7f81ecef8df7 in AcquireQuantumMemory /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/memory.c:537                                                                     
    #3 0x7f81ecd97037 in ContrastStretchImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/enhance.c:1052:20                                                                
    #4 0x7f81ecbd2280 in SetImageType /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/attribute.c:1262:18
    #5 0x7f81e5acd5bd in WriteTIFFImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/coders/tiff.c:3245:16
    #6 0x7f81eccc4026 in WriteImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:1114:14
    #7 0x7f81eccc55a9 in WriteImages /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:1333:13
    #8 0x7f81ec50f456 in ConvertImageCommand /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/convert.c:3280:11
    #9 0x7f81ec62e225 in MagickCommandGenesis /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/mogrify.c:183:14
    #10 0x5093e9 in MagickMain /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:149:10
    #11 0x5093e9 in main /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:180
    #12 0x7f81eb206680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/enhance.c:1248:13 in .omp_outlined..32
Shadow bytes around the buggy address:
  0x0c047fff88e0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c047fff88f0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c047fff8900: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c047fff8910: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c047fff8920: fa fa 00 03 fa fa 00 04 fa fa 05 fa fa fa 00 00
=>0x0c047fff8930: fa fa 00 07 fa fa 00 04 fa fa 00 04 fa fa 00[fa]
  0x0c047fff8940: fa fa 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8970: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==109188==ABORTING

Affected version:
7.0.6-5

Fixed version:
7.0.6-6 (not released atm)

Commit fix:
https://github.com/ImageMagick/ImageMagick/commit/1cc6f0ccc92c20c7cab6c4a7335daf29c91f0d8e

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12876

Reproducer:
https://github.com/asarubbo/poc/blob/master/00306-imagemagick-heapoverflow-enhance_c

Timeline:
2017-08-09: bug discovered and reported to upstream
2017-08-10: blog post about the issue
2017-08-15: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

imagemagick: heap-based buffer overflow in .omp_outlined..32 (enhance.c)

Description:
imagemagick is a software suite to create, edit, compose, or convert bitmap images.

The complete ASan output of the issue:

# convert $FILE null
==151587==ERROR: AddressSanitizer: heap-use-after-free on address 0x627000037d50 at pc 0x7f4697f94380 bp 0x7ffd1011d370 sp 0x7ffd1011d368
READ of size 8 at 0x627000037d50 thread T0
    #0 0x7f4697f9437f in DestroyImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/image.c:1186:3
    #1 0x7f4690bfaebf in ReadMATImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/coders/mat.c:1374:14
    #2 0x7f4697dc8844 in ReadImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:497:13
    #3 0x7f4697dcbf01 in ReadImages /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:866:9
    #4 0x7f469760e319 in ConvertImageCommand /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/convert.c:641:18
    #5 0x7f4697737225 in MagickCommandGenesis /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/mogrify.c:183:14
    #6 0x5093e9 in MagickMain /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:149:10
    #7 0x5093e9 in main /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:180
    #8 0x7f469630f680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #9 0x41a1f8 in _init (/usr/bin/magick+0x41a1f8)

0x627000037d50 is located 13392 bytes inside of 13488-byte region [0x627000034900,0x627000037db0)
freed by thread T0 here:
    #0 0x4cf9f0 in __interceptor_cfree /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:55
    #1 0x7f4698003b8a in RelinquishMagickMemory /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/memory.c:1042:3
    #2 0x7f4697f942ed in DestroyImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/image.c:1221:19
    #3 0x7f4697fd6454 in DeleteImageFromList /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/list.c:298:12
    #4 0x7f4690bfab12 in ReadMATImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/coders/mat.c:1344:11
    #5 0x7f4697dc8844 in ReadImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:497:13
    #6 0x7f4697dcbf01 in ReadImages /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:866:9
    #7 0x7f469760e319 in ConvertImageCommand /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/convert.c:641:18
    #8 0x7f4697737225 in MagickCommandGenesis /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/mogrify.c:183:14
    #9 0x5093e9 in MagickMain /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:149:10
    #10 0x5093e9 in main /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:180
    #11 0x7f469630f680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289

previously allocated by thread T0 here:
    #0 0x4cfba8 in malloc /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:66
    #1 0x7f4697f8c474 in AcquireImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/image.c:169:19
    #2 0x7f4697f9026b in AcquireNextImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/image.c:395:15
    #3 0x7f4690bfd5ad in ReadMATImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/coders/mat.c:1284:5
    #4 0x7f4697dc8844 in ReadImage /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:497:13
    #5 0x7f4697dcbf01 in ReadImages /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/constitute.c:866:9
    #6 0x7f469760e319 in ConvertImageCommand /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/convert.c:641:18
    #7 0x7f4697737225 in MagickCommandGenesis /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickWand/mogrify.c:183:14
    #8 0x5093e9 in MagickMain /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:149:10
    #9 0x5093e9 in main /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/utilities/magick.c:180
    #10 0x7f469630f680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289

SUMMARY: AddressSanitizer: heap-use-after-free /var/tmp/portage/media-gfx/imagemagick-7.0.6.5/work/ImageMagick-7.0.6-5/MagickCore/image.c:1186:3 in DestroyImage
Shadow bytes around the buggy address:
  0x0c4e7fffef50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4e7fffef60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4e7fffef70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4e7fffef80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4e7fffef90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c4e7fffefa0: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x0c4e7fffefb0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
  0x0c4e7fffefc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4e7fffefd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4e7fffefe0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4e7fffeff0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==151587==ABORTING

Affected version:
7.0.6-5

Fixed version:
7.0.6-6 (not released atm)

Commit fix:
https://github.com/ImageMagick/ImageMagick/commit/04178de2247e353fc095846784b9a10fefdbf890

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12877

Reproducer:
https://github.com/asarubbo/poc/blob/master/00305-imagemagick-UAF-DestroyImage

Timeline:
2017-08-09: bug discovered and reported to upstream
2017-08-10: blog post about the issue
2017-08-15: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

imagemagick: use-after-free in DestroyImage (image.c)

August 09, 2017

Description:
libfpx is a library for manipulating FlashPIX images.

I’m aware that the link to the upstream website does not work. I’m keeping it as well because in the future the upstream website could appear again.
Libfpx is not actively developed, I contacted the imagemagick project if they were available to patch security issues, but they said the they are only accepting patches and push new releases.
This issue was found using the gm command line tool of graphicsmagick.

The complete ASan output of the issue:

# gm identify $FILE
==11203==ERROR: AddressSanitizer: FPE on unknown address 0x7fc9f8a8a403 (pc 0x7fc9f8a8a403 bp 0x7fffbf287b28 sp 0x7fffbf287ae0 T0)
    #0 0x7fc9f8a8a402 in CDirVector::GetTable(unsigned int, unsigned int, CDirSect**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/h/dirfunc.hxx:250
    #1 0x7fc9f8a8a402 in CDirectory::GetDirEntry(unsigned int, unsigned int, CDirEntry**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/dir.cxx:1102
    #2 0x7fc9f8a91cff in CDirectory::GetSize(unsigned int, unsigned int*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/h/dirfunc.hxx:316
    #3 0x7fc9f8a91cff in CMStream::Init() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/mstream.cxx:431
    #4 0x7fc9f8a912e7 in DllMultiStreamFromStream(CMStream**, ILockBytes**, unsigned int) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/msf.cxx:88
    #5 0x7fc9f8a9388b in CRootExposedDocFile::InitRoot(ILockBytes*, unsigned int, unsigned short, unsigned short**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/rexpdf.cxx:124
    #6 0x7fc9f8a8bad6 in DfFromLB(ILockBytes*, unsigned short, unsigned int, unsigned short**, CExposedDocFile**, _XGUID*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/docfile.cxx:66
    #7 0x7fc9f8a8bdfc in DfOpenStorageOnILockBytesW(ILockBytes*, IStorage*, unsigned int, unsigned short**, unsigned int, IStorage**, _XGUID*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/docfile.cxx:277
    #8 0x7fc9f8a88878 in DfOpenStorageOnILockBytes(ILockBytes*, IStorage*, unsigned int, char**, unsigned int, IStorage**, _XGUID*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/ascii.cxx:461
    #9 0x7fc9f8a9458e in StgOpenStorageOnILockBytes /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/storage.cxx:116
    #10 0x7fc9f8a9461a in StgOpenStorage /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/storage.cxx:70
    #11 0x7fc9f8a7008e in OLEFile::OpenOLEFile(_XGUID&, OLEStorage**, unsigned int) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/olefiles.cpp:184
    #12 0x7fc9f8a70557 in OLEFile::GetCLSID(_XGUID*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/olefiles.cpp:346
    #13 0x7fc9f8a52d64 in PFlashPixImageView::PFlashPixImageView(FicNom&, char const*, mode_Ouverture, long, PSearchHookObject*, FPXStatus*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:389
    #14 0x7fc9f8a55c81 in OpenImageByFilename(FicNom&, char const*, unsigned long, unsigned int*, unsigned int*, unsigned int*, unsigned int*, FPXColorspace*, PFlashPixImageView**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:1629
    #15 0x7fc9f8a55dc9 in FPX_OpenImageByFilename /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:1686
    #16 0x7fc9f8cc45e6 in ReadFPXImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/coders/fpx.c:226:16
    #17 0x7fc9fe564e2b in ReadImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1607:13
    #18 0x7fc9fe561e8c in PingImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1370:9
    #19 0x7fc9fe42dae5 in IdentifyImageCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8379:17
    #20 0x7fc9fe434065 in MagickCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8869:17
    #21 0x7fc9fe4df7fb in GMCommandSingle /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17396:10
    #22 0x7fc9fe4dc931 in GMCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17449:16
    #23 0x7fc9fcd47680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #24 0x419cd8 in _init (/usr/bin/gm+0x419cd8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: FPE /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/h/dirfunc.hxx:250 in CDirVector::GetTable(unsigned int, unsigned int, CDirSect**)
==11203==ABORTING

Affected version:
1.3.1_p6

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12924

Reproducer:
https://github.com/asarubbo/poc/blob/master/00313-libfpx-FPE-CDirVector_GetTable

Timeline:
2017-08-01: bug discovered
2017-08-09: blog post about the issue
2017-08-17: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

libfpx: divide-by-zero in CDirVector::GetTable (dirfunc.hxx)

Description:
libfpx is a library for manipulating FlashPIX images.

I’m aware that the link to the upstream website does not work. I’m keeping it as well because in the future the upstream website could appear again.
Libfpx is not actively developed, I contacted the imagemagick project if they were available to patch security issues, but they said the they are only accepting patches and push new releases.
This issue was found using the gm command line tool of graphicsmagick.

The complete ASan output of the issue:

# gm identify $FILE
==11182==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fbac56ca5f6 bp 0x7ffc56dee420 sp 0x7ffc56dedba8 T0)
==11182==The signal is caused by a READ memory access.
==11182==Hint: address points to the zero page.
    #0 0x7fbac56ca5f5 in strlen /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/string/../sysdeps/x86_64/strlen.S:76
    #1 0x43ea3c in __interceptor_strlen /var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:282
    #2 0x7fbac1376493 in OLEStream::WriteVT_LPSTR(char*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/olestrm.cpp:1472
    #3 0x7fbac1372e06 in OLEPropertySection::Write() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/oleprops.cpp:477
    #4 0x7fbac1373101 in OLEPropertySet::Commit() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/oleprops.cpp:131
    #5 0x7fbac134da36 in PFlashPixFile::Commit() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxformt.cpp:581
    #6 0x7fbac134da8f in PFlashPixFile::~PFlashPixFile() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxformt.cpp:276
    #7 0x7fbac134db78 in PFlashPixFile::~PFlashPixFile() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxformt.cpp:306
    #8 0x7fbac1379ed3 in PHierarchicalImage::~PHierarchicalImage() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ri_image/ph_image.cpp:168
    #9 0x7fbac1349c38 in PFileFlashPixIO::~PFileFlashPixIO() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxio.cpp:277
    #10 0x7fbac13536a5 in PFlashPixImageView::~PFlashPixImageView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:519
    #11 0x7fbac13536b8 in PFlashPixImageView::~PFlashPixImageView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:532
    #12 0x7fbac135529e in FPX_CloseImage /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:766
    #13 0x7fbac15c7bf4 in ReadFPXImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/coders/fpx.c:344:14
    #14 0x7fbac6e89e2b in ReadImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1607:13
    #15 0x7fbac6e86e8c in PingImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1370:9
    #16 0x7fbac6d52ae5 in IdentifyImageCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8379:17
    #17 0x7fbac6d59065 in MagickCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8869:17
    #18 0x7fbac6e047fb in GMCommandSingle /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17396:10
    #19 0x7fbac6e01931 in GMCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17449:16
    #20 0x7fbac566c680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #21 0x419cd8 in _init (/usr/bin/gm+0x419cd8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/string/../sysdeps/x86_64/strlen.S:76 in strlen
==11182==ABORTING

Affected version:
1.3.1_p6

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12923

Reproducer:
https://github.com/asarubbo/poc/blob/master/00312-libfpx-NULLptr-OLEStream_WriteVT_LPSTR

Timeline:
2017-08-01: bug discovered
2017-08-09: blog post about the issue
2017-08-17: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

libfpx: NULL pointer dereference in OLEStream::WriteVT_LPSTR (olestrm.cpp)

Description:
libfpx is a library for manipulating FlashPIX images.

I’m aware that the link to the upstream website does not work. I’m keeping it as well because in the future the upstream website could appear again.
Libfpx is not actively developed, I contacted the imagemagick project if they were available to patch security issues, but they said the they are only accepting patches and push new releases.
This issue was found using the gm command line tool of graphicsmagick.

The complete ASan output of the issue:

# gm identify $FILE
==11430==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fc529a4a4e7 bp 0x000000000001 sp 0x7ffefe672888 T0)
==11430==The signal is caused by a READ memory access.
==11430==Hint: address points to the zero page.
    #0 0x7fc529a4a4e6 in PFileFlashPixView::GetGlobalInfoProperty(unsigned int, OLEProperty**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:791
    #1 0x7fc529a4b40f in PFileFlashPixView::Init() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:293
    #2 0x7fc529a4bde9 in PFileFlashPixView::PFileFlashPixView(FicNom&, char const*, mode_Ouverture, unsigned int) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:121
    #3 0x7fc529a52e92 in PFlashPixImageView::PFlashPixImageView(FicNom&, char const*, mode_Ouverture, long, PSearchHookObject*, FPXStatus*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:405
    #4 0x7fc529a55c81 in OpenImageByFilename(FicNom&, char const*, unsigned long, unsigned int*, unsigned int*, unsigned int*, unsigned int*, FPXColorspace*, PFlashPixImageView**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:1629
    #5 0x7fc529a55dc9 in FPX_OpenImageByFilename /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:1686
    #6 0x7fc529cc45e6 in ReadFPXImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/coders/fpx.c:226:16
    #7 0x7fc52f599e2b in ReadImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1607:13
    #8 0x7fc52f596e8c in PingImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1370:9
    #9 0x7fc52f462ae5 in IdentifyImageCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8379:17
    #10 0x7fc52f469065 in MagickCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8869:17
    #11 0x7fc52f5147fb in GMCommandSingle /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17396:10
    #12 0x7fc52f511931 in GMCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17449:16
    #13 0x7fc52dd7c680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #14 0x419cd8 in _init (/usr/bin/gm+0x419cd8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:791 in PFileFlashPixView::GetGlobalInfoProperty(unsigned int, OLEProperty**)
==11430==ABORTING

Affected version:
1.3.1_p6

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12921

Reproducer:
https://github.com/asarubbo/poc/blob/master/00311-libfpx-NULLptr-PFileFlashPixView_GetGlobalInfoProperty

Timeline:
2017-08-01: bug discovered
2017-08-09: blog post about the issue
2017-08-17: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

libfpx: NULL pointer dereference in PFileFlashPixView::GetGlobalInfoProperty (f_fpxvw.cpp)

libfpx: NULL pointer dereference in wchar.c (August 09, 2017, 17:42 UTC)

Description:
libfpx is a library for manipulating FlashPIX images.

I’m aware that the link to the upstream website does not work. I’m keeping it as well because in the future the upstream website could appear again.
Libfpx is not actively developed, I contacted the imagemagick project if they were available to patch security issues, but they said the they are only accepting patches and push new releases.
This issue was found using the gm command line tool of graphicsmagick.

The complete ASan output of the issue:

# gm identify $FILE
==11400==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fdead094f30 bp 0x608000000fa0 sp 0x7fff5867d3a8 T0)
==11400==The signal is caused by a READ memory access.
==11400==Hint: address points to the zero page.
    #0 0x7fdead094f2f  /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/wchar.c:140
    #1 0x7fdead0765db in OLEStream::WriteVT_LPWSTR(unsigned short*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/olestrm.cpp:1538
    #2 0x7fdead072e06 in OLEPropertySection::Write() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/oleprops.cpp:477
    #3 0x7fdead073101 in OLEPropertySet::Commit() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/oleprops.cpp:131
    #4 0x7fdead049f16 in PFileFlashPixView::Commit() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:583
    #5 0x7fdead049fbf in PFileFlashPixView::~PFileFlashPixView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:444
    #6 0x7fdead04a0c8 in PFileFlashPixView::~PFileFlashPixView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxvw.cpp:487
    #7 0x7fdead05365c in PFlashPixImageView::~PFlashPixImageView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:524
    #8 0x7fdead0536b8 in PFlashPixImageView::~PFlashPixImageView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:532
    #9 0x7fdead05529e in FPX_CloseImage /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:766
    #10 0x7fdead2c7bf4 in ReadFPXImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/coders/fpx.c:344:14
    #11 0x7fdeb2b5be2b in ReadImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1607:13
    #12 0x7fdeb2b58e8c in PingImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1370:9
    #13 0x7fdeb2a24ae5 in IdentifyImageCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8379:17
    #14 0x7fdeb2a2b065 in MagickCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8869:17
    #15 0x7fdeb2ad67fb in GMCommandSingle /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17396:10
    #16 0x7fdeb2ad3931 in GMCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17449:16
    #17 0x7fdeb133e680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #18 0x419cd8 in _init (/usr/bin/gm+0x419cd8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/wchar.c:140 
==11400==ABORTING

Affected version:
1.3.1_p6

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12922

Reproducer:
https://github.com/asarubbo/poc/blob/master/00310-libfpx-NULLptr-wchar_c

Timeline:
2017-08-01: bug discovered
2017-08-09: blog post about the issue
2017-08-17: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

libfpx: NULL pointer dereference in wchar.c

Description:
libfpx is a library for manipulating FlashPIX images.

I’m aware that the link to the upstream website does not work. I’m keeping it as well because in the future the upstream website could appear again.
Libfpx is not actively developed, I contacted the imagemagick project if they were available to patch security issues, but they said the they are only accepting patches and push new releases.
This issue was found using the gm command line tool of graphicsmagick.

The complete ASan output of the issue:

# gm identify $FILE
ASAN:DEADLYSIGNAL
=================================================================
==11276==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000258 (pc 0x7f508f88a3f5 bp 0x000000000007 sp 0x7fffbde029f0 T0)
==11276==The signal is caused by a READ memory access.
==11276==Hint: address points to the zero page.
    #0 0x7f508f88a3f4 in CDirectory::GetDirEntry(unsigned int, unsigned int, CDirEntry**) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/dir.cxx:1097
    #1 0x7f508f88a64e in CDirectory::SetSize(unsigned int, unsigned int) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/dir.cxx:437
    #2 0x7f508f89440a in CDirectStream::WriteAt(unsigned int, void const*, unsigned int, unsigned int*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/sstream.cxx:367
    #3 0x7f508f88e7cd in CExposedStream::Write(void const*, unsigned int, unsigned int*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/expst.cxx:206
    #4 0x7f508f875a3e in OLEStream::Write(void const*, unsigned long) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/olestrm.cpp:143
    #5 0x7f508f875119 in OLEStream::WriteVT_I4(unsigned int*) /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/olestrm.cpp:1370
    #6 0x7f508f872ded in OLEPropertySection::Write() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/oleprops.cpp:476
    #7 0x7f508f873101 in OLEPropertySet::Commit() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ole/oleprops.cpp:131
    #8 0x7f508f84da36 in PFlashPixFile::Commit() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxformt.cpp:581
    #9 0x7f508f84da8f in PFlashPixFile::~PFlashPixFile() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxformt.cpp:276
    #10 0x7f508f84db78 in PFlashPixFile::~PFlashPixFile() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxformt.cpp:306
    #11 0x7f508f879ed3 in PHierarchicalImage::~PHierarchicalImage() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/ri_image/ph_image.cpp:168
    #12 0x7f508f849c38 in PFileFlashPixIO::~PFileFlashPixIO() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/f_fpxio.cpp:277
    #13 0x7f508f8536a5 in PFlashPixImageView::~PFlashPixImageView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:519
    #14 0x7f508f8536b8 in PFlashPixImageView::~PFlashPixImageView() /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpximgvw.cpp:532
    #15 0x7f508f85529e in FPX_CloseImage /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/fpx/fpxlibio.cpp:766
    #16 0x7f508fac7bf4 in ReadFPXImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/coders/fpx.c:344:14
    #17 0x7f5095333e2b in ReadImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1607:13
    #18 0x7f5095330e8c in PingImage /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/constitute.c:1370:9
    #19 0x7f50951fcae5 in IdentifyImageCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8379:17
    #20 0x7f5095203065 in MagickCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:8869:17
    #21 0x7f50952ae7fb in GMCommandSingle /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17396:10
    #22 0x7f50952ab931 in GMCommand /var/tmp/portage/media-gfx/graphicsmagick-1.3.26/work/GraphicsMagick-1.3.26/magick/command.c:17449:16
    #23 0x7f5093b16680 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #24 0x419cd8 in _init (/usr/bin/gm+0x419cd8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /var/tmp/portage/media-libs/libfpx-1.3.1_p6/work/libfpx-1.3.1-6/oless/dir.cxx:1097 in CDirectory::GetDirEntry(unsigned int, unsigned int, CDirEntry**)
==11276==ABORTING

Affected version:
1.3.1_p6

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12920

Reproducer:
https://github.com/asarubbo/poc/blob/master/00308-libfpx-NULLptr-CDirectory_GetDirEntry

Timeline:
2017-08-01: bug discovered
2017-08-09: blog post about the issue
2017-08-17: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core Infrastructure Initiative.

Permalink:

libfpx: NULL pointer dereference in CDirectory::GetDirEntry (dir.cxx)

August 08, 2017
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
ScyllaDB meets Gentoo Linux (August 08, 2017, 14:19 UTC)

I am happy to announce that my work on packaging ScyllaDB for Gentoo Linux is complete!

Happy or curious users are very welcome to share their thoughts and ping me to get it into portage (which will very likely happen).

Why Scylla?

Ever heard of the Cassandra NoSQL database and Java GC/Heap space problems?… if you do, you already get it 😉

I will not go into the details as their website does this way better than me but I got interested into Scylla because it fits the Gentoo Linux philosophy very well. If you remember my writing about packaging Rethinkdb for Gentoo Linux, I think that we have a great match with Scylla as well!

  • it is written in C++ so it plays very well with emerge
  • the code quality is so great that building it does not require heavy patching on the ebuild (feels good to be a packager)
  • the code relies on system libs instead of bundling them in the sources (hurrah!)
  • performance tuning is handled by smart scripting and automation, allowing the relationship between the project and the hardware is strong

I believe that these are good enough points to go further and that such a project can benefit from a source based distribution like Gentoo Linux. Of course compiling on multiple systems is a challenge for such a database but one does not improve by staying in their comfort zone.

Upstream & contributions

Packaging is a great excuse to get to know the source code of a project but more importantly the people behind it.

So here I got to my first contributions to Scylla to get Gentoo Linux as a detected and supported Linux distribution in the different scripts and tools used to automatically setup the machine it will run upon (fear not, I contributed bash & python, not C++)…

Even if I expected to contribute using Github PRs and got to change my habits to a git-patch+mailing list combo, I got warmly welcomed and received positive and genuine interest in the contributions. They got merged quickly and thanks to them you can install and experience Scylla in Gentoo Linux without heavy patching on our side.

Special shout out to Pekka, Avi and Vlad for their welcoming and insightful code reviews!

I’ve some open contributions about pushing further on the python code QA side to get the tools to a higher level of coding standards. Seeing how upstream is serious about this I have faith that it will get merged and a good base for other contributions.

Last note about reaching them is that I am a bit sad that they’re not using IRC freenode to communicate (I instinctively joined #scylla and found myself alone) but they’re on Slack (those “modern folks”) and pretty responsive to the mailing lists 😉

Java & Scylla

Even if scylla is a rewrite of Cassandra in C++, the project still relies on some external tools used by the Cassandra community which are written in Java.

When you install the scylla package on Gentoo, you will see that those two packages are Java based dependencies:

  • app-admin/scylla-tools
  • app-admin/scylla-jmx

It pained me a lot to package those (thanks to help of @monsieurp) but they are building and working as expected so this gets the packaging of the whole Scylla project pretty solid.

emerge dev-db/scylla

The scylla packages are located in the ultrabug overlay for now until I test them even more and ultimately put them in production. Then they’ll surely reach the portage tree with the approval of the Gentoo java team for the app-admin/ packages listed above.

I provide a live ebuild (scylla-9999 with no keywords) and ebuilds for the latest major version (2.0_rc1 at time of writing).

It’s as simple as:

$ sudo layman -a ultrabug
$ sudo emerge -a dev-db/scylla
$ sudo emerge --config dev-db/scylla

Try it out and tell me what you think, I hope you’ll start considering and using this awesome database!

August 06, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)

Update: I moved to disroot.org now.

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
IPv6 Horror Story: Telegram (August 06, 2017, 12:04 UTC)

This starts to become an interesting series of blog posts, my ranting about badly implemented IPv6. Some people may suggest that this is because I have a problem with IPv6, but the reality is that I like IPv6 and I want more of it, and properly implemented.

IPv6 enthusiasts often forget that implementing IPv6 is not just a matter of having a routable address, and being able to connect the IPv6 network. Indeed, the network layer is just one of the many layers that need to be updated for your applications to be IPv6-compatible, and that is not even going into the problem of optimizing for IPv6.

Screenshot of Telegram from Android declaring my IP as 10.160.34.127.

To the right you can see the screenshot of Telegram messenger on my personal Android phone – I use Telegram to keep in touch with just a handful of people, of course – once I log in on Telegram from my laptop on the web version.

The text of the message I receive on Android at that login is

Diego Elio,

We detected a login into your account from a new device on 06/08/2017 at 10:46:58 UTC.

Device: Web Location: Unknown (IP = 10.160.34.127)

If this wasn’t you, you can go to Settings - Privacy and Security - Sessions and terminate that session.

If you think that somebody logged in to your account against your will, you can enable two-step verification in Privacy and Security settings.

Sincerely, The Telegram Team

This may not sound as obviously wrong to you as it did to me. Never mind the fact that the IP is unknown. 10.160.34.127 is a private network address, as all the IPv4 addresses starting with 10. Of course when I noticed that the first thing I did was checking whether web.telegram.org published a AAAA (IPv6) record, and of course it does.

A quick check using TunnelBear (that does not support IPv6) also shows that the message is correct when logging in from an IPv4 connection instead, showing the public egress IP used by the connection.

I reported this problem to Telegram well over a month ago and it’s still unfixed. I don’t think this count as a huge security gap, as the message still provides some level of information on new login, it does remove the ability to ensure the connection is direct from the egress you expect. Say for instance that your connection is being VPN’d to an adversarial network, you may notice that your connecting IP is appearing as from a different country than you’re in, and you know there’s a problem. When using IPv6, Telegram is removing this validation, because they not only not show you a location, but they give you a clearly wrong IPv4 in place of the actual IPv6.

Since I have no insight of what the Telegram internal infrastructure looks like, I cannot say exactly which part of the pipeline is failing for them. I would expect that this is not a problem with NAT64, because that does not help to receive inbound connections — I find it more likely it’s either the web frontend looking for an IPv4, not finding it, and having some sort of default (maybe to its own IP address) to pass along to the next service, or alternatively the service that is meant to send the message (or possibly the one that tries to geocode the address) that mis-parses the IPv6 into an IPv4, and discard the right address.

Whatever the problem on their side is, it makes a very good example of how IPv6 is not as straightforward to implement as many enthusiasts who have never looked into converting a complicated app stack would think. And documenting this here, is my way to remind people that these things are more complicated than they appear, and that it will take still a significant amount of time to fix all these problems, and until then it’s unreasonable to expect consumers to deal with IPv6-only networks.

Screenshot of Telegram from Android declaring three open sessions for 10.160.34.127.

Update (2017-08-07): a helpful (for once) comment on a reddit thread over this pointed me at the fact that even the Active Sessions page is also showing addresses in the private ten-dot space — as you can see now on the left here. It also shows that my Android session is coming (correctly) from my ISP’s CGNAT endpoint IPv4.

What this tells us is that not only there is a problem with the geocoding of an IP address that Telegram notifies the user on connection, but as long as the connection is proxied over IPv6, the user is not able to tell whether the connection was hijacked or not. If the source IP address and location was important enough to add in the first place, it feels like it should be important enough to get right. Although it may be just wait to be pushed and fixed, as the commenter on Reddit pointed out they could see the right IPv6 addresses nowadays — I can’t.

Also, I can now discard the theory that they may be parsing an IPv6 as if it was IPv4 and mis-rendering it. Of the three connections listed, one is from my office that uses an entirely different IPv6 prefix from the one at my apartment. So there.

The fact that my Android session appears on v4, despite my phone being connected to a working IPv6 network also tells me that the Telegram endpoint for Android (that is, their main endpoint) is not actually compatible with IPv6, which makes it even the more surprising that they would actually publish a record for it for their web endpoint, and show off the amount of missing details in their implementation.

I would also close the update by taking a look at one of the comments from that reddit thread:

In theory, could be that telegram is actually accepting ipv6 on their internet facing port and translating it to v4 inside the network. I’d change my opinion but given the lack of information like the IP address you had on your machine or the range you use for your network, I really don’t see enough to say this is an issue with ipv6 implementation.

Ignore the part where the commenter thinks the fact I did not want to disclose my network setup is any relevant, which appears to imply I’m seeing my own IP space reflected there (I’m not) like it was something that was even possible, and focus on the last part. «I really don’t see enough to say this is an issue with ipv6 implementation.»

This is an issue with Telegram’s IPv6 implementation. Whether they are doing network translation at the edge and then miss a X-Forwarded-For, or one of their systems defaults to a local IP if they can’t parse the remote one (as it’s in a different format), it does not really have to matter. An IPv6 implementation does not stop at the TCP level, it requires changes at application level just fine. I have unfortunately met multiple network people who think that as long as an app connects and serves data, IPv6 is supported — for an user, this is completely bollocks because they want to be able to use the app, use the app correctly, and use the app safely. Right now that does not appear to be the case for Telegram.

August 03, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
In defence of ads over subscriptions (August 03, 2017, 01:04 UTC)

This is a third draft trying to focus my opinion on one facet of a very complicated, multi-faceted discussion about Internet, content and monetization. Hopefully, I’ll be able to keep my mind focused enough on the problem I want to talk about to come out with a concise and interesting post.

You may have read my opinion on Internet ads and the more recent commentary on my own “monetization”. If you haven’t, the short version of the story is that I do not think that ads are implicitly evil, but at the same time I don’t think that they are worth the attrition for most sites, particularly if very small. My blog, and Autotools Mythbuster, are not exactly unknown sites, but they are still small enough that dealing with Ads is barely worth it for most people, and clearly not worth it for me.

For big, global and media sites, ads are still (for the most part) the way to get money, and to keep running. And that is true whether those sites started as “blogosphere” Internet-native sites, like Ars Technica, or as online versions of establishment news papers like The New York Times. And particularly as you go big, and you want to have curated content, secure content and (extremely important here!) good and verified content, you have costs that add up and that require some level of income to maintain.

You can see that with LWN.net, whose content is very well curated, and is for the most part paid contributions — I can tell you that the folks over there, starting from Jonathan, would not be able to afford such a great edited content if they had no income on it. It takes time to do so, and you can’t run it as a side project or moonlighting job. Or for a more mainstream example, take what’s happening with Snopes.

One of a common talking point I hear from many companies that are involved in advertising on the Internet, including the company I work for, is that we “enable free content publishing on a global scale”. It may sounds as a platitude written this way, and sound hollow. But once you think it twice or thrice over, it starts to ring true. Because all that content would not be able to pay for the wages of the people involved in charitable donations only. Ads do really help with that.

But then I hear the extremists that decide that all ads are bad, install uBlock, and refuse to even consider that the vast majority of both sites and advertisers are trying to make a honest living, and they are neither clickfarms nor malvertisers, even though these two problems are real. And I’m not fond of SEO tactics either, but that’s also something that is not evil by itself. What is the solution, according to some of those people I engaged with at many times? For some of them the answer is to just not expect money for your online content. This works fine for me, nowadays, but it wouldn’t have worked for me before. Also, the people I heard suggesting this before are usually people who have not tried producing high-quality content before. As I wrote years ago the amount of effort spent writing a blog post is minuscule compared to that needed for me to write a simple article for LWN. And indeed, I got paid for my effort for those — $100 sounds like a lot for writing, but turns out it’s well below minimum wage by the amount of time involved.

The alternative is of course saying that the answer is to sell ad-free subscriptions instead. Which is cool by me in general, but that can only work if those who can’t afford the subscriptions are indeed not using adblockers for everything. But that is clearly not the case, and it can’t be the case as long as malvertising is indeed a risk for most people. And my reason to have this opinion is that by relying on a subscription-only model, you end up either significantly limiting the gamut of content that is available on the Internet, or its availability to the public.

We have for instance all cursed at the bother of paywalls when trying to get to content that may be interesting, or even vital, to us. But the paywall is probably the only way to make sure that the content is paid for, with a subscription. This way you don’t risk the “cheaters” that will not subscribe but also won’t offer their attention so that advertisers can fill in for those subscriptions. Paywalls are nasty and I would not advocate for them. They limit access to information, dividing people into those who can afford to be informed, and those who can’t.

There is the other option of this of course, which is the “tip-jar” option: you provide free access to the information, with no advertisement, but you have a public support campaign that suggests people to provide money for the content, either by direct subscription or by providing fundraising campaigns, such as Patreon. The problem with this system is that it’s skewed towards the content that is of interests to the people with more disposable income. It’s the same problem as the prototypes coming in on Kickstarter and similar sites: since I’m a well-paid tech employee, I can afford to finance hobby projects and content that interests me, which means the creator will try to pander to my taste by providing projects and content that appeal to well paid tech employees. And then there is the occasional situation in which those tech employees grow a heart, a passion, or more properly just want to be some sane, decent, human beings and decide to pay for content, projects, services or charities that they have no stakes in.

But if you write content that is targeted to a group of people with no disposable income, you probably would feel better if you knew that honest companies with an advertising budget underpin your wages, rather than feeling at the mercy of the magnanimity of some people who are not your target audience, and may one day just disagree with you, your message or your method, and just cut your funds. Although of course, this is also a double-edged sword, as we’ve seen that hateful content can be cut from ads revenue but persist through the deep pocket of the people wanting the message to stay out.

Where does this leave us? I’m not sure. I just think that the subscription model is not the nice good solution that I hear some people suggest. I also find it quite ironic that a number of people who suggested this option – and told me explicitly that they think content should either be able to make people pay for their content or not get money out of it – are further to the left of me, and appear disgusted at the suggestion that the free market is here to stay. Well, guess what they just described for content? And yeah it’s not perfect, far from it.

Honestly, I really would like for super-partes agencies to be able to distribute funds, fairly gathered, into diverse content creators. But that is very hard to accomplish. The obvious answer would be to let the government collect taxes and redistribute it in funds for the art. But we also know that providing the government with the task of redistributing money for content leads to the government publishing only content that looks favourable to those in charge. And I’m not really sure there is much of a better option nowadays.

I should probably write another post talking about funding sources like the old Flattr and the Brave browser, but that’s a topic for another time.

August 02, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)

Just a quick note that Expat 2.2.3 has been released. For Windows users, it fixes DLL hijacking (CVE-2017-11742). On Linux, extracting entropy for Hash DoS protection no longer blocks, which affected D-Bus and systems that are low on entropy early in the boot process. For more details, please check the change log.

July 30, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Why is `git send-email` so awful in 2017? (July 30, 2017, 10:04 UTC)

I set myself out to send my first Linux kernel patch using my new Dell XPS13 as someone contacted me to ask for help supporting a new it87 chip in the gpio-it87 driver I originally contributed.

Writing the (trivial) patch was easy, since they had some access to the datasheet, but then the problem came to figure out how to send it over to the right mailing list. And that took me significantly more time than it should have, and significantly more time than writing the patch, too.

So why is it that git send-email is still so awful, in 2017?

So the first problem is that the only way you can send these email is either through a sendmail-compatible interface, which is literally an interface older than me (by two years), or through SMTP directly (this is even older, as RFC 821 is from 1982 — but being a protocol, that I do expect). The SMTP client has at least support for TLS, provided you have the right Perl modules installed, and authentication, though it does not support more complex forms of authentication such as Gmail’s XOAUTH2 protocol (ignore the fact it says IMAP, it is meant to apply to both IMAP and SMTP).

Instead, the documented (in the man page) approach for users with Gmail and 2FA enabled – which should be anybody who wants to contribute to the Linux kernel! – is to request an app-specific password and saving it through the credential store mechanism. Unfortunately the default credential store just stores it as unencrypted plaintext. Instead there are a number of credential helpers you can use, either using Gnome Keyring or libsecret, and so on.

Microsoft maintains and releases its own Credential Manager which is designed to support multi-factor login to a number of separate services, including GitHub and BitBucket. Thank you, Microsoft, although it appears to only be available for Windows, sigh!

Unfortunately it does not appear there is a good credential helper for either KWallet or LastPass which would have been interesting — to a point of course. I would probably never give LastPass an app-specific password to my Google account, as it would defeat my point of not keeping that particular password in a password manager.

So I start looking around and I find that there is a tool called keyring2 which supposedly has kwallet support, though on Arch Linux it does not appear to be working (the kwallet support, not the tool, which appear to work fine with Gnome Keyring). So I checked out the issues, and the defaulting to gnome-keyring is known, and there is a feature request for a LastPass backend. That sounds promising, right? Except that the author suggests building it as a separate library, which makes sense to a point. Unfortunately the implicit reference to their keyrings.alt (which does not appear to support KDE/Plasma), drove me away from the whole thing. Why?

License is indicated in the project metadata (typically one or more of the Trove classifiers). For more details, see this explanation.

And the explanation then says:

I acknowledge that there might be some subtle legal ramifications with not having the license text with the source code. I’ll happily revisit this issue at such a point that legal disputes become a real issue.

Which effectively reads to me as “I know what the right thing to do is, but it cramps my style and I don’t want to do it.” The fact that there have been already people pointing out the problem, and the fact that multiple issues have been reported and then marked as duplicate of this master issue, should speak clearly enough.

In particular, if I wanted to contribute anything to these repositories I would have no hope to do so but in my free time, if I decide to apply for a personal project request, as these projects are likely considered “No License” by the sheer lack of copyright information or licenses.

Now, I know I have not been the best person for this as well. But at least for glucometerutils I have made sure that each file lists its license clearly, and the license is spelt out in the README file too. And I will be correcting some of my past mistakes at some point soon, together with certain other mistakes.

But okay, so this is not a viable option. What else remains to use? Well, turns out that there is an actual FreeDesktop.org specification, or at least a draft, which appears to have been last touched seven years ago, for a common API to share between GNOME and KWallet, and for which there are a few other implementations already out there… but the current KWallet does not support it, and the replacement (KSecretService) appears to be stalled/gone/deprecated. And that effectively means that you can’t use that either.

Now, on Gentoo I know I can use msmtp integrated with KWallet and the sendmail interface, but I’m not sure if in Arch Linux it would work correctly. After all I even found out that I needed to install a number of Perl modules manually, because they are not listed in the dependencies and I don’t think I want to screw with PKGBUILD files if I can avoid it.

So at the end of the day, why is git send-email so awful? I guess the answer is that in so many years we still don’t have a half-decent, secure replacement for sending email. We need what they would now call “disruptive technology”, akin to how SSH killed Telnet, to bring up a decent way to send email, or at least submit Git patches to the Linux kernel. Sigh.

July 28, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Password Advice (July 28, 2017, 08:04 UTC)

Somehow in all the posts I’ve written about 2FA and other security concerns, I have not really spent much time talking about passwords themselves, except to say that I believe in password managers and use LastPass.

The topic of what a strong password is always a tricky one, as the infosec professionals tend to divide themselves between those that subscribe to XKCD’s policy that raw password length is more important than composition, and those that keep insisting that dictionary attacks are the only thing worth protecting against, with some graduation between these two positions. I would say that in this case, the correct answer is somewhere in between.

There are indeed too many ways for a simple password to become just too easy. A password that requires to only have numbers (effectively, a PIN), is not only bad because of the small number of combination possible, but also because humans will lead to something they can memorize easily… a date. Or, if they are mathematicians, physics, chemists, or spent enough time in one of those disciplines, they’ll end up using one of the important constants.

Similarly, if only letters are allowed, lots of people will go for names, or nouns, or in general what we call “dictionary words” (which is confusing for a Scrabble player, because proper names are not considered part of the dictionary there). And once you ask for both letters and numbers, you just end up with leetspeak, which is pretty much a simple translation layer on top of the dictionary, which may extend the entropy a little bit, but not really in a significant way.

Effectively, the password composition rules, probably designed to stop people from using dictionary passwords, just slightly increased the size of the dictionary, but not really the safety of those passwords. Which is why most of everybody got very happy when NIST dropped the composition rules in their recommendations.

And as the Naked Security article points out, the recommendation focused a lot on the length of passwords. Because a longer password is a stronger password, if it’s not in the dictionary. And an easy way for it not to be in the dictionary is following XKCD’s approach and using a set of words, instead of a password. This is unfortunately not as scalable as we would like, as too many websites refuse strong passwords.

So what you need is to maximise the entropy extracted from your memorable passphrase so that it fits within the rules of the various websites. One of this is SuperGenPass, which I used to use myself. It applies a derivation function on top of the passphrase and the website domain do generate a new variable-length password. It sound cooler than it is, because unless you also keep a mapping between equivalent domains, and you keep in mind which website still has composition rules that require symbols and which don’t, and the length of the passwords for each, you now augmented your memorable passphrase with a constellation of parameters that are nearly impossible to remember. Including how many times the password was compromised.

On the other hand, SuperGenPass is still a better option that Manuel Blum (of CAPTCHA fame) tried to suggest to people during the last ENIGMA conference. Blum’s suggested method is, supposedly, a way to mentally apply a formula that collapses a “master password” and the name of a given site into a string of letters and numbers that can be used as a password. Most of the people in the known at the conference have been openly disagreeing with this option being even remotely friendly to users, not just for the requirement of being able to perform complex mental arithmetic (which I’m not fond of myself either, to be clear) but also because it does not solve any of the actual issues with memorizing passwords.

In particular, Blum’s method ends up generating passwords that are relatively short, bringing them within the reach of non-dictionary bruteforce attacks. They also do not the memory problem because in addition to the master password, and the formula, you would have to know the length of the password expected by each site. And in addition to that you need to have a “generation” number to use when the password gets invalidated (and the site refuses to let you keep or re-use the old password). It effectively only solve one issue: the fact that once your password is leaked, it’s very hard (if at all possible) to figure out the master password and thus compromise another website.

I’ll also add that I find Blum’s theatricals annoying. In the talk he goes on to say that when he’s asked for the password of a website like REI, he can say he does not remember whether he had an account with that website, but if he did, he can tell what the password is. This obviously can’t be true, given the limitations I listed above: you’d have to know the length of the password, and you would have to know the “generation” of it, if you have ever burnt through any because of compromised passwords.

Okay, so SuperGenPass and Blum’s methods are both useless. I have already said that my personal preference is to use LastPass (but you can use whichever password manager you want), and have it generate random passwords that are long enough, so that they can’t be bruteforced. You don’t have to remember a 40 characters password for each website, you just need a strong, long, memorable password for your manager, and from there, you can just have the new password stored in encrypted form. No derivation function, no need to remember how many times you changed password for a given site: in case of compromise, just go and generate a new 40 characters password.

But all of this just pushes the problem around, as the password manager, your system and some high-value accounts now need strong, long, memorable passwords, or even better passphrases. Your password manager, because that’s what protects every other password you have. You system, because you log in to it so many times a day (don’t you?), that you need something you can remember rather than something you can store. And valuable accounts, because some of them are too important to store in a single point of failure such as a password manager (my personal Google account password is not stored in LastPass — which is why I can actually trust it to unlock access to my LastPass account when I’m in countries I have not visited before).

This brings me to my personal suggestions on how to make such a memorable password in the first place. I don’t think you can do so with mental algorithms, or with a long word association game like Randall Munroe thinks — I never can remember the order of the four words in his example. I can tell you that’s the case because one of the throwaway accounts I shared with people used that as password, and I always had to look up xkcd for it. Oh and by the way, don’t ever use that string for a password for Internet-accessible services, because it’s high-up in the dictionaries of common passwords, together with love, sex, secret and god.

Instead, I like the option that my high school computer science teacher provided us with. Take a memorable quote, and mangle it, in a memorable way. His example, in Italian, would be to take the first line of a famous poetry from Giacomo Leopardi (La donzelletta vien dalla campagna), replace one word with another non-sense one, and another for an alternative, but similarly sounding word (La donzellotta vien dalla montagna — replacing “countryside” with “mountain”). This works well for a number of uses, and I have indeed been using it in various forms for many years, when long passphrases are needed.

You can extend this not only to poetry, but to quotes from movies or lines from songs. Even an unmodified phrase, such as the title of a book, can be much stronger than your average password, particularly if the book is not extremely famous and its title long. Even if there was a dictionary of book titles to go through, there are significantly more combinations of those that can be used than just single-words. The best thing, though, is to just mutate the title, quote or line in such a way that it’s not obvious. This also prevent an unexpected information leakage if you start whistling the song to yourself as you login on the system.

For instance, if you take Monty Python’s famous song, you can take the line

Always look at the bright side of life

This by itself, if looking at mindless bruteforce, is a fairly strong passphrase. Indeed a random online calculator of password “search space” (that at first sight appears to not spew nonsense, so I thought it made sense linking to it), reports the search space to 2.10×10e73 combinations. I actually think they are lowballing it, as they assume that you only need to search the space of the characters you’re composing from, but adding a digit to the passphrase does not extend its strength.

It’s the fact that you can use digits, that increases the number of combinations that brute forcing needs to go through. There is a caveat that if you don’t use numbers, your password will fall into a simpler search space, but that requires the attacker to either settle for a smaller search space to begin with, or target you and know you’re not using numbers. And in that case, shoulder-surfing is still probably easier.

As the website I linked shows, though, this number does not actually represent a password strength by itself. In addition to the fact that these are all normal English words, this is a phrase that can be found not only as a song title, or as a song line, but also as a common phrase in use. If an attacker is bruteforcing strong passphrases, a list of commonly used phrases would be the first part of a dictionary attack to use.

So what if you mutate some of the words?

Sometimes look at the dark side of death

This phrase is actually longer than the other, and the search space is reported as 1.52×10e77. But it truly is not that more complicated than the other if the attacker is focusing on strong passwords. Indeed, you’re just replacing a some of the words with either their antonyms, or with similar adverbs. Replacing “Always” with “Never”, “Sometimes”, “Often”, and so on means that the brute forcing just need to go and mutate some of the words separately with a list. It’s the passphrase equivalent to replacing ‘l’ with ‘1’ and ’t’ with ‘7’. If “password” is the first dictionary word tried, “p4s5w0rd” is not far down the list either.

Always look at the bright grail of Brian

This phrase has the same search space, but since you’re not replacing antonyms, but rather mashing up a famous Monty Python song with the title of the movie it came in and another effectively unrelated movie. You can still sing it to yourself in your head to the same tune, while at the same time it makes it nearly impossible for a mutating algorithm to think of doing it.

See, here is the trick for safe, strong, and memorable passwords: creativity. Because computers don’t have creativity and so they can’t figure out your passphrase that easily.

It gets even better if the passphrase is in a different language than the site it’s for. Even better, if the passphrase itself is actually composed in one language, but typed in another, with a 1:1 substitution of words that do not build a sensible phrase:

Sempre guarda a il splendente graal di Brian

The hard part with these two suggestions is that if you go too over the top with the substitutions, you end up with not memorizing your passphrase, and that makes things… complicated. But that’s really up to you to be careful with, and there’s no hard and fast rule to help you in that context.

July 27, 2017
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Last evening, I ran some updates on one of my servers. One of the updates was from MariaDB 10.1 to 10.2 (some minor release as well). After compiling, I went to restart, but it failed with:

# /etc/init.d/mysql start
* Checking mysqld configuration for mysql ...
[ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
[ERROR] Aborting

* mysql config check failed [ !! ]
* ERROR: mysql failed to start

I’m not sure why this just hit me now, but it looks like it is a function within the init script that’s causing it to look for files in the nonexistent directory of /usr/share/mysql/ instead of the appropriate /usr/share/mariadb/. The fast fix here (so that I could get everything back up and running as quickly as possible) was to simply symlink the directory:

cd /usr/share
ln -s mariadb/ mysql

Thereafter, MariaDB came up without any problem:

# /etc/init.d/mysql start
* Caching service dependencies ... [ ok ]
* Checking mysqld configuration for mysql ... [ ok ]
* Starting mysql ... [ ok ]
# /etc/init.d/mysql status
* status: started

I hope that information helps if you’re in a pinch and run into the same error message.

Cheers,
Zach

UPDATE: It seems as if the default locations for MySQL/MariaDB configurations have changed (in Gentoo). Please see this comment for more information about a supportable fix for this problem moving forward. Thanks to Brian Evans for the information. 🙂

July 24, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)

Since, as I announced some time ago, I’m moving to London in a few months, I’ve been spending the past few weeks organizing the move, deciding what I’ll be bringing with me and what I won’t. One of the things I decided to do was trying to figure out which hardware I would want with me, as I keep collecting hardware both for my own special projects and just out of curiosity.

I decided that having so many laptops as I have right now is a bad idea, and it is due time to consolidate on one or two machines if possible. In particular, my ZenBook has been showing its age, with only 4GB of RAM, and my older Latitude which is now over seven years old does not have a working battery anymore (but with 8GB of RAM it would actually been quite usable!), plus it’s way too bulky for me to keep travelling with, given my usual schedule. Indeed, to be able to have something I can play with on the road, I ended up buying an IdeaPad last year.

So thanks to the lowered value of the Sterling (which I won’t be particularly happy about once I start living there), I decided to get myself a new laptop. I decided for the Dell XPS 13, which is not quite an Ultrabook but it’s also quite handy and small. The killer feature of it for me has been having a USB-C connector and being able to charge through it, since my work laptop is a HP Chromebook 13, which also charges over USB-C, and that gives me the ability to travel with a single power brick.

I ordered it from Dell UK, delivered it to Northern Ireland then reshipped to me, and it arrived this past Friday. The configuration I bought is the i7, 16GB, QHD (3200x1800) display with Ubuntu (rather than Windows 10). I turned it on at the office, as I wanted to make sure it was all in one piece and working, and the first surprise was the musical intro that it started up with. I’m not sure if it’s Ubuntu’s or Dell’s but it’s annoying. I couldn’t skip it with the Esc button, and I didn’t figure out how to make it shut the heck up (although that may have been me not figuring out yet that the function keys are bound to special meanings first).

I also found myself confused by the fact that Dell only provided the BIOS (well, EFI) update file in MS-DOS/Windows format. Turns out that not only the firmware itself can read the file natively (after all EFI uses PE itself), but also Dells is providing the firmware through the LVFS service, that you may remember from Richard Hughes’s blog. The latest firmware for this device is not currently available, but it should be relatively soon.

Update (2017-07-26): The new firmware was release on LVFS and I tried updating it with the fwupd tool. Unfortunately the Arch Linux package does not work at all on my Antergos install. I’m not sure if it’s because the Antergos install changes some subtle parameter from the EFI install of Arch Linux itself, or because the package is completely broken. In particular it looks like the expected paths within the EFI System Partition (ESP) are completely messed up, and fwupd does not appear to identify them dynamically. Sigh.

The hardware of the laptop is pretty impressive, although I’m not a fan of the empty space near the top, that looks to me like an easy catch for cables and ties, which make me afraid for its integrity. The device is also quite denser than I was expecting: it’s quite heavier than the Zenbook, although it packs much more of a punch. The borderless screen is gorgeous but it also means the webcam is in the bottom corner of the screen rather than at the top, likely making it awkward to have a videocall. The keyboard is a bit tricky to get used to, because it’s not quite as good as the one in the ZenBook, but it’s still fairly good quality.

By the way, one of the first thing I did was replacing the Ubuntu install with an install of Antergos (which is effectively Arch Linux with an easier installer). This did mean disabling Secure Boot, but I guess I’ll have to live with it until we get a better idea of how to do Secure Boot properly on Linux and full-disk encryption.

Once I got home, I did what I do with my work laptop too: I connected it to my Anker USB-C dock, and it seemed to work alright. Except for some video corruption here and there, particularly on Konsole. Then I started noticing the lack of sound — but that turned out to be a red herring. The answer is that both the on-board speakers and the HDMI audio output are wired through the same sound interface, just appear as different “profiles”.

It wasn’t until I was already compiling webkitgtk for an hour that I noticed that the laptop wasn’t actually charging, and I thought the problem was with the dock. Instead the answered turned out to be that the HP Chromebook 13 charger is not compatible with the XPS 13, while the Chromebook Pixel charger worked fine. Why the difference? I’m not sure, I guess I need to figure out a way to inspect what is seen by the USB-C bus to figure out what the problem is with that charger. It should not be a problem of wattage, as both the HP charger and the original Dell charger provided with the laptop are 45W.

Speaking of the USB-C dock, there is a funny situation: if the laptop boots with it connected, and the lid closed, it does not appear to return the monitor on (all is fine if it boots with it disconnected). Also, it looks like the default DM provided by Antergos only shows the login page on the laptop’s screen, making it hard to log in at all. And in the usual mess that multi-screen support is with modern Linux desktops, Plasma needs to be killed and restarted to switch between the two monitors. Sigh!

As for the screen corruption that I have noted earlier, it seems to be fixed by one of these two options: upgrading to Linux 4.12 (from Arch Linux testing repository) or changing the compositor’s setting from OpenGL 2.0 to OpenGL 3.1. I think it may be the latter but I have no intention to try this out yet.

It looks like I’ll be very happy with this laptop, I just need to figure out some new workflows so that I don’t feel out of place not having Gentoo on my “workstation”.

Also, to keep with my usual Star Trek host naming, this new laptop is named everett after the (non-canon) USS Everett, which is the same class (Nova-class) as my previous laptop, which was named equinox.

July 23, 2017
Michał Górny a.k.a. mgorny (homepage, bugs)
Optimizing ccache using per-package caches (July 23, 2017, 18:03 UTC)

ccache can be of great assistance to Gentoo developers and users who frequently end up rebuilding similar versions of packages. By providing a caching compiler frontend, it can speed up builds by removing the need to build files that have not changed again. However, it uses a single common cache directory by default which can be suboptimal even if you are explicitly enabling ccache only for a subset of packages needing that.

The likeliness of cross-package ccache hits is pretty low — majority of the hits occurs within a single package. If you use a single cache directory for all affected packages, it grows pretty quick. Besides a possible performance hit from having a lot of files in every directory, this means that packages built later can shift earlier packages out of the cache, resulting in meaninglessly lost cache hits. A simple way to avoid both of the problems is to use separate ccache directories.

In my solution, a separate subdirectory of /var/cache/ccache is used for every package, named after the category, package name and slot. While the last one is not strictly necessary, it can be useful for slotted packages such as LLVM where I do not want frequently changing live package sources to shift the release versions out of the cache.

To use it, put a code similar to the following in your /etc/portage/bashrc:

if [[ ${FEATURES} == *ccache* && ${EBUILD_PHASE_FUNC} == src_* ]]; then
	if [[ ${CCACHE_DIR} == /var/cache/ccache ]]; then
		export CCACHE_DIR=/var/cache/ccache/${CATEGORY}/${PN}:${SLOT}
		mkdir -p "${CCACHE_DIR}" || die
	fi
fi

The first condition makes sure the code is only run when ccache is enabled, and only for src_* phases where we can rely on userpriv being used consistently. The second one makes sure the code only applies to a specific (my initial) value of CCACHE_DIR and therefore avoids both nesting the cache indefinitely when Portage calls subsequent phase functions, and applying the replacement if user overrides CCACHE_DIR.

You need to either adjust the value used here to the directory used on your system, or change it in your /etc/portage/make.conf:

CCACHE_DIR="/var/cache/ccache"

Once this is done, Portage should start creating separate cache directories for every package where you enable ccache. This should improve the cache hit ratio, especially if you are using ccache for large packages (why else would you need it?). However, note that you will no longer have a single cache size limit — every package will have its own limit. Therefore, you may want to reduce the limits per-package, or manually look after the cache periodically.

July 21, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Fake “Free” Software hurts us all (July 21, 2017, 10:04 UTC)

Brian Krebs, the famous information security reporter, posted today (well, at the time of writing) an article on security issues with gSOAP library. I found this particularly interesting to me because I remembered seeing the name before. Indeed, Krebs links to the project’s SourceForge page, which is something to see. It has a multi-screen long list of “features”, including a significant number of variant and options for the protocol, which ends in the following:

Licenses: GPLv2, gSOAP public license (for engine and plugins), commercial non-GPL license available upon request (software is 100% in-house developed, no third-party GPL contributions included)

Ah, there we go. “Free” Software.

You may remember my post just a few days ago about the fact that Free Software to actually make a difference in society the way Stallman prophesizes needs a mediating agency, and at least right now that agency is companies and the free market. I argued that making your software usable by companies that provide services or devices is good, as it makes for healthy, usable and used projects, and increase competition and reduce costs of developing new solutions. So is gSOAP the counterexample? I really don’t think so. gSOAP is the perfect example for me that matches my previous rant about startups and fake communities.

The project at first look like the poster child for FSF-compatible software, since it’s licensed under GPL-2, and it clearly has no CLA (Contributor License Agreement), though the company provides a “way out” from GPL-2 obligations by buying a commercial license. This is not, generally speaking, a bad thing. I have seen many examples, including in the open multimedia cliques, of using this trick to foster the development of open solutions while making money to pay salary or build new Free Software companies.

But generally, those projects allow third party contributions with a CLA or similar agreement that allows the “main” copyright holders to still issue proprietary licenses, or enforce their own rights. You can for instance see what Google’s open source projects do about it. Among other things, this contribution method also disallows re-licensing the software, as that requires agreement from all copyright holders. In the case of gSOAP, that is not the case: as they say their software is «100% in-house developed».

They are very well proud of this situation, because it gives them all the power: if you want to use gSOAP without paying, you’re tied to the GPL, which may or may not become a compliance problem. And if you happen to violate the license, they have all the copyright to sue you or just ask you to settle. It’s a perfect situation for copyright trolls.

But, because of this, even though the software is on paper “Free Software” according to FSF, it’s a piece of proprietary software. Sure you can fork the library and build your own GPL-2 instead, as you have the freedom of fork, but that does not make it a community, or a real Free Software project. And it also means you can’t contribute patches to it to make it more secure, safer, or better for society. You could report bugs, including security bugs, but what’s the likeliness that you would actually care to do so, given that one of the first thing they make clear on their “project” page is that they are not interested in your contributions? And we clearly can see that the particular library could have used some care from the community, given its widespread adoption.

What does this mean to me, is that gSOAP is a clear example that just releasing something under GPL-2 is not enough to make it Free Software, and that even “Free” Software released under GPL-2 can be detrimental to society. And it also touches on the other topic I brought up recently, that is that you need to strike a balance between making code usable to companies (because they will use, and thus very likely help you extend or support your project) and keeping it as a real community or a real project. Clearly in this case the balance was totally off. If gSOAP was available with a more liberal license, even LGPL-2, they would probably lose a lot in license fees, as for most companies, just using this as a shared library would be enough. But it would then allow developers, both hobbyists and working for companies, to contribute fixes so that they trickle down on everybody’s device.

Since I do not know what the proprietary license that the company behind gSOAP requires their customers to agree with, I cannot say whether there is any incentive in said companies to provide fixes back to the project, but I assume if they were to do so, they wouldn’t be contributing them under GPL-2, clearly. What I can say is that for the companies I worked for in the past, actually receiving the library under GPL-2 and being able to contribute the fixes back would have been a better deal. The main reason is that as much as a library like this can be critical to connected devices, it does not by itself contain any of the business logic. And there are ways around linking GPL-2 code into the business logic application, that usually involve some kind of RPC between the logic and the frontend. And being able to contribute the changes back to the open source project would allow them to not have to maintain a long set of patches to sync release after release — I had the unfortunate experience of having to deal with something in that space before.

My advice, is once again, to try figuring out what you want to achieve by releasing a piece of software. Are you trying to make a statement, sticking it to the man, by forcing companies not to use your software? Are you trying to make money by forcing companies interested in using your work to buy a special license from you? Or are you contributing to Free Software because you want to improve society? In the latter case, I would suggest you consider a liberal license, to make sure that your code (that can be better than proprietary, closed-source software!) is actually available for those mediating agencies that transform geeky code into usable gadgets and services.

I know, it would be oh so nicer, if by just releasing something awesome under GPL-2 you’d force every company to release all their firmware as Free Software as well, but that’s not going to happen. Instead, if they feel they have to, they will look for worse alternatives, or build their own (even worse) alternatives, and keep them to themselves, and we’ll all be the poorer for it. So if in doubt, consider MIT or Apache 2 licenses. The latter in particular appears to be gaining more and more traction as both Google and Microsoft appear to be fond of the license, and Facebook’s alternative is tricky.

Some of you may consider me a bit of a hypocrite since I have released a number of projects under more restrictive Free licenses (including AGPL-3!) before I came to the conclusion that’s actually bad for society. Or at least before I came back to that idea, as I was convinced of that back in 2004, when I wrote (In Italian) of why I think MySQL is bad for Free Software (I should probably translate that post, just for reference). But what I decided is that I’ll try now to do my best to re-license the projects for which I own the full copyright under Apache 2 licenses. This may take a little while until I figure out all the right steps, but I feel is the right thing to do.

July 20, 2017
Hanno Böck a.k.a. hanno (homepage, bugs)

KeyLately, some attention was drawn to a widespread problem with TLS certificates. Many people are accidentally publishing their private keys. Sometimes they are released as part of applications, in Github repositories or with common filenames on web servers.

If a private key is compromised, a certificate authority is obliged to revoke it. The Baseline Requirements – a set of rules that browsers and certificate authorities agreed upon – regulate this and say that in such a case a certificate authority shall revoke the key within 24 hours (Section 4.9.1.1 in the current Baseline Requirements 1.4.8). These rules exist despite the fact that revocation has various problems and doesn’t work very well, but that’s another topic.

I reported various key compromises to certificate authorities recently and while not all of them reacted in time, they eventually revoked all certificates belonging to the private keys. I wondered however how thorough they actually check the key compromises. Obviously one would expect that they cryptographically verify that an exposed private key really is the private key belonging to a certificate.

I registered two test domains at a provider that would allow me to hide my identity and not show up in the whois information. I then ordered test certificates from Symantec (via their brand RapidSSL) and Comodo. These are the biggest certificate authorities and they both offer short term test certificates for free. I then tried to trick them into revoking those certificates with a fake private key.

Forging a private key

To understand this we need to get a bit into the details of RSA keys. In essence a cryptographic key is just a set of numbers. For RSA a public key consists of a modulus (usually named N) and a public exponent (usually called e). You don’t have to understand their mathematical meaning, just keep in mind: They’re nothing more than numbers.

An RSA private key is also just numbers, but more of them. If you have heard any introductory RSA descriptions you may know that a private key consists of a private exponent (called d), but in practice it’s a bit more. Private keys usually contain the full public key (N, e), the private exponent (d) and several other values that are redundant, but they are useful to speed up certain things. But just keep in mind that a public key consists of two numbers and a private key is a public key plus some additional numbers. A certificate ultimately is just a public key with some additional information (like the host name that says for which web page it’s valid) signed by a certificate authority.

A naive check whether a private key belongs to a certificate could be done by extracting the public key parts of both the certificate and the private key for comparison. However it is quite obvious that this isn’t secure. An attacker could construct a private key that contains the public key of an existing certificate and the private key parts of some other, bogus key. Obviously such a fake key couldn’t be used and would only produce errors, but it would survive such a naive check.

I created such fake keys for both domains and uploaded them to Pastebin. If you want to create such fake keys on your own here’s a script. To make my report less suspicious I searched Pastebin for real, compromised private keys belonging to certificates. This again shows how problematic the leakage of private keys is: I easily found seven private keys for Comodo certificates and three for Symantec certificates, plus several more for other certificate authorities, which I also reported. These additional keys allowed me to make my report to Symantec and Comodo less suspicious: I could hide my fake key report within other legitimate reports about a key compromise.

Symantec revoked a certificate based on a forged private key

SymantecComodo didn’t fall for it. They answered me that there is something wrong with this key. Symantec however answered me that they revoked all certificates – including the one with the fake private key.

No harm was done here, because the certificate was only issued for my own test domain. But I could’ve also fake private keys of other peoples' certificates. Very likely Symantec would have revoked them as well, causing downtimes for those sites. I even could’ve easily created a fake key belonging to Symantec’s own certificate.

The communication by Symantec with the domain owner was far from ideal. I first got a mail that they were unable to process my order. Then I got another mail about a “cancellation request”. They didn’t explain what really happened and that the revocation happened due to a key uploaded on Pastebin.

I then informed Symantec about the invalid key (from my “real” identity), claiming that I just noted there’s something wrong with it. At that point they should’ve been aware that they revoked the certificate in error. Then I contacted the support with my “domain owner” identity and asked why the certificate was revoked. The answer: “I wanted to inform you that your FreeSSL certificate was cancelled as during a log check it was determined that the private key was compromised.”

To summarize: Symantec never told the domain owner that the certificate was revoked due to a key leaked on Pastebin. I assume in all the other cases they also didn’t inform their customers. Thus they may have experienced a certificate revocation, but don’t know why. So they can’t learn and can’t improve their processes to make sure this doesn’t happen again. Also, Symantec still insisted to the domain owner that the key was compromised even after I already had informed them that the key was faulty.

How to check if a private key belongs to a certificate?

SSLShopper checkIn case you wonder how you properly check whether a private key belongs to a certificate you may of course resort to a Google search. And this was fascinating – and scary – to me: I searched Google for “check if private key matches certificate”. I got plenty of instructions. Almost all of them were wrong. The first result is a page from SSLShopper. They recommend to compare the MD5 hash of the modulus. That they use MD5 is not the problem here, the problem is that this is a naive check only comparing parts of the public key. They even provide a form to check this. (That they ask you to put your private key into a form is a different issue on its own, but at least they have a warning about this and recommend to check locally.)

Furthermore we get the same wrong instructions from the University of Wisconsin, Comodo (good that their engineers were smart enough not to rely on their own documentation), tbs internet (“SSL expert since 1996”), ShellHacks, IBM and RapidSSL (aka Symantec). A post on Stackexchange is the only result that actually mentions a proper check for RSA keys. Two more Stackexchange posts are not related to RSA, I haven’t checked their solutions in detail.

Going to Google results page two among some unrelated links we find more wrong instructions and tools from Symantec, SSL247 (“Symantec Specialist Partner Website Security” - they learned from the best) and some private blog. A documentation by Aspera (belonging to IBM) at least mentions that you can check the private key, but in an unrelated section of the document. Also we get more tools that ask you to upload your private key and then not properly check it from SSLChecker.com, the SSL Store (Symantec “Website Security Platinum Partner”), GlobeSSL (“in SSL we trust”) and - well - RapidSSL.

Documented Security Vulnerability in OpenSSL

So if people google for instructions they’ll almost inevitably end up with non-working instructions or tools. But what about other options? Let’s say we want to automate this and have a tool that verifies whether a certificate matches a private key using OpenSSL. We may end up finding that OpenSSL has a function x509_check_private_key() that can be used to “check the consistency of a private key with the public key in an X509 certificate or certificate request”. Sounds like exactly what we need, right?

Well, until you read the full docs and find out that it has a BUGS section: “The check_private_key functions don't check if k itself is indeed a private key or not. It merely compares the public materials (e.g. exponent and modulus of an RSA key) and/or key parameters (e.g. EC params of an EC key) of a key pair.”

I think this is a security vulnerability in OpenSSL (discussion with OpenSSL here). And that doesn’t change just because it’s a documented security vulnerability. Notably there are downstream consumers of this function that failed to copy that part of the documentation, see for example the corresponding PHP function (the limitation is however mentioned in a comment by a user).

So how do you really check whether a private key matches a certificate?

Ultimately there are two reliable ways to check whether a private key belongs to a certificate. One way is to check whether the various values of the private key are consistent and then check whether the public key matches. For example a private key contains values p and q that are the prime factors of the public modulus N. If you multiply them and compare them to N you can be sure that you have a legitimate private key. It’s one of the core properties of RSA that it’s secure based on the assumption that it’s not feasible to calculate p and q from N.

You can use OpenSSL to check the consistency of a private key:
openssl rsa -in [privatekey] -check

For my forged keys it will tell you:
RSA key error: n does not equal p q

You can then compare the public key, for example by calculating the so-called SPKI SHA256 hash:
openssl pkey -in [privatekey] -pubout -outform der | sha256sum
openssl x509 -in [certificate] -pubkey |openssl pkey -pubin -pubout -outform der | sha256sum

Another way is to sign a message with the private key and then verify it with the public key. You could do it like this:
openssl x509 -in [certificate] -noout -pubkey > pubkey.pem
dd if=/dev/urandom of=rnd bs=32 count=1
openssl rsautl -sign -pkcs -inkey [privatekey] -in rnd -out sig
openssl rsautl -verify -pkcs -pubin -inkey pubkey.pem -in sig -out check
cmp rnd check
rm rnd check sig pubkey.pem

If cmp produces no output then the signature matches.

As this is all quite complex due to OpenSSLs arcane command line interface I have put this all together in a script. You can pass a certificate and a private key, both in ASCII/PEM format, and it will do both checks.

Summary

Symantec did a major blunder by revoking a certificate based on completely forged evidence. There’s hardly any excuse for this and it indicates that they operate a certificate authority without a proper understanding of the cryptographic background.

Apart from that the problem of checking whether a private key and certificate match seems to be largely documented wrong. Plenty of erroneous guides and tools may cause others to fall for the same trap.

Update: Symantec answered with a blog post.

July 18, 2017
Sven Vermeulen a.k.a. swift (homepage, bugs)
Project prioritization (July 18, 2017, 18:40 UTC)

This is a long read, skip to “Prioritizing the projects and changes” for the approach details...

Organizations and companies generally have an IT workload (dare I say, backlog?) which needs to be properly assessed, prioritized and taken up. Sometimes, the IT team(s) get an amount of budget and HR resources to "do their thing", while others need to continuously ask for approval to launch a new project or instantiate a change.

Sizeable organizations even require engineering and development effort on IT projects which are not readily available: specialized teams exist, but they are governance-wise assigned to projects. And as everyone thinks their project is the top-most priority one, many will be disappointed when they hear there are no resources available for their pet project.

So... how should organizations prioritize such projects?

Structure your workload, the SAFe approach

A first exercise you want to implement is to structure the workload, ideas or projects. Some changes are small, others are large. Some are disruptive, others are evolutionary. Trying to prioritize all different types of ideas and changes in the same way is not feasible.

Structuring workload is a common approach. Changes are grouped in projects, projects grouped in programs, programs grouped in strategic tracks. Lately, with the rise in Agile projects, a similar layering approach is suggested in the form of SAFe.

In the Scaled Agile Framework a structure is suggested that uses, as a top-level approach, value streams. These are strategically aligned steps that an organization wants to use to build solutions that provide a continuous flow of value to a customer (which can be internal or external). For instance, for a financial service organization, a value stream could focus on 'Risk Management and Analytics'.

SAFe full framework

SAFe full framework overview, picture courtesy of www.scaledagileframework.com

The value streams are supported through solution trains, which implement particular solutions. This could be a final product for a customer (fitting in a particular value stream) or a set of systems which enable capabilities for a value stream. It is at this level, imo, that the benefits exercises from IT portfolio management and benefits realization management research plays its role (more about that later). For instance, a solution train could focus on an 'Advanced Analytics Platform'.

Within a solution train, agile release trains provide continuous delivery for the various components or services needed within one or more solutions. Here, the necessary solutions are continuously delivered in support of the solution trains. At this level, focus is given on the culture within the organization (think DevOps), and the relatively short-lived delivery delivery periods. This is the level where I see 'projects' come into play.

Finally, you have the individual teams working on deliverables supporting a particular project.

SAFe is just one of the many methods for organization and development/delivery management. It is a good blueprint to look into, although I fear that larger organizations will find it challenging to dedicate resources in a manageable way. For instance, how to deal with specific expertise across solutions which you can't dedicate to a single solution at a time? What if your organization only has two telco experts to support dozens of projects? Keep that in mind, I'll come back to that later...

Get non-content information about the value streams and solutions

Next to the structuring of the workload, you need to obtain information about the solutions that you want to implement (keeping with the SAFe terminology). And bear in mind that seemingly dull things such as ensuring your firewalls are up to date are also deliverables within a larger ecosystem. Now, with information about the solutions, I don't mean the content-wise information, but instead focus on other areas.

Way back, in 1952, Harry Markowitz introduced Modern portfolio theory as a mathematical framework for assembling a portfolio of assets such that the expected return is maximized for a given level of risk (quoted from Wikipedia). This was later used in an IT portfolio approach by McFarlan in his Portfolio Approach to Information Systems article, published in September 1981.

There it was already introduced that risk and return shouldn't be looked at from an individual project viewpoint, but how it contributes to the overall risk and return. A balance, if you wish. His article attempts to categorize projects based on risk profiles on various areas. Personally, I see the suggested categorization more as a way of supporting workload assessments (how many mandays of work will this be), but I digress.

Since then, other publications came up which tried to document frameworks and methodologies that facilitate project portfolio prioritization and management. The focus often boils down to value or benefits realization. In The Information Paradox John Thorp comes up with a benefits realization approach, which enables organizations to better define and track benefits realization - although it again boils down on larger transformation exercises rather than the lower-level backlogs. The realm of IT portfolio management and Benefits realization management gives interesting pointers as to the lecture part of prioritizing projects.

Still, although one can hardly state the resources are incorrect, a common question is how to make this tangible. Personally, I tend to view the above on the value stream level and solution train level. Here, we have a strong alignment with benefits and value for customers, and we can leverage the ideas of past research.

The information needed at this level often boils down to strategic insights and business benefits, coarse-grained resource assessments, with an important focus on quality of the resources. For instance, a solution delivery might take up 500 days of work (rough estimation) but will also require significant back-end development resources.

Handling value streams and solutions

As we implement this on the highest level in the structure, it should be conceivable that the overview of the value streams (a dozen or so) and solutions (a handful per value stream) is manageable, and something that at an executive level is feasible to work with. These are the larger efforts for structuring and making strategic alignment. Formal methods for prioritization are generally not implemented or described.

In my company, there are exercises that are aligning with SAFe, but it isn't company-wide. Still, there is a structure in place that (within IT) one could map to value streams (with some twisting ;-) and, within value streams, there are structures in place that one could map to the solution train exercises.

We could assume that the enterprise knows about its resources (people, budget ...) and makes a high-level suggestion on how to distribute the resources in the mid-term (such as the next 6 months to a year). This distribution is challenged and worked out with the value stream owners. See also "lean budgeting" in the SAFe approach for one way of dealing with this.

There is no prioritization of value streams. The enterprise has already made its decision on what it finds to be the important values and benefits and decided those in value streams.

Within a value stream, the owner works together with the customers (internal or external) to position and bring out solutions. My experience here is that prioritization is generally based on timings and expectations from the customer. In case of resource contention, the most challenging decision to make here is to put a solution down (meaning, not to pursue the delivery of a solution), and such decisions are hardly taken.

Prioritizing the projects and changes

In the lower echelons of the project portfolio structure, we have the projects and changes. Let's say that the levels here are projects (agile release trains) and changes (team-level). Here, I tend to look at prioritization on project level, and this is the level that has a more formal approach for prioritization.

Why? Because unlike the higher levels, where the prioritization is generally quality-oriented on a manageable amount of streams and solutions, we have a large quantity of projects and ideas. Hence, prioritization is more quantity-oriented in which formal methods are more efficient to handle.

The method that is used in my company uses scoring criteria on a per-project level. This is not innovative per se, as past research has also revealed that project categorization and mapping is a powerful approach for handling project portfolio's. Just look for "categorizing priority projects it portfolio" in Google and you'll find ample resources. Kendal's Advanced Project Portfolio Management and the PMO (book) has several example project scoring criteria's. But allow me to explain our approach.

It basically is like so:

  1. Each project selects three value drivers (list decided up front)
  2. For the value drivers, the projects check if they contribute to it slightly (low), moderately (medium) or fully (high)
  3. The value drivers have weights, as do the values. Sum the resulting products to get a priority score
  4. Have the priority score validated by a scoring team

Let's get to the details of it.

For the IT projects within the infrastructure area (which is what I'm active in), we have around 5 scoring criteria (value drivers) that are value-stream agnostic, and then 3 to 5 scoring criteria that are value-stream specific. Each scoring criteria has three potential values: low (2), medium (4) and high (9). The numbers are the weights that are given to the value.

A scoring criteria also has a weight. For instance, we have a scoring criteria on efficiency (read: business case) which has a weight of 15, so a score of medium within that criteria gives a total value of 60 (4 times 15). The potential values here are based on the "return on investment" value, with low being a return less than 2 years, medium within a year, and high within a few months (don't hold me on the actual values, but you get the idea).

The sum of all values gives a priority score. Now, hold your horses, because we're not done yet. There is a scoring rule that says a project can only be scored by at most 3 scoring criteria. Hence, project owners need to see what scoring areas their project is mostly visible in, and use those scoring criteria. This rule supports the notion that people don't bring around ideas that will fix world hunger and make a cure for cancer, but specific, well scoped ideas (the former are generally huge projects, while the latter requires much less resources).

OK, so you have a score - is that your priority? No. As a project always falls within a particular value stream, we have a "scoring team" for each value stream which does a number of things. First, it checks if your project really belongs in the right value stream (but that's generally implied) and has a deliverable that fits the solution or target within that stream. Projects that don't give any value or aren't asked by customers are eliminated.

Next, the team validates if the scoring that was used is correct: did you select the right values (low, medium or high) matching the methodology for said criteria? If not, then the score is adjusted.

Finally, the team validates if the resulting score is perceived to be OK or not. Sometimes, ideas just don't map correctly on scoring criteria, and even though a project has a huge strategic importance or deliverable it might score low. In those cases, the scoring team can adjust the score manually. However, this is more of a fail-safe (due to the methodology) rather than the norm. About one in 20 projects gets its score adjusted. If too many adjustments come up, the scoring team will suggest a change in methodology to rectify the situation.

With the score obtained and validated by the scoring team, the project is given a "go" to move to the project governance. It is the portfolio manager that then uses the scores to see when a project can start.

Providing levers to management

Now, these scoring criteria are not established from a random number generator. An initial suggestion was made on the scoring criteria, and their associated weights, to the higher levels within the organization (read: the people in charge of the prioritization and challenging of value streams and solutions).

The same people are those that approve the weights on the scoring criteria. If management (as this is often the level at which this is decided) feels that business case is, overall, more important than risk reduction, then they will be able to put a higher value in the business case scoring than in the risk reduction.

The only constraint is that the total value of all scoring criteria must be fixed. So an increase on one scoring criteria implies a reduction on at least one other scoring criteria. Also, changing the weights (or even the scoring criteria themselves) cannot be done frequently. There is some inertia in project prioritization: not the implementation (because that is a matter of following through) but the support it will get in the organization itself.

Management can then use external benchmarks and other sources to gauge the level that an organization is at, and then - if needed - adjust the scoring weights to fit their needs.

Resource allocation in teams

Portfolio managers use the scores assigned to the projects to drive their decisions as to when (and which) projects to launch. The trivial approach is to always pick the projects with the highest scores. But that's not all.

Projects can have dependencies on other projects. If these dependencies are "hard" and non-negotiable, then the upstream project (the one being dependent on) inherits the priority of the downstream project (the one depending on the first) if the downstream project has a higher priority. Soft dependencies however need to validate if they can (or have to) wait, or can implement workarounds if needed.

Projects also have specific resource requirements. A project might have a high priority, but if it requires expertise (say DBA knowledge) which is unavailable (because those resources are already assigned to other ongoing projects) then the project will need to wait (once resources are fully allocated and the projects are started, then they need to finish - another reason why projects have a narrow scope and an established timeframe).

For engineers, operators, developers and other roles, this approach allows them to see which workload is more important versus others. When their scope is always within a single value stream, then the mentioned method is sufficient. But what if a resource has two projects, each of a different value stream? As each value stream has its own scoring criteria it can use (and weight), one value stream could systematically have higher scores than others...

Mixing and matching multiple value streams

To allow projects to be somewhat comparable in priority values, an additional rule has been made in the scoring methodology: value streams must have a comparable amount of scoring criteria (value drivers), and the total value of all criteria must be fixed (as was already mentioned before). So if there are four scoring criteria and the total value is fixed at 20, then one value stream can have its criteria at (5,3,8,4) while another has it at (5,5,5,5).

This is still not fully adequate, as one value stream could use a single criteria with the maximum amount (20,0,0,0). However, we elected not to put in an additional constraint, and have management work things out if the situation ever comes out. Luckily, even managers are just human and they tend to follow the notion of well-balanced value drivers.

The result is that two projects will have priority values that are currently sufficiently comparable to allow cross-value-stream experts to be exchangeable without monopolizing these important resources to a single value stream portfolio.

Current state

The scoring methodology has been around for a few years already. Initially, it had fixed scoring criteria used by three value streams (out of seven, the other ones did not use the same methodology), but this year we switched to support both value stream agnostic criteria (like in the past) as well as value stream specific ones.

The methodology is furthest progressed in one value stream (with focus of around 1000 projects) and is being taken up by two others (they are still looking at what their stream-specific criteria are before switching).

July 16, 2017
Michał Górny a.k.a. mgorny (homepage, bugs)
GLEP 73 check results explained (July 16, 2017, 08:40 UTC)

The pkgcheck instance run for the Repo mirror&CI project has finished gaining a full support for GLEP 73 REQUIRED_USE validation and verification today. As a result, it can report 5 new issues defined by that GLEP. In this article, I’d like to shortly summarize them and explain how to interpret and solve the reports.

Technical note: the GLEP number has not been formally assigned yet. However, since there is no other GLEP request open at the moment, I have taken the liberty of using the next free number in the implementation.

GLEP73Syntax: syntax violates GLEP 73

GLEP 73 specifies a few syntax restrictions as compared to the pretty much free-form syntax allowed by the PMS. The restrictions could be shortly summarized as:

  • ||, ^^ and ?? can not not be empty,
  • ||, ^^ and ?? can not not be nested,
  • USE-conditional groups can not be used inside ||, ^^ and ??,
  • All-of groups (expressed using parentheses without a prefix) are banned completely.

The full rationale for the restrictions, along with examples and proposed fixes is provided in the GLEP. For the purpose of this article, it is enough to say that in all the cases found, there was a simpler (more obvious) way of expressing the same constraint.

Violation of this syntax prevents pkgcheck from performing any of the remaining checks. But more importantly, the report indicates that the constraint is unnecessarily complex and could result in REQUIRED_USE mismatch messages that are unnecessarily confusing to the user. Taking a real example, compare:

  The following REQUIRED_USE flag constraints are unsatisfied:
    exactly-one-of ( ( !32bit 64bit ) ( 32bit !64bit ) ( 32bit 64bit ) )

and the effect of a valid replacement:

  The following REQUIRED_USE flag constraints are unsatisfied:
	any-of ( 64bit 32bit )

While we could debate about usefulness of the Portage output, I think it is clear that the second output is simpler to comprehend. And the best proof is that you actually need to think a bit before confirming that they’re equivalent.

GLEP73Immutability: REQUIRED_USE violates immutability rules

This one is rather simple: it means this constraint may tell user to enable (disable) a flag that is use.masked/forced. Taking a trivial example:

a? ( b )

GLEP73Immutability report will trigger if a profile masks the b flag. This means that if the user has a enabled, the PM would normally tell him to enable b as well. However, since b is masked, it can not be enabled using normal methods (we assume that altering use.mask is not normally expected).

The alternative is to disable a then. But what’s the point of letting user enable it if we afterwards tell him to disable it anyway? It is more friendly to disable both flags together, and this is pretty much what the check is about. So in this case, the solution is to mask a as well.

How to read it? Given the generic message of:

REQUIRED_USE violates immutability rules: [C] requires [E] while the opposite value is enforced by use.force/mask (in profiles: [P])

It indicates that in profiles P (a lot of profiles usually indicates you’re looking for base or top-level arch profile), E is forced or masked, and that you probably need to force/mask C appropriately as well.

GLEP73SelfConflicting: impossible self-conflicting condition

This one is going to be extremely rare. It indicates that somehow the REQUIRED_USE nested a condition and its negation, causing it to never evaluate to true. It is best explained using the following trivial example:

a? ( !a? ( b ) )

This constraint will never be enforced since a and !a can not be true simultaneously.

Is there a point in having such a report at all? Well, such a thing is extremely unlikely to happen. However, it would break the verification algorithms and so we need to account for it explicitly. Since we account for it anyway and it is a clear mistake, why not report it?

GLEP73Conflict: request for conflicting states

This warning indicates that there are at least two constraints that can apply simultaneously and request the opposite states for the same USE flag. Again, best explained on a generic example:

a? ( c ) b? ( !c )

In this example, any USE flag set with both a and b enabled could not satisfy the constraint. However, Portage will happily led us astray:

  The following REQUIRED_USE flag constraints are unsatisfied:
	a? ( c )

If we follow the advice and enable c, we get:

  The following REQUIRED_USE flag constraints are unsatisfied:
	b? ( !c )

The goal of this check is to avoid such a bad advices, and to require constraints to clearly indicate a suggested way forward. For example, the above case could be modified to:

a? ( !b c ) b? ( !c )

to indicate that a takes precedence over b, and that b should be disabled to avoid the impossible constraint. The opposite can be stated similarly — however, note that you need to reorder the constraints to make sure that the PM will get it right:

b? ( !a !c ) a? ( c )

How to read it? Given the generic message of:

REQUIRED_USE can request conflicting states: [Ci] requires [Ei] while [Cj] requires [Ej]

It means that if the user enables Ci and Cj simultaneously, the PM will request conflicting Ei and Ej. Depending on the intent, the solution might involve negating one of the conditions in the other constraint, or reworking the REQUIRED_USE towards another solution.

GLEP73BackAlteration: previous condition starts applying

This warning is the most specific and the least important from all the additions at the moment. It indicates that the specific constraint may cause a preceding condition to start to apply, enforcing additional requirements. Consider the following example:

b? ( c ) a? ( b )

If the user has only a enabled, the second rule will enforce b. Then the condition for the first rule will start matching, and additionally enforce c. Is this a problem? Usually not. However, for the purpose of GLEP 73 we prefer that the REQUIRED_USE can be enforced while processing left-to-right, in a single iteration. If a previous rule starts applying, we may need to do another iteration.

The solution is usually trivial: to reorder (swap) the constraints. However, in some cases developers seem to prefer copying the enforcements into the subsequent rule, e.g.:

b? ( c ) a? ( b c )

Either way works for the purposes of GLEP 73, though the latter increases complexity.

How to read it? Given the generic message of:

REQUIRED_USE causes a preceding condition to start applying: [Cj] enforces [Ej] which may cause preceding [Ci] enforcing [Ei] to evaluate to true

This indicates that if Cj is true, Ej needs to be true as well. Once it is true, a preceding condition of Ci may also become true, adding another requirement for Ei. To fix the issue, you need to either move the latter constraint before the former, or include the enforcement of Ei in the rule for Cj, rendering the application of the first rule unnecessary.

Constructs using ||, ^^ and ?? operators

GLEP 73 specifies a leftmost-preferred behavior for the ||, ^^ and ?? operators. It is expressed in a simple transformation into implications (USE-conditional groups). Long story short:

  • || and ^^ groups force the leftmost unmasked flag if none of the flags are enabled already, and
  • ?? and ^^ groups disable all but the leftmost enabled flag if more than one flag is enabled.

All the verification algorithms work on the transformed form, and so their output may list conditions resulting from it. For example, the following construct:

|| ( a b c ) static? ( !a )

will report a conflict between !b !c ⇒ a and static ⇒ !a. This indicates the fact that per the forementioned rule, || group is transformed into !b? ( !c? ( a ) ) which explains that if none of the flags are enabled, the first one is preferred, causing a conflict with the static flag.

In this particular case you could debate that the algorithm could choose b or c instead in order to avoid the problem. However, we determined that this kind of heuristic is not a goal for GLEP 73, and instead we always obide the developer’s preference expressed in the ordering. The only exception to this rule is when the leftmost flag can not match due to a mask, in which case the first unmasked flag is used.

For completeness, I should add that ?? and ^^ blocks create implications in the form of: a ⇒ !b !c…, b ⇒ !c… and so on.

At some point I might work on making the reports include the original form to avoid ambiguity.

The future

The most important goal for GLEP 73 is to make it possible for users to install packages out-of-the-box without having to fight through mazes of REQUIRED_USE, and for developers to use REQUIRED_USE not only sparingly but whenever possible to improve the visibility of resulting package configuration. However, there is still a lot of testing, some fixing and many bikesheds before that could happen.

Nevertheless, I think we can all agree that most of the reports produced so far (with the exception of the back-alteration case) are meaningful even without automatic enforcing of REQUIRED_USE, and fixing them would benefit our users already. I would like to ask you to look for the reports on your packages and fix them whenever possible. Feel free to ping me if you need any help with that.

Once the number of non-conforming packages goes down, I will convert the reports successively into warning levels, making the CI report new issues and the pull request scans proactively complain about them.

July 14, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)
Expat 2.2.2 released (July 14, 2017, 17:32 UTC)

(This article first appeared on XML.com.)

A few weeks after release 2.2.1 of the free software XML parsing library Expat, version 2.2.2 now improves on few rough edges (mostly related to compilation) but also fixes security issues.

Windows binaries compiled with _UNICODE now use proper entropy for seeding the SipHash algorithm. On Unix-like platforms, accidentally missing out on high quality entropy sources is now prevented from going unnoticed: It would happen when some other build system than the configure script was used, e.g. the shipped CMake one or when the source code was copied into some parent project’s build system without paying attention to the new compile flags (that the configure script would auto-detect for you). After some struggle with a decision about C99, Expat requires a C99 compiler now; 18 years after its definition, that’s a defendable move. The uint64_t type and ULL integer literals (unsigned long long) for SipHash made us move.

Expat would like to thank the community for the bug reports and patches that went into Expat 2.2.2. If you maintain a bundled copy of Expat somewhere, please make sure it gets updated.

Sebastian Pipping
for the Expat development team

July 12, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Google-Summer-of-Code-day20 (July 12, 2017, 08:54 UTC)

Google Summer of Code day 20

What was my plan for today?

  • work on the livepatch downloader and make the kpatch creator flexible

What i did today?

  • Created .travis.yml for validating changes https://github.com/aliceinwire/elivepatch/blob/master/.travis.yml
  • Finished making the live patch downloader https://github.com/aliceinwire/elivepatch/commit/6eca2eec3572cad0181b3ce61f521ff40fa85ec1
  • Testing elivepatch

The POC generally works but I had a problem with building the Linux kernel 4.9.29 on my notebook One problem with the POC is that still some variable are hard coded.

WARNING: Skipping gcc version matching check (not recommended)
Skipping cleanup
Using source directory at /usr/src/linux-4.9.29-gentoo
Testing patch file
checking file fs/exec.c
Hunk #1 succeeded at 238 (offset -5 lines).
Reading special section data
Building original kernel
Building patched kernel
Extracting new and modified ELF sections
/usr/libexec/kpatch/create-diff-object: ERROR: exec.o: find_local_syms: 136: find_local_syms for exec.c: found_none
ERROR: 1 error(s) encountered. Check /root/.kpatch/build.log for more details.

the function find_local_syms https://github.com/dynup/kpatch/blob/master/kpatch-build/lookup.c#L80

Now i'm rebuilding everything with debug options for see some more useful information I'm also thinking to add a debug option to the elivepatch server

One question is if can be useful to work on making a feature for getting the kernel version from the Kernel configuration file header.

like this:

.config
#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 4.9.29-gentoo Kernel Configuration
#

like parsing this for get the version file without need to give it manually.

Another option is to passing it by rest as command line option.

something like -g 4.9.29

Interesting thing is that as now kernel-build have already embedded some way of dealing with most problems, and works better with distribution like ubuntu or fedora.

like for example is already copying the .config file and building the kernel with the option that we are giving from the rest api. cp -f /home/alicef/IdeaProjects/elivepatch/elivepatch_server/config /usr/src/linux-4.9.29-gentoo/.config

and the patch cp /home/alicef/IdeaProjects/elivepatch/elivepatch_server/1.patch kpatch.patch

Is also checking the .config for missing configurations. grep -q CONFIG_DEBUG_INFO_SPLIT=y /home/alicef/IdeaProjects/elivepatch/elivepatch_server/config

what i will do next time?
* Testing elivepatch * Getting the kernel version dynamically * Updating kpatch-build for work with Gentoo better

July 11, 2017
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)
Best sushi in St. Louis? J Sushi in Arnold. (July 11, 2017, 04:09 UTC)

As a self-proclaimed connoisseur of Asian cuisine, I’m constantly searching out the best restaurants in Saint Louis of the various regions and genres (Thai, Japanese, Vietnamese, as well as sushi, dim sum, et cetera). Having been to many of the staples of St. Louis sushi—Drunken Fish, Kampai, Wasabi, Cafe Mochi, and others—I’ve always been satisfied with their offerings, but yet felt like they missed the mark in one way or another. Don’t get me wrong, all of those places have some great dishes, but I just found them to be lacking that spark to make them stand out as the leader of the pack.

… and then it happened. One day when I was driving north on 61-67 (Jeffco Boulevard / Lemay Ferry), I noticed that the storefront in Water Tower Place that previously housed a mediocre Thai restaurant was set to reopen as a sushi joint. My first thought was “oh no, that’s probably not going to go over well in Arnold” but I hoped for the best. A couple weeks later, it opened as J Sushi. I added it to my ever-growing list of restaurants to try, but didn’t actually make it in for several more weeks.

Salmon Killer roll at J Sushi in St. Louis, MO
The Salmon Killer Roll with spicy crab, asparagus, salmon, cream cheese, mango sauce and Japanese mayo
(click for full quality)

Named for the original owner, Joon Kim, (who, as of this writing, is the owner of Shogun in Farmington, MO), J Sushi came onto the scene offering a huge variety of Japanese fare. From a smattering of traditional appetisers like tempura and gyoza, to a gigantic list of rolls and sashimi, to the “I don’t particularly care for raw fish” offerings in their Bento boxes, J Sushi offers dishes to appease just about anyone interested in trying Japanese cuisine.

Since their initial opening, some things have changed at J Sushi. One of the biggest differences is that it is now owned by an employee that Joon himself trained in the ways of sushi over the years: Amanda, and her partner, Joseph. The two of them have taken an already-outstanding culinary experience and elevated it even further with their immediately noticeable hospitality and friendliness (not to mention, incredible aptitude for sushi)!

VIP roll at J Sushi in St. Louis, MO
The VIP Roll with seared salmon, and shrimp tempura… it’s on fire!
(click for full quality)

So, now that you have a brief history of the restaurant, let’s get to the key components that I look for when rating eateries. First and foremost, the food has to be far above par. I expect the food to not only be tasty, but also a true representation of the culture, elegantly plated, and creative. J Sushi delivers on all four of those aspects! I’ve had many of their appetisers, rolls, sushi/sashimi plates, and non-fish dishes, and have yet to find one that wasn’t good. Of course I have my favourites, but so far, nothing has hit the dreaded “do not order again” list. As for plating, the sushi chefs recognise that one firstly eats with the eyes. Dishes are presented in a clean fashion and many of them warrant taking a minute to appreciate them visually before delving in with your chopsticks.

Second, the service has to be commendable. At J Sushi, Amanda, Joe, and the members of the waitstaff go out of their way to greet everyone as they come in and thank them after their meal. The waiters and waitresses come to the table often to check on your beverages, and to see if you need to order anything else. At a sushi restaurant, it’s very important to check for reorders as it’s commonplace to order just a couple rolls at a time. I can imagine that one of the complaints about the service is how long it takes to get your food after ordering. Though it is a valid concern, great sushi is intricate and takes time to execute properly. That being said, I have personally found the wait times to be completely acceptable, even when they’re really busy with dine-ins and take-away orders.

Mastercard roll at J Sushi in St. Louis, MO
The Master Card Roll with shrimp tempura, and gorgeously overlapped tuna, salmon, & mango
(click for full quality)

Third, the restaurant has to be a good value. Does that mean that it has to be inexpensive? No, not at all. When I’m judging a restaurant’s value, I take into consideration the quality of the ingredients used, the time and labour involved in preparation, the ambience, and the overall dining experience. J Sushi, in my opinion, excels in all of these areas, and still manages to keep their prices affordable. Yes, there are cheaper places to get sushi, and even some that offer “all you can eat” options, but you’re certainly exchanging quality for price at those types of establishments. I, for one, would rather pay a little more money to ensure that I’m getting very high quality fish (especially since the flavours and textures of the fish are exponentially heightened when consumed raw).

The Dragon Bowl at J Sushi in St. Louis, MO
The stunningly beautiful Dragon Bowl – as much artwork as it is food!
(click for full quality)

Now for the meat and potatoes (or in this case, the fish and rice): what dishes stand out to me? As I previously said, I haven’t found anything that I dislike on the menu; just dishes that I like more than others. I enjoy changing up my order and trying new things, but there are some items that I keep going back to time and time again. Here are some of my absolute favourites:

Appetisers:

  • Japanese Crab Rangoon
    • Expecting those Chinese-style fried wontons filled with cream cheese? Think again. This amazing “roll” has spicy pulled crab and cream cheese wrapped in soy paper (Mamenori) and rice. It’s deep-fried and served with eel sauce. NOT to be missed!
  • Tuna Tataki
    • Perfectly seared (read: “nearly raw”) tuna served with shredded radish and a light sauce.

Rolls:

  • Master Card Roll
    • Shrimp tempura and spicy tuna inside, topped with fresh tuna, salmon, and slices of mango (see the photo above).
  • Sweet Ogre Roll
    • One of my original favourites, this roll has shrimp tempura and cucumber inside. On top, there’s seared tuna, Sriracha, a little mayo, crunch, and masago.
  • Missouri Thunder Roll
  • Derby Roll
    • Spicy crab and avocado (I swap that for cucumber). Topped with eight beautifully-grilled shrimp.
  • Poison Spider Roll
    • HUGE, double-stuffed roll with a whole deep fried soft-shell crab and cucumber. On top, a bunch of spicy pulled crab, masago, crunch, and eel sauce.

Other:

  • Tai Nigiri
    • Simple Nigiri of Red Snapper
  • Hamachi Nigiri
    • Simple Nigiri of Yellowtail
  • Sushi sampler
    • 5 pieces of various Nigiri (raw fish on rice with a little wasabi)

If your mouth isn’t watering by now, then you must not care all that much for sushi (or Pavlov was sorely misguided 🙂 ). I hope that you try some of the amazing food that I mentioned above, but more importantly, I hope that you check out J Sushi and find your the dishes that speak to you personally!

Cheers,
Zach

Important!

The photographs in this post were taken by me. If you would like to use them elsewhere, please just give credit to Nathan Zachary and link back to my blog. Thanks!

June 27, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Open Source Summit Japan-2017 (June 27, 2017, 13:57 UTC)

Open Source Summit Japan 2017 summary

OSS Japan 2017 was a really great experience.

I sended my paper proposal and waited for a replay, some week after I got a
invite to partecipate at the Kernel Keynote.
I thought partecipating at the Kernel Keynote as mentor and doing a presentation
was a good way to talk about Gentoo Kernel Project and how to contribute in the
Linux Kernel and Gentoo Kernel Project.
Also my paper got accepted so I could join OSS Japan 2017 as speaker.
It was three really nice days.

Presentation:

Fast Releasing and Testing of Gentoo Kernel Packages and Future Plans of the Gentoo Kernel Project

My talk was manly about the Gentoo Kernel related Projects past and future
specifically about the Gentoo Kernel Continuos Integreting system we are creating:
https://github.com/gentoo/Gentoo_kernelCI

Why is needed:

  • We need some way for checking the linux-patches commits automatically, can also check pre-commit by pushing to a sandbox branch
  • Check the patches signatures
  • Checking the ebuild committed to https://github.com/gentoo/gentoo/commits/master/sys-kernel
  • Checking the kernel eclass commits
  • Checking the pull request to the sys-kernel/*
  • Use Qemu for testing kernel vmlinux correct execution

For any issue or contribution feel free to send here:
https://github.com/gentoo/Gentoo_kernelCI

For see Gentoo Kernel CI in action:
http://kernel1.amd64.dev.gentoo.org:8010

slides:
http://schd.ws/hosted_files/ossjapan2017/39/Gentoo%20Kernel%20recent%20and%20Future%20project.pdf

Open Source Summit Japan 2017
Keynote: Linux Kernel Panel - Moderated by Alice Ferrazzi, Gentoo Kernel Project Leader

The keynote was with:
Greg Kroah-Hartman - Fellow, Linux Foundation
Steven Rostedt - VMware
Dan Williams - Intel Open Source Technology Center
Alice Ferrazzi - Gentoo Kernel Project Leader, Gentoo

One interesting part was about how to contribute to the Linux Kernel.
After some information about Linux Kernel contribution numbers the talk moved on
ho to contribute in the Linux Kernel.
For contribute in the Linux Kernel there is need of some understanding of C
and running test in the Linux Kernel.
Like fuego, kselftest, coccinelle, and many others.
And also a good talk from Steven Rostedt about working with Real-Time patch.

Who can find the Gentoo logo in this image:

June 26, 2017
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

With the release of Lab::Measurement 3.550 we've switched to Dist::Zilla as maintenance tool. If you're not involved in hacking Lab::Measurement, you should essentially not notice this change. However, for the authors of the package, Dist::Zilla makes it much easier to keep track of dependencies, prepare new releases, and eventually also improve and unify the documentation... At the side we've also fixed Issue 4 and Lab::Measurement should now work out of the box with recent Gnuplot on Windows again.

June 25, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Open Source Summit Japan 2017 (June 25, 2017, 17:59 UTC)

Open Source Summit Japan 2017 summary

OSS Japan 2017 was a really great experience.

I sended my paper proposal and waited for a replay, some week after I got a invite to partecipate at the Kernel Keynote. I thought partecipating at the Kernel Keynote was a good way to talk about Gentoo Kernel Project and how to contribute in the Linux Kernel and Gentoo Kernel Project. Also my paper got accepted so I could join OSS Japan 2017 as speaker.

It was three really nice days.

[[!tags draft]]

Google-Summer-of-Code-summary week04 (June 25, 2017, 17:59 UTC)

Google Summer of Code summary week 04

What I did in this week 04 summary:

elivepatch:

  • Created the elivepatch client command line argument parser
  • Added function for sending patch and configuration files
  • Divided the call for sending (patch, config) and the call for building the livepatch
  • made send_file function more generic for sending all kind of files using RESTful api
  • Cleaned code following pep8
  • Decided to use only SSL and to don't use basic auth
  • Sending informations about the kernel version when requesting a livepatch build
  • We can now build livepatch using the RESTful API
  • Returning information about the livepatch building status

Kpatch:

  • Working on making kpatch-build working also with gentoo with all the features (As now kpatch-build can only automatically build livepatch for Ubuntu, Debian, Red Hat, Fedora)

Others:

  • Ask infra for a server for install the elivepatch server

What I need to do next time:

  • Finish the function for download the livepatch to the client
  • Testing elivepatch
  • Implementing the CVE patch uploader
  • Installing elivepatch to the Gentoo server
  • Fix kpatch-build for automatically work with gentoo-sources
  • Add more features to elivepatch

Google-Summer-of-Code-day18 (June 25, 2017, 17:59 UTC)

Google Summer of Code day 18

What was my plan for today?

  • going on with the code for retriving the livepatch and installing it

What i did today?

checked about kpatch-build required folder.

kpatch-build find_dirs function:

find_dirs() {
  if [[ -e "$SCRIPTDIR/create-diff-object" ]]; then
      # git repo
      TOOLSDIR="$SCRIPTDIR"
      DATADIR="$(readlink -f $SCRIPTDIR/../kmod)"
  elif [[ -e "$SCRIPTDIR/../libexec/kpatch/create-diff-object" ]]; then
      # installation path
      TOOLSDIR="$(readlink -f $SCRIPTDIR/../libexec/kpatch)"
      DATADIR="$(readlink -f $SCRIPTDIR/../share/kpatch)"
  else
      return 1
  fi
}

$SCRIPTDIR is the kpatch-build directory. kpatch-build is installed in /usr/bin/ so /usr/kmod /usr/libexe are all under such directory.

error "CONFIG_FUNCTION_TRACER, CONFIG_HAVE_FENTRY, CONFIG_MODULES, CONFIG_SYSFS, CONFIG_KALLSYMS_ALL kernel config options are required" Require by kmod/core.c: https://github.com/dynup/kpatch/blob/master/kmod/core/core.c#L62

We probably need someway for check that this setting are configured in the kernel we are going to build.

Updating kpatch-build for work automatically with gentoo (as now fedora for example can automatically download the kernel rpm and install it, we could do similar thing with gentoo): https://github.com/aliceinwire/kpatch/commits/gentoo

Starting to write the live patch downloader: https://github.com/aliceinwire/elivepatch/commit/d26611fb898223f2ea2dcf323078347ca928cbda

Now the elivepatch server can call and build the livepatch with kpatch:

sudo kpatch-build -s /usr/src/linux-4.10.14-gentoo/ -v /usr/src/linux-4.10.14-gentoo//vmlinux -c config 1.patch --skip-gcc-check
ERROR: kpatch build failed. Check /root/.kpatch/build.log for more details.
127.0.0.1 - - [25/Jun/2017 05:27:06] "POST /elivepatch/api/v1.0/build_livepatch HTTP/1.1" 201 -
WARNING: Skipping gcc version matching check (not recommended)
Using source directory at /usr/src/linux-4.10.14-gentoo
Testing patch file
checking file fs/exec.c
Hunk #1 succeeded at 259 (offset 16 lines).
Reading special section data
Building original kernel

Fixed some minor pep8

what i will do next time?
* work on the livepatch downloader and make the kpatch creator flexible

June 23, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Google-Summer-of-Code-day17 (June 23, 2017, 23:12 UTC)

Google Summer of Code day 16

What was my plan for today?

  • going on with the code for retriving the livepatch and installing it
  • Ask infra for a server where to install elivepatch sever

What i did today?
Sended request for the server that will offer the elivepatch service as talked with my mentor. https://bugs.gentoo.org/show_bug.cgi?id=622476

Fixed some pep8 warnings.

Livepatch server is now returning information about the livepatch building status.

Removed basic auth as we will go with SSL.

The client is now sending information about the kernel version when requesting a new build.

The kernel directory under the server is now a livepatch class variable.

what i will do next time?

  • going on with the code for retriving the livepatch and installing it

Google-Summer-of-Code-day16 (June 23, 2017, 23:12 UTC)

Google Summer of Code day 16

What was my plan for today?

  • Divide call for sending (patch, config) and the call for build the livepatch
  • Make the livepatch call more flexible (as now is hardcoded)
  • Ask infra for a server where to install elivepatch sever

What i did today?

Added patch file path argument to the elivepatch server API and added patch call to elivepatch client.

Adding way for dividing the call for sending the configuration with a POST call sending the patch with a POST calland than start the livepatch build and getting the result.

patch sended work and working on calling livepatch.

Added docstring to the build patch function.

Cleaned GetLive dispatcher function.

Added call from client to build livepatch of the server API.

made send_file function more generic for send all kind of file.

what i will do next time?

  • going on with the code for retriving the livepatch and installing it

June 21, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
2017-06-21 Google-Summer-of-Code-day14 (June 21, 2017, 17:57 UTC)

Google Summer of Code day 15

What was my plan for today?
working on sending the configuration file on RESTful api,
and starting to work on making the patch.ko file in the server.

What i did today?
using wekzeug.datastructures.FileStorage in elivepatch_server,
I could receive the file from the elivepatch_client POST request
using the RESTful API.

def post(self):
    parse = reqparse.RequestParser()
    parse.add_argument('file', type=werkzeug.datastructures.FileStorage, location)

so as now we can get the kernel configuration file, extract if is .gz filename and send it to the elivepatch server.

elivepatch server need to read the configuration, compare it with the
current kernel configuration and if different recompile the kernel.
After we can start making the livepatch with kpatch-build.

This is the example of using kpatch-build:

kpatch-build/kpatch-build -s /usr/src/linux-4.9.16-gentoo/ -v /usr/src/linux-4.9.16-gentoo/vmlinux examples/test.patch --skip-gcc-check
gsoc-2017 kpatch (gentoo) # kpatch-build/kpatch-build --help
usage: kpatch-build [options] <patch file>
            -h, --help         Show this help message
            -r, --sourcerpm    Specify kernel source RPM
            -s, --sourcedir    Specify kernel source directory
            -c, --config       Specify kernel config file
            -v, --vmlinux      Specify original vmlinux
            -t, --target       Specify custom kernel build targets
            -d, --debug        Keep scratch files in /tmp
            --skip-cleanup     Skip post-build cleanup
            --skip-gcc-check   Skip gcc version matching check
                               (not recommended)

This command is called automatically by the elivepatch server after receiving the configuration file.

we need also to send the patch file.

what i will do next time?

  • Divide call for sending (patch, config) and the call for build the livepatch
  • Make the livepatch call more flexible (as now is hardcoded)
  • Ask infra for a server where to install elivepatch sever

Nathan Zachary a.k.a. nathanzachary (homepage, bugs)
The Book of Henry film review (June 21, 2017, 03:06 UTC)

Notice

Though I make every effort to not include spoilers in my reviews (personal opinions and biases, however, are fair game 🙂 ), some of the linked pages may contain them.

The Book of Henry opening title

This past weekend I went to the theatre to see the newly-released film The Book of Henry (WARNING: the IMDb and Wikipedia pages contain massive spoilers) starring Jaeden Lieberher, Jacob Tremblay, and Naomi Watts. I’ve been anxiously awaiting the release of the film because 1) the trailer piqued my interest, and 2) the cast is filled with some outstanding talent.

Without giving too much away, the basic premise is that Henry (Lieberher) is an incredibly intelligent 11-year-old who essentially plays the “head of household” role for the family. His mother Susan (Watts) has come to rely on him for many adult-oriented tasks, such as managing the finances, setting ground rules, and taking care of his younger brother Peter (Tremblay). When Henry learns that a girl in his class (who is also his neighbour) is being abused by her stepfather, he devises a plan to rescue her and his mother is tasked with helping execute that plan.

Brothers Peter and Henry Carpenter in The Book of HenryBrothers Peter and Henry Carpenter

I went into this film not knowing all that much about it (which is what I prefer: I would rather form my own impressions than start viewing it with preconceived notions and expectations). Based on the opening credits, I anticipated being introduced to a precocious kid that was wise beyond his years. I thought that it may turn into a dramatic representation of his sociocultural interactions at school and how they impacted his outlook on life. Prepared for that type of somewhat slower-paced progression, I was pleasantly surprised to find a stronger emphasis on the family dynamics between Henry, Peter, Susan, and Susan’s coworker and friend, Sheila (Sarah Silverman). As the film continues, we do get to see some of the philosophically advanced aspects of Henry’s worldview, but more importantly we see—through his focus on making the world a better place and his desire for justice—the love that he has for other people.

 

Our legacy isn’t what we write on a résumé or how many commas we have in our bank account. It’s who we’re lucky enough to have in our lives, and what we can leave them with. The one thing we do know: we’re here now, so I say we do the best we can while we’re on this side of the dirt.—Henry Carpenter

Henry and his mother discuss helping othersHenry and his mother discussing helping others

Is that what sets The Book of Henry apart from other films? No, not at all. There are plenty of examples of films where a child protagonist is longing to make a difference in the world (including another one of Liberher’s recent roles [and Watts’s, for that matter] in St. Vincent). Instead, what makes this film special is the way in which it is able to elicit the full gamut of human emotion in such a strong, sweeping fashion. For instance, in one scene we’re reminded of the frivolity of childhood by seeing Henry and Peter playing and inventing in their elaborate fort in the woods behind their home. Thereafter, we see some of the strain on Susan as she tries to balance her own needs and wants with doing the best that she in can in raising her two boys (as illustrated in a scene where she may have slightly overindulged with her friend Sheila 😉 ). Following that somewhat more serious tone comes a tender moment with Susan putting Henry and Peter to bed by singing a song to them and playing the ukulele. It’s this juxtaposition of emotional evocation that elevates The Book of Henry to levels above others in the genre.

Henry and Peter in their backyard fortHenry and Peter in their backyard fort

Focusing again on emotionality, there have been very few movies over the years that have actually made me cry (for the curious, Radio Flyer, Life is Beautiful, and The Fountain are a few). I’ve been misty-eyed before, but there were a few scenes in this film that actually brought me to tears. One in particular was so gut-wrenching for me that I thought I might have to briefly excuse myself from the theatre (thankfully, though, I toughened up). Will everyone react in the same way to these types of scenes? No, but that’s one of the beautiful bonds between well-made films and its viewers.

Even with the moments that were extremely difficult to watch due to their forlorn sentiments, there were plenty of convivial parts that provided an overall undertone of warmth throughout the movie. One key example that sticks out in my mind is at the Calvary Elementary School talent show where Peter performs a magic trick. He comes out in full magician garb (reminiscent of one of Tremblay’s first roles in The Magic Ferret [which, by the way, is a cute short film that can be purchased through director Alison Parker’s website]), and wows the audience all the while smiling ear-to-ear. These types of scenes provided the much needed comic relief from the overarching sentimentality.

The Book of Henry - Jacob Tremblay as Peter the GreatJacob Tremblay as Peter the Great

It is likely this roller coaster ride of emotions (or some of the unusual/unresolved details) that led to the overwhelmingly negative Rotten Tomatoes Critics’ Consensus that the movie “deserves a few points for ambition, but its tonal juggling act—and a deeply maudlin twist—may leave viewers gaping in disbelief rather than choking back tears”. Again, that’s the beauty of the world of artistic expression (which, of course, includes film): it can speak to different people in vastly different ways based on what they individually bring to the table. I, for one, thought that it was beautifully done and in no way “maudlin”. Instead of relying solely on the reviews (including this one) of professional critics or viewers, though, I urge you to go see it for yourself and make your own assessment.

8 out of 10 stars:
Filled starFilled starFilled starFilled starFilled starFilled starFilled starFilled starUnfilled starUnfilled star

Cheers,
Zach

June 17, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)

Expat 2.2.1 has been released. It’s a security release with a variety of security fixes, for instance: An infinite loop denial-of-service fix (that Rhodri James wrote more about), introduction of SipHash against sophisticated hash flooding, use of OS-specific high quality entropy providers like getrandom, integer overflow fixes, and more. We also got better code coverage, moved all but the downloads from SourceForge to GitHub, … but maybe have a look at the detailed change log yourself 🙂

So if you control copies of Expat somewhere, please get them updated.

Let me use the occasion to point out that we are looking for help with a few things Expat. There are tickets with details up here. If you can help, please get in touch.

Thanks and best

 

Sebastian

June 16, 2017
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Recently, Gentoo Linux put GCC 6.3 (released in December 2016) into the testing branch. For a source-based distribution like Gentoo, GCC is a critical part of the toolchain, and sometimes can lead to packages failing to compile or run. I recently ran into just this problem with Audacity. The error that I hit was not during compilation, but during runtime:


$ audacity
Fatal Error: Mismatch between the program and library build versions detected.
The library used 3.0 (wchar_t,compiler with C++ ABI 1009,wx containers,compatible with 2.8),
and your program used 3.0 (wchar_t,compiler with C++ ABI 1010,wx containers,compatible with 2.8).
Aborted

The Gentoo Wiki has a nice, detailed page on Upgrading GCC, and explicitly calls out ABI changes. In this particular case of Audacity, the problematic library is referenced in the error above: “wx containers”. WX containers are handled by the wxGTK package. So, I simply needed to rebuild the currently-installed version of wxGTK to fix this particular problem.

Hope that helps.

Cheers,
Zach

June 15, 2017
Hanno Böck a.k.a. hanno (homepage, bugs)
Don't leave Coredumps on Web Servers (June 15, 2017, 09:20 UTC)

CoreCoredumps are a feature of Linux and other Unix systems to analyze crashing software. If a software crashes, for example due to an invalid memory access, the operating system can save the current content of the application's memory to a file. By default it is simply called core.

While this is useful for debugging purposes it can produce a security risk. If a web application crashes the coredump may simply end up in the web server's root folder. Given that its file name is known an attacker can simply download it via an URL of the form http://example.org/core. As coredumps contain an application's memory they may expose secret information. A very typical example would be passwords.

PHP used to crash relatively often. Recently a lot of these crash bugs have been fixed, in part because PHP now has a bug bounty program. But there are still situations in which PHP crashes. Some of them likely won't be fixed.

How to disclose?

With a scan of the Alexa Top 1 Million domains for exposed core dumps I found around 1.000 vulnerable hosts. I was faced with a challenge: How can I properly disclose this? It is obvious that I wouldn't write hundreds of manual mails. So I needed an automated way to contact the site owners.

Abusix runs a service where you can query the abuse contacts of IP addresses via a DNS query. This turned out to be very useful for this purpose. One could also imagine contacting domain owners directly, but that's not very practical. The domain whois databases have rate limits and don't always expose contact mail addresses in a machine readable way.

Using the abuse contacts doesn't reach all of the affected host operators. Some abuse contacts were nonexistent mail addresses, others didn't have abuse contacts at all. I also got all kinds of automated replies, some of them asking me to fill out forms or do other things, otherwise my message wouldn't be read. Due to the scale I ignored those. I feel that if people make it hard for me to inform them about security problems that's not my responsibility.

I took away two things that I changed in a second batch of disclosures. Some abuse contacts seem to automatically search for IP addresses in the abuse mails. I originally only included affected URLs. So I changed that to include the affected IPs as well.

In many cases I was informed that the affected hosts are not owned by the company I contacted, but by a customer. Some of them asked me if they're allowed to forward the message to them. I thought that would be obvious, but I made it explicit now. Some of them asked me that I contact their customers, which again, of course, is impractical at scale. And sorry: They are your customers, not mine.

How to fix and prevent it?

If you have a coredump on your web host, the obvious fix is to remove it from there. However you obviously also want to prevent this from happening again.

There are two settings that impact coredump creation: A limits setting, configurable via /etc/security/limits.conf and ulimit and a sysctl interface that can be found under /proc/sys/kernel/core_pattern.

The limits setting is a size limit for coredumps. If it is set to zero then no core dumps are created. To set this as the default you can add something like this to your limits.conf:

* soft core 0

The sysctl interface sets a pattern for the file name and can also contain a path. You can set it to something like this:

/var/log/core/core.%e.%p.%h.%t

This would store all coredumps under /var/log/core/ and add the executable name, process id, host name and timestamp to the filename. The directory needs to be writable by all users, you should use a directory with the sticky bit (chmod +t).

If you set this via the proc file interface it will only be temporary until the next reboot. To set this permanently you can add it to /etc/sysctl.conf:

kernel.core_pattern = /var/log/core/core.%e.%p.%h.%t

Some Linux distributions directly forward core dumps to crash analysis tools. This can be done by prefixing the pattern with a pipe (|). These tools like apport from Ubuntu or abrt from Fedora have also been the source of security vulnerabilities in the past. However that's a separate issue.

Look out for coredumps

My scans showed that this is a relatively common issue. Among popular web pages around one in a thousand were affected before my disclosure attempts. I recommend that pentesters and developers of security scan tools consider checking for this. It's simple: Just try download the /core file and check if it looks like an executable. In most cases it will be an ELF file, however sometimes it may be a Mach-O (OS X) or an a.out file (very old Linux and Unix systems).

Image credit: NASA/JPL-Université Paris Diderot

June 13, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Moving back to ikiwiki (June 13, 2017, 03:40 UTC)

I was trying to use blogs.gentoo.org/alicef/ Gentoo official blog
based on wordpress.
As far as I could like the draft feature, it had some big drawback.
Most big one I couldn't post any syntax highlighted code.
And wordpress maintenance takes lots of time, in particular managing plugins.
Also because I cannot change Gentoo blog plugins without admin privilege, is bit too much to have to ask every time I have problem with plugins or I need a new one.

So I decided to come back to ikiwiki.
Also in ikiwiki I can make some sort of draft function where post tagged as draft dosen't come up on the blog list.
This is simply done by using this on the blog.mdwn
pages="page(blog/) and !/Discussion and !*/local.css and !tagged(draft)"
That will remove the blog page tagged with draft from the blog view.

For the syntax highlight I used a plugin for pygments.py made by
tylercipriani.com
That you can find here pygments.pm

Pygments.pm output example:

#include <stdio.h>

int main(void) {
    // your code goes here
    return 0;
}

And last one, on Ikiwiki comments, I decided to delegate comments spam to disqus, also if I don't like so much to use a private business plugin, do is job of managing comments well enough.
I also just discovered that disqus is part of Y combinator, the company behind hacker news.

[2017 05] Gentoo activity summary (June 13, 2017, 01:49 UTC)

June 12, 2017
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

I've just uploaded Lab::Measurement 3.544 to CPAN. This is our first release containing support for Zurich Instruments equipment, in particular the ZI MFLI digital lock-in amplifier, via Simon's Lab::Zhinst package. Enjoy!

June 11, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)

About rel=noopener (mathiasbynens.github.io)