Here's what happened in the Reproducible Builds effort between Sunday February 19 and Saturday February 25 2017:

Reproducible work in other projects

• Christos Zaulas wrote a blog post entitled "NetBSD fully reproducible builds" which generated some discussion on Hacker News and was mentioned on DistroWatch.

• Christos also reported that that NetBSD's base system is now 100.0% reproducible in our current test framework. Whilst we do less variations than in Debian here, this is highly commendable.

• Microsoft requires reproducible binaries for signing shim code for secure boot.

• Thanks to Ed Maste, FreeBSD base system reached 99.6% too, after he'd seen the news about NetBSD. This too has been achieved using non-default settings and is to be considered merely as a "hey we can do that too" (though Ed is slightly sad they missed 100%). The real plan is still to achieve 100% reproducibility with default settings.

Upcoming Events

Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.

On March 23rd Holger Levsen will give a talk at the German Unix User Group's "Frühjahrsfachgespräch" about Reproducible Builds everywhere.

Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Packages reviewed and fixed, and bugs filed

Chris Lamb:

• #855674 filed against sugar-physics-activity.
• #855909 filed against publicsuffix.

Reviews of unreproducible packages

9 package reviews have been added, 3 have been updated and 1 has been removed in this week, adding to our knowledge about identified issues.

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (4)

diffoscope development

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

• diffoscope 77 was unblocked by the release team for stretch.
• Mattia Rizzolo uploaded 77~bpo8+1 to jessie-backports.

buildinfo.debian.net development

buildinfo.debian.net is our experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them.

• Chris Lamb:
  • Drop orphaned binary package instances.
  • Install to /usr/share with dh-virtualenv 1.0
  • Drop Installed-Build-Depends to recover diskspace.

Website development

• Holger Levsen:
  • Add link to https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds

tests.reproducible-builds.org

• Ed Maste made the upcoming FreeBSD release almost 100% reproducible (see above).
• Holger Levsen added the number of configured and running builder jobs to the performance stats page.
• Holger Levsen improved the scheduler, so that untested packages and versions are tried sooner.
• Holger added logging for submitting .buildinfo files to `buildinfo.debian.net and added notification about this failure.
• Holger also made some minor improvements to the generated HTML.

Misc.

This week's edition was written by Chris Lamb, Ed Maste & Levsen and reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday February 12 and Saturday February 18 2017:

Upcoming Events

The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.

Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.

Toolchain development and fixes

Ximin Luo posted a preliminary spec for BUILD_PATH_PREFIX_MAP, bringing together work and research from previous weeks.

Ximin refactored and consolidated much of our existing documentation on both SOURCE_DATE_EPOCH and BUILD_PATH_PREFIX_MAP into one unified page, Standard Environment Variables, with extended discussion on related solutions and how these all fit into people's ideas of what reproducible builds should look like in the long term. The specific pages for each variable still remain, at Timestamps Proposal and Build Path Proposal, only without content that was previously duplicated on both pages.

Ximin filed #855282 against devscripts for debsign(1) to support buildinfo files, and wrote an initial series of patches for it with some further additions from Guillem Jover.

Packages reviewed and fixed, and bugs filed

Chris Lamb:

• #854999 filed against moin, and forwarded upstream.
• #855002 filed against samplv1, and forwarded upstream.
• #855426 filed against fritzing-parts.
• #855480 filed against examl.

Reviews of unreproducible packages

35 package reviews have been added, 1 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues.

1 issue type has been added:

• nondeterminism_in_java_classes_generated_by_jxc

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (2)

diffoscope development

diffoscope 77 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Chris Lamb:
  • Some fixes to tests and testing config
  • Don't track archive directory locations, a better fix for CVE-2017-0359.
  • Add --exclude option. Closes: #854783
• Mattia Rizzolo:
  • Add my key to debian/upstream/signing-key.asc
  • Add CVE-2017-0359 to the changelog of v76
• Ximin Luo:
  • When extracting archives, try to keep directory sizes small

strip-nondeterminism development

strip-nondeterminism 0.031-1 was uploaded to unstable by Chris Lamb. It included contributions from:

• Chris Lamb:
  • Make the tests less brittle, by not testing for stat(2) blksize and blocks. #854937

strip-nondeterminism 0.031-1~bpo8+1 was uploaded to jessie-backports by Mattia.

tests.reproducible-builds.org

• Vagrant Cascadian and Holger Levsen set up two new armhf nodes for Debian tests, p64b and p64c running on pine64 boards with an arm64 kernel and armhf userland. This introduces kernel variations to armhf.
• Holger also added new setup & maintenance jobs, plus 6 new builder jobs for Debian armhf.
• Hans-Christoph Steiner continued work on setting up reproducible tests for F-Droid, now with daily tests for faster progress. These tests are now also using the Android SDK from Debian/stretch packages.
• Mattia Rizzolo added IRC notification to the job testing for mismatches between diffoscope's pypi and Debian archive versions.
• Mattia also improved the tempfile handling of the Debian builder jobs.
• Since we've deployed pbuilder 0.228.4 everywhere, Mattia could also simplify the pbuilder configuration and reenable the build directory name variation for Debian reproducibility tests.

Misc.

This week's edition was written by Ximin Luo & Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday February 5 and Saturday February 11 2017:

Upcoming events

• Holger proposed a hackathon with various possible dates -- please reply with your preferred dates!

• "Reproducible builds: Status update" CfP submitted for Debconf17 in Montreal.

Patches sent upstream

• pnmixer (Chris Lamb)
• cloud-sptheme (Chris Lamb)
• python-hypothesis (Chris Lamb)
• cython (Jelmer Vernooij)

Packages reviewed and fixed, and bugs filed

Chris Lamb:

• #854332 filed against cloud-sptheme.
• #854512 filed against ftpcopy.
• #854549 filed against python-hypothesis.

Daniel Shahaf:

• #854492 filed against xlsx2csv.
• #854541 filed against sogo.

"Z. Ren":

• #854293 filed against manpages-tr.
• #854294 filed against regina-rexx.
• #854362 filed against fonts-uralic.

Reviews of unreproducible packages

83 package reviews have been added, 8 have been updated and 32 have been removed in this week, adding to our knowledge about identified issues.

5 issue types have been added:

• randomness_in_swf_files_generated_by_as3compile
• randomness_in_t3g_files_generated_tslmendian
• absolute_build_paths_in_dot_packlist_file_generated_by_perl_extutils_packlist
• formatdb_from_ncbi_blastplus_captures_build_time
• timestamp_and_build_path_captured_by_python_cheetah

1 issue type has been updated:

• build_id_differences_only

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (7)
• gregory bahde (1)

diffoscope development

diffoscope versions 71, 72, 73, 74 & 75 were uploaded to unstable by Chris Lamb:

• Chris Lamb:
  • New features:
    • Add --exclude option. (Closes: #854783)
    • Apply --max-report-size to --text reports. (Closes: #851147)
    • Add a machine-readable JSON output format. (Closes: #850791)
    • Show results from debugging packages last (Closes: #820427)
    • Specify lang="en" in HTML output. (re. #849411)
  • Bug fixes:
    • Fix errors when comparing directories with non-directories. (Closes: #835641)
    • Clean all temp files in signal handler thread instead of attempting to bubble exception back to the main thread. (Closes: #852013)
    • Correct logic of module_exists, ensuring we correctly skip tests when python3-debian is not installed. (Closes: #854745)
    • Extract archive members using an auto-incrementing integer, avoiding the need to sanitise filenames. (Closes: #854723)
    • Importing submodules (ie. parent.child) will attempt to import parent so we must catch that. (Closes: #854670)
    • Add missing Recommends for comparators. (Closes: #854655)
    • Device and RPM fallback comparisons needs xxd due to fixtures. (Closes: #854593)
    • Fix behaviour of setting report maximums to zero (ie. no limits)
  • Misc:
    • Don't uselessly run xxd(1) on non-directories.
    • Add .travis.yml from http://travis.debian.net.
    • Use diffoscope.tempfiles over tempfile.TemporaryDirectory to ensure correct cleanup at end of diffoscope run.
  • Tests:
    • Smoke test --progress output.
    • Test the --status-fd output.
    • Move many tests to use new @skip_unless_module_exists decorator.
    • Show local variables in pytest tracebacks.
• Mattia Rizzolo:
  • No need to do complex string formatting just to convert an integer in a string
• Holger Levsen:
  • README: Keep history, explain this was started in Debian.
• Ximin Luo:
  • Better way of performing the entry name sanitisation
  • Simplify call to subprocess.Popen
  • Remove pointless use of a thread
• Brett Smith:
  • diffoscope.diff: Improve FIFO writing robustness.

strip-nondeterminism development

strip-nondeterminism 0.030-1 was uploaded to unstable by Chris Lamb:

• Print log entry when fixing a file. (Closes: #777239)
• dh_strip_nondeterminism: Use error() from Dh_Lib.pm over manual die().

buildinfo.debian.net development

• Chris Lamb:
  • Drop raw_text fields now; we've moved them to default_storage (S3)

reproducible-website development

• Joshua Lock:
  • Link to Yocto Project on projects page

Misc.

This week's edition was written by Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday January 29 and Saturday February 4 2017:

Media coverage

Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" (Video, Slides) at Devconf.cz on February 27th 2017.

On February 1st, stretch/armhf reached 90% reproducible packages in our test framework, so that now all four tested architectures are ≥ 90% reproducible in stretch. Yay! For armhf this means 22472 reproducible source packages (in main); for amd64, arm64 and i386 these figures are 23363, 23062 and 22607 respectively.

Chris Lamb appeared on the Changelog podcast to talk about reproducible builds:

Holger Levsen pitched Reproducible Builds and our need for a logo in the "Open Source Design" room at FOSDEM 2017 (Video, 09:36 - 12:00).

Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.

• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.

• Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Reproducible work in other projects

We learned that the "slightly more secure" Heads firmware (a Coreboot payload) is now reproducibly built regardless of host system or build directory. A picture says more than a thousand words:

Docker started preliminary work on making image builds reproducible.

Toolchain development and fixes

Ximin Luo continued to write code and test cases for the BUILD_PATH_PREFIX_MAP environment variable. He also did extensive research on cross-platform and cross-language issues with enviroment variables, filesystem paths, and character encodings, and started preparing a draft specification document to describe all of this.

Chris Lamb asked CPython to implement an environment variable PYTHONREVERSEDICTKEYORDER to add an an option to reverse iteration order of items in a dict. However this was rejected because they are planning to formally fix this order in the next language version.

Bernhard Wiedemann and Florian Festi added support for our SOURCE_DATE_EPOCH environment variable, to the RPM Package Manager.

James McCoy uploaded devscripts 2.17.1 with a change from Guillem Jover for dscverify(1), adding support for .buildinfo files. (Closes: #852801)

Piotr Ożarowski uploaded dh-python 2.20170125 with a change from Chris Lamb for a patch to fix #835805.

Chris Lamb added documentation to diffoscope, strip-nondeterminism, disorderfs, reprotest and trydiffoscope about uploading signed tarballs when releasing. He also added a link to these on our website's tools page.

Packages reviewed and bugs filed

Bugs filed:

• "Z. Ren":
  • #854293 filed against manpages-tr.
  • #854294 filed against regina-rexx.
• Chris Lamb:
  • #853039 filed against fontypython.
  • #853912 filed against python-testfixtures, merged upstream as PR #56.
  • #854111 filed against aprx.
  • #854112 filed against pnmixer.
• Reiner Herrmann:
  • #854145 filed against daemontools.
  • #854146 filed against diploma.

Reviews of unreproducible packages

83 package reviews have been added, 86 have been updated and 276 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been updated:

• captures_build_path_via_assert
• randomness_in_documentation_generated_by_epydoc

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (6)

diffoscope development

Work on the next version (71) continued in git this week:

• Mattia Rizzolo:
  • Override a lintian warning.
• Chris Lamb:
  • Update and consolidate documentation
  • Many test additions and improvements
  • Various code quality and software architecture improvements
• anthraxx:
  • Update arch package, cdrkit -> cdrtools.

reproducible-website development

Daniel Shahaf added more notes on our "How to chair a meeting" document.

tests.reproducible-builds.org

Holger unblacklisted pspp and tiledarray. If you think further packages should also be unblacklisted (possibly only on some architectures), please tell us.

Misc.

This week's edition was written by Ximin Luo, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday January 22 and Saturday January 28 2017:

Media coverage

• The Reproducible Build Zoo was presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon on February 22nd.

• Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" at Devconf.cz on February 27th.

Upcoming Events

• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.

• "Verifying Software Freedom with Reproducible Builds" will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Reproducible work in other projects

John Gilmore wrote an interesting mail about how Cygnus.com worked on reproducible builds in the early 1990s. It's eye opening to see how the dealt with basically the very same problems we're dealing with today, how they solved them and then to realize that most of this has been forgotten and bit-rotted in the last 20 years. How will we prevent history repeating itself here?

Toolchain development and fixes

Christoph Biedl wrote a mail describing an interesting problem in to the way binNMUs are done in Debian.

Guillem Jover made a number of changes to dpkg that affect the Reproducible Builds effort within Debian:

• Always set SOURCE_DATE_EPOCH in dpkg-buildpackage and dpkg-source. Use the current date if the changelog does not have one. Closes: #849081

• Add initial support for DEB_BUILD_OPTIONS to dpkg-genbuildinfo. This will make it possible to enable or disable specific features that should be recorded in the .buildinfo file. For now only “all” and “path” are supported. Closes: #848705

• Include .buildinfo files also for source-only uploads in dpkg-genchanges. Closes: #846164

• Add support for signed .buildinfo files to dpkg-buildpackage. Add new -ui and --unsigned-buildinfo options. Closes: #843925

• Make dpkg-buildpackage --unsigned-changes not sign .buildinfo either. This breaks the expectations of users and tools, because there was no way previously to request no signing at all. Closes: #852822

Packages reviewed and fixed, and bugs filed

Chris Lamb:

• #852482 filed against flask-limiter.
• #853039 filed against fontypython.

Dhole:

• #852289 filed against python-passlib.

Reviews of unreproducible packages

17 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been added:

• python_versioneer_uses_parentdir
• captures_kernel_version_via_CMAKE_SYSTEM

1 issue type has been removed:

• ftbfs_due_to_jenkins_semaphore_setup

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (6)
• Holger Levsen (1)

diffoscope development

• diffoscope 70 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Chris Lamb:

  • tests.presenters: Prevent FTBFS by loading fixtures as UTF-8 in case surrounding terminal is not Unicode-aware. (Closes: #852926)
  • comparators: Tidy re_tests with list comprehensions and implicit "x, y" unpacking over indexing; lambda/filter is not idiomatic Python 3.
  • tests: Increase coverage by adding "# noqa" in relevant parts.
  • tests: Test that no arguments (beyond the filenames) prints the text output.
  • tests: Test --text-color output format.
  • tests: Add a test comparing two empty directories.
  • tests: Don't warn about coverage lines that raise NotImplementedError.

• anthraxx:

  • tools: switch Arch Linux dependency for pedump to mono

• Ximin Luo:

  • Fix lazy expression, filter is lazy in Python 3
  • Fix bug introduced in commit 36d1c964 that only worked "accidentally"

reprotest development

• reprotest 0.6 was uploaded to unstable by Holger Levsen. It included contributions:

• Ximin Luo:

  • Test the extra variations we added recently and ensure they don't get missed in the future
  • Better logging messages that actually get controlled by the verbosity flag
  • Add a man page using rst2man and help2man
  • Add a --config-file option and fix the loading of configs
  • Fix the reading of config options
  • Fix a bug where the sha256sum of a reproduction won't be displayed if --store-dir is not given

buildinfo.debian.net development

• Chris Lamb:
  • Show SHA256 checksums on larger displays
  • Move default storage to S3.
  • Store files directly onto S3.
  • Add tool to move data to S3.
  • Load data from S3 if we don't have it locally.

tests.reproducible-builds.org

• h01ger experimented with reusing SSH control connections but stopped that experiment when we ran into more network issues than before. To be continued, as we're having 10k SSH connections per day and saving 2 seconds each time would sum up, especially on the Jenkins host itself.
• h01ger made the scheduler run 3 times a day, 2.5h after dinstall runs, instead of every 3h as before.
• h01ger restructured the https://tests.reproducible-builds.org/debian/index_breakages.html and improved the corresponding Jenins job.
• h01ger also unblacklisted xmds2, sofia-sip and ck - if you think other packages should be unblacklisted (maybe only on some architectures), please do tell us.

Misc.

This week's edition was written by Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday January 15 and Saturday January 21 2017:

Media Coverage

• Valerie Young presented Reproducible Builds for a Better Future (video) at linux.conf.au 2017.

• Chris Lamb presented Reproducible Builds: Two years in the trenches (video) at linux.conf.au 2017.

Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.

• Dennis Gilmore and Holger Levsen will present on "Reproducible Builds and Fedora" at Devconf.cz on February 27th.

• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.

• Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Toolchain development and fixes

Ximin Luo continued work on data formats, code, and test cases for SOURCE_PREFIX_MAP. He also continued to talk with the rustc team on the topic.

Chris Lamb submitted a patch to implement SOURCE_DATE_EPOCH for wordwarvi, a game which gave extra points to people who built it from source within one hour. This fixes Debian #786593.

Launchpad bug 1657704 was filed for them to start accepting buildinfo files.

Bugs filed

• Chris Lamb:
  • #851809 filed against mono.
• Daniel Shahaf:
  • #851764 filed against hhsuite.

Reviews of unreproducible packages

10 package reviews have been added, 149 have been updated and 153 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been updated:

• ftbfs_due_to_jenkins_semaphore_setup
• clilibs_line_order

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (3)
• Ondřej Kobližek (1)

diffoscope development

diffoscope 69 was uploaded to unstable by Chris Lamb. It included contributions from:

• Maria Glukhova:
  • Bug fixes: #850055, #850501.
  • Improve comparison of APK files: #850502.
  • Add comparison of image metadata: #849395.
• Chris Lamb:
  • Many bug fixes: #849411, #850485, #850807, #850850, #851588.
  • Add comparison of ICO files: #850730.
  • Optimisations: disable profiling unless needed, and pre-compile regexes.
  • Many code quality improvements.
• Mattia Rizzolo:
  • Deduplicate code for recognising file types based on RE_FILE_TYPE and RE_FILE_EXTENSION.
  • Improve code quality in tests.

Further development continued in Git, and will be released as version 70 next week:

• Chris Lamb:
  • Add tests for --html-dir output and improve code quality elsewhere in tests.
  • Add markdown and reStructuredText output, as well as tests for these.
  • Improve software architecture of presenters.
  • Fix error-checking in the Haskell comparator.
• James Clarke:
  • Haskell comparator: properly extract version from interface files.
• Mattia Rizzolo:
  • Improve some documentation.
• Brett Smith:
  • Improve documentation including --help output.

reproducible-builds.org website development

• Brett Smith:
  • berlin2016: List Conservancy consistently as a participant.
• Chris Lamb:
  • Add Valerie's talk to resources page.
• Daniel Shahaf:
  • Improved the "How to chair a meeting" section.

tests.reproducible-builds.org

• Holger added arm64 to https://tests.reproducible-builds.org/debian/index_variations.html

• Mattia improved our process for building the performance page so that stats for new architectures are computed correctly without manual intervention.

• Holger enhanced the build node maintenance scripts to correctly detect if /dev/shm is mounted incorrectly (due to #851427) and deployed an /etc/rc.local startup script to all systems which works around it. As a result, jenkins_semaphore_setup_issue should be obsolete.

• Mattia improved the diskspace monitoring visible at our munin page for the 44 nodes we're currently running.

• Holger added 6GB more RAM to jenkins.debian.net, for a total of 64GB RAM, to better cope with the new jobs due to `arm64. As usual, thanks to profitbricks.com for the hardware resources enabling this work.

Misc.

This week's edition was written by Ximin Luo, Vagrant Cascadian, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday January 8 and Saturday January 14 2017:

Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd

• Dennis Gilmore and Holger Levsen will present about "Reproducible Builds and Fedora" at Devconf.cz on February, 27th.

• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th

Reproducible work in other projects

Reproducible Builds have been mentioned in the FSF high-priority project list.

The F-Droid Verification Server has been launched. It rebuilds apps from source that were built by f-droid.org and checks that the results match.

Bernhard M. Wiedemann did some more work on reproducibility for openSUSE.

Bootstrappable.org (unfortunately no HTTPS yet) was launched after the initial work was started at our recent summit in Berlin. This is another topic related to reproducible builds and both will be needed in order to perform "Diverse Double Compilation" in practice in the future.

Toolchain development and fixes

Ximin Luo researched data formats for SOURCE_PREFIX_MAP and explored different options for encoding a map data structure in a single environment variable. He also continued to talk with the rustc team on the topic.

Daniel Shahaf filed #851225 ('udd: patches: index by DEP-3 "Forwarded" status') to make it easier to track our patches.

Chris Lamb forwarded #849972 upstream to yard, a Ruby documentation generator. Upstream has fixed the issue as of release 0.9.6.

Alexander Couzens (lynxis) has made mksquashfs reproducible and is looking for testers. It compiles on BSD systems such as FreeBSD, OpenBSD and NetBSD.

Bugs filed

Chris Lamb:

• #850683 filed against gerbv.

Lucas Nussbaum:

• #850973 filed against shogun.

Nicola Corna:

• #851190 filed against gerbv.

Reviews of unreproducible packages

13 package reviews have been added and 13 have been removed in this week, adding to our knowledge about identified issues.

1 issue type has been added:

• help2man_puts_traceback_in_generated_man_page

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (3)
• Lucas Nussbaum (11)
• Nicola Corna (1)

diffoscope development

Many bugs were opened in diffoscope during the past few weeks, which probably is a good sign as it shows that diffoscope is much more widely used than a year ago. We have been working hard to squash many of them in time for Debian stable, though we will see how that goes in the end…

• Mattia Rizzolo:
  • Code quality and style improvements.
• Maria Glukhova:
  • Add image metadata comparison.
  • Zipinfo included in APK files comparison. (Closes: #850502)
  • Remove archive name from apktool.yml and rename it. (Closes: #850501)
• Chris Lamb:
  • Correctly escape value of href="" elements (re. #849411)
  • Support comparing .ico files using img2txt (Closes: #850730) and fixes and extra tests in subsequent commits.
  • comparators.utils.file: Include magic file type when we know the file format but can't find file-specific details. (Closes: #850850)
  • And other code quality and style improvements.

reproducible-website development

• Daniel Shahaf:
  • Add how-to-chair-a-meeting.md.
• Holger Levsen:
  • Add links to reports by several attendees of berlin2016

tests.reproducible-builds.org

• Ximin Luo and Holger Levsen worked on stricter tests to check that /dev/shm and /run/shm are both mounted with the correct permissions. Some of our build machines currently still fail this test, and the problem is probably the root cause of the FTBFS of some packages (which fails with issues regarding sem_open). The proper fix is still being discussed in #851427.

• Valerie Young worked on creating and linking autogenerated schema documentation for our database used to store the results.

• Holger Levsen added a graph with diffoscope crashes and timeouts.

• Holger also further improved the daily mail notifications about problems.

Misc.

This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday January 1 and Saturday January 7 2017:

GSoC and Outreachy updates

• Maria Glukhova blogged about Getting to know diffoscope better.

Toolchain development

• #849999 was filed: "dpkg-dev should not set SOURCE_DATE_EPOCH to the empty string"

Packages reviewed and fixed, and bugs filed

Chris Lamb:

• #849968 filed against pciutils.
• #849972 filed against yard.
• #850556 filed against sed.

Dhole:

• #849926 filed against nxt-firmware.

Reviews of unreproducible packages

13 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been added/updated:

• Add new randomness_in_documentation_generated_by_yardoc toolchain issue.
• Add fix for randomness_in_documentation_generated_by_yardoc toolchain issue.

Upstreaming of reproducibility fixes

Merged:

• calibre
• jna
• rocksndiamonds
• avfs
• jackaudio1
• jackaudio2
• gri

Opened:

• rockdodger

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (4)

diffoscope development

diffoscope 67 was uploaded to unstable by Chris Lamb. It included contributions from :

[ Chris Lamb ]

* Optimisations:
  - Avoid multiple iterations over archive by unpacking once for an ~8X
    runtime optimisation.
  - Avoid unnecessary splitting and interpolating for a ~20X optimisation
    when writing --text output.
  - Avoid expensive diff regex parsing until we need it, speeding up diff
    parsing by 2X.
  - Alias expensive Config() in diff parsing lookup for a 10% optimisation.

* Progress bar:
  - Show filenames, ELF sections, etc. in progress bar.
  - Emit JSON on the the status file descriptor output instead of a custom
    format.

* Logging:
  - Use more-Pythonic logging functions and output based on __name__, etc.
  - Use Debian-style "I:", "D:" log level format modifier.
  - Only print milliseconds in output, not microseconds.
  - Print version in debug output so that saved debug outputs can standalone
    as bug reports.

* Profiling:
  - Also report the total number of method calls, not just the total time.
  - Report on the total wall clock taken to execute diffoscope, including
    cleanup.

* Tidying:
  - Rename "NonExisting" -> "Missing".
  - Entirely rework diffoscope.comparators module, splitting as many separate
    concerns into a different utility package, tidying imports, etc.
  - Split diffoscope.difference into diffoscope.diff, etc.
  - Update file references in debian/copyright post module reorganisation.
  - Many other cleanups, etc.

* Misc:
  - Clarify comment regarding why we call python3(1) directly. Thanks to Jérémy
    Bobbio <[email protected]>.
  - Raise a clearer error if trying to use --html-dir on a file.
  - Fix --output-empty when files are identical and no outputs specified.

[ Reiner Herrmann ]
* Extend .apk recognition regex to also match zip archives (Closes: #849638)

[ Mattia Rizzolo ]
* Follow the rename of the Debian package "python-jsbeautifier" to
  "jsbeautifier".

[ siamezzze ]
* Fixed no newline being classified as order-like difference.

reprotest development

reprotest 0.5 was uploaded to unstable by Chris Lamb. It included contributions from:

[ Ximin Luo ]

* Stop advertising variations that we're not actually varying.
  That is: domain_host, shell, user_group.
* Fix auto-presets in the case of a file in the current directory.
* Allow disabling build-path variations. (Closes: #833284)
* Add a faketime variation, with NO_FAKE_STAT=1 to avoid messing with
  various buildsystems. This is on by default; if it causes your builds
  to mess up please do file a bug report.
* Add a --store-dir option to save artifacts.

Other contributions (not yet uploaded):

• Chris Lamb:
  • setup.py: Specify a Python3 env shebang
  • Drop reprotest/{runner,tools} from DEP-5 copyright wildcard matching.

reproducible-builds.org website development

• `website.git notifications now go to#reproducible-builds. Thanks todanielsh` and the KGB bot maintainers.

• Daniel Shahaf:

  • events: add IRC meeting
  • Move mailing list from 'who' to 'resources'.

• Holger Levsen:

  • link to meetbot.debian.net IRC meetings archives

• Chris Lamb:

  • tools: Underscore that disorderfs is deliberately adding such randomness.

tests.reproducible-builds.org

• Debian arm64 architecture was fully tested in all three suites in just 15 days. Thanks again to Codethink.co.uk for their support!
• Log diffoscope profiling info. (lamby)
• Run pg_dump with -O --column-inserts to make easier to import our main database dump into a non-PostgreSQL database. (mapreri)
• Debian armhf network: CPU frequency scaling was enabled for three Firefly boards, enabling the CPUs to run at full speed. (vagrant)
• Arch Linux and Fedora tests have been disabled (h01ger)
• Improve mail notifications about daily problems. (h01ger)

Misc.

This week's edition was written by Chris Lamb, Holger Levsen and Vagrant Cascadian, reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday December 25 and Saturday December 31 2016:

Media coverage

• Brett Smith wrote a report on our recent summit meeting on behalf of the Software Freedom Conservancy.

Reproducible bugs filed

Chris West:

• #849667 filed against htslib.

Chris Lamb:

• #849314 filed against node-gulp.
• #849333 filed against faker.

Rob Browning:

• #849692 filed against oneliner-el.

Reviews of unreproducible packages

7 package reviews have been added, 12 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been updated:

• build_id_differences_only
• cython_captures_build_path

Weekly QA work

During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris West (19)
• Chris Lamb (7)
• Rob Browning (1)

diffoscope development

• Chris Lamb:
  • Optimisations:
    • Avoid unnecessary string manipulation writing --text output (~20x speedup).
    • Avoid n iterations over archive files (~8x speedup).
    • Memoize calls to distutils.spawn.find_executable to avoid excessive stat(1) syscalls.
  • Progress bar:
    • Show current file / ELF section under analysis etc. in progress bar.
    • Move the --status-fd output to use JSON and to include the current filename.
  • Code tidying:
    • Split out the try.diffoscope.org client so that it can be released separately on PyPI.
    • Completely rework the diffoscope and diffoscope.comparators modules, grouping similar utilities into their own modules, etc.
  • Miscellaneous:
    • Ensure that running from Git will always use that checkout's Python modules.
• Mattia Rizzolo:
  • Follow the rename of the debian package 'python-jsbeautifier' to 'jsbeautifier'
• Reiner Herrmann:
  • apk: Extend recognition regex to also match zip archives
• siamezzze:
  • Fixed no newline being classified as order-like difference.

strip-nondeterminism development

• strip-nondeterminism 0.029-2 was uploaded to unstable by Mattia Rizzolo to include a fix by Chris Lamb to fix the autopkgtest tests.

try.diffoscope.org development

• Chris Lamb:
  • Show progress bar and position in queue, etc.
  • Promote command-line client with PyPI instructions.
  • Increase comparison time limit to 90 seconds.

tests.reproducible-builds.org

• Run half of the arm64 build nodes in the future. (h01ger)
• Resume testing scheduling (on i386 and armhf) now #846564 and #844701 bugs in dpkg are fixed in that suite. (h01ger)

Misc.

This week's edition was written by Chris Lamb, Holger Levsen and was reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday December 18 and Saturday December 24 2016:

Media coverage

100% Of The 289 Coreboot Images Are Now Built Reproducibly by Phoronix, with more details in German by Pro-Linux.de.

We have further reports on our Reproducible Builds World summit #2 in Berlin from Rok Garbas of NixOS as well as Clemens Lang of MacPorts

Debian infrastructure work

Dak now archives buildinfo files thanks to a patch from Chris Lamb. We also have mostly finalised a design of how they will be distributed by the Debian FTP mirror network which we will start implementing soon. This is great for the future of Debianb but unfortunately this also means that we won't have .buildinfo files for Stretch as Debian will not rebuild its source packages and because these binary packages currently in the archive were mostly built with dpkg > 1.18.11.

reprepro/5.0.0-1 has added support for dealing with .buildinfo files that are included in .changes files. (Closes: #843402)

Reproducible work in other projects

The Chromium project is now working on making their build process (mostly) deterministic.

Their motivation is to save both "[money] (less hardware is required) and developer time (reduced latency by having less work to do on the TS and CI)".

Unreproducible bugs filed

• Chris Lamb:
  • #848721 filed against apt.
  • #848866 filed against libcorelinux.
  • #849314 filed against node-gulp.
  • #849333 filed against faker.
• Dhole:
  • #848633 filed against sugar-toolkit-gtk3.

Reviews of unreproducible packages

39 package reviews have been added, 75 have been updated and 44 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been updated:

• Updated captures_build_path
• Added nondeterministic_ordering_in_desktop_files_by_python_sugar3

Weekly QA work

During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Adrian Bunk (1)
• Chris Lamb (7)
• Lucas Nussbaum (4)

diffoscope development

diffoscope 66 was uploaded to unstable by Chris Lamb. It included contributions from:

• Emanuel Bronshtein:
  • Use ssh-keygen for comparing OpenSSH public keys
  • Use js-beautify as JavaScript code beautifier for .js files (with tests).
  • Many CSS & HTML improvements.
  • Change all HTTP URLs to HTTPS where applicable.
• anthraxx:
  • Enable the use of ssh-keygen on Arch Linux.
• Maria Glukhova:
  • Add detection of order-only difference in plain text format. (Closes: #848049)
  • Change icc-recognizing regexp to reflect changes in file type description. (Closes: #848814)
• Chris Lamb:
  • Update tests for compatibility with enjarify >= 1.0.3. (Closes: #849142)
  • When skipping tests because the version of an external is too low, print the detected version.
  • Avoid unpacking packages twice when comparing .changes. (Closes: #843531)
  • Add a simple profiling framework (enabled via --profile).
  • Various code quality and reliability improvements.
  • Document how to sign PyPI uploads.

strip-nondeterminism development

strip-nondeterminism 0.029-1 was uploaded to unstable by Chris Lamb. It included no new content from this week, but rather included contributions from previous weeks.

reproducible-website development

The website is now also accessible via the https://www.reproducible-builds.org URL.

• Clemens Lang:
  • Add the definition of "reproducible", as drafted at the reproducible builds world summit in Berlin. Thanks to all participants in the sessions that worked these out!
• Valerie R Young:
  • Force ordering of titles.
  • Various formatting improvements.
• Holger Levsen:
  • Various usability and formatting improvements.
  • https://www.reproducible-builds.org
• Chris Lamb:
  • Various usability, style and wording improvements.
  • Add Debconf15, Skroutz.gz and MiniDebconfCambridge15 talks to resouces page.

tests.reproducible-builds.org

• We changed the data storage backend from a single sqlite3 database file (651 MB) to a PostgreSQL database. With this change we'll be able to scale a lot more and add testing of the arm64 architecture.
  • Valerie Young wrote most of the code, Mattia Rizzolo reviewed and helped improve the code and Holger deployed it and found some minor bugs which have been fixed.
• We are now testing the arm64 architecture for all packages on all suites, arranged by Holger. Many thanks to codethink for providing us with access to eight 8-core arm64 machines with 64GB memory, which allows us to rebuild Debian very fast!

Misc.

This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC and the mailing lists.