pkgsrc Change Log

What We Did Last Summer

Jörg Sonnenberger

joerg@NetBSD.org

libnbcompat magic (I)

• Added SHA256 / SHA512
• Fixed SHA256 / SHA512 for strict alignment platforms
• Check that vsnprintf returns expected output bytes
• New (v)snprintf with vfprintf to /dev/null and vsprintf

libnbcompat magic (I)

• Dropped buildlink3.mk, optional PIC version
• A BSDish getopt can be required (optreset)
• USE_FEATURES= asprintf err glob

Infrastructure cleanup

• Remove USE_MTREE, keep single mtree file for print-PLIST
• Drop PKG_TOOLS_PREFIX handling — never remove system programs!
• Don't safe guard against Zoularis use
• 3 years of bmake development merged

pkg_install cleanup (I)

• Remove mtree support:
  • pkg_create refuses to create such packages
  • pkg_add ignores mtree files
• Remove support for required scripts in pkg_add
• Don't leak memory in Dewey — after a few thousand times it hurts
• Made pkg_add look for best match for missing dependency, not the version built against

pkg_install cleanup (II)

• Remove master/slave mode in pkg_add
• Remove extract-in-place option in pkg_add
• Remove preserve option in pkg_add
• pkg_create uses libarchive, proper owner/group information for DESTDIR
• pkg_info uses libarchive and libfetch: much faster for archive operations

digest

• Use AC_C_BIGENDIAN to unbreak HP-UX/Itanium and simplify the code
• Still not sure how to deal with autoconf 2.62 and Universal Builds
• Tiger has the largest hash — allocate enough space to not crash
• Tiger output for the empty file was wrong
• SHA512 had a subtile bug when size of last block = size of short fragment

Spring cleaning (I)

• GNOME 1.x
• BIND 4.x
• MySQL 4.0 — 4.1 got ressurrected
• Blackdown and Sun JDK/JRE 1.3

Spring cleaning (II)

• SuSE 9.1
• Firefox / Thunderbird 1.5
• Python 2.0 and 2.2
• Xorg 6.9 (good bye, releng: 2007/09/24 - 2007/09/28 - 2008/01/17)

Bootstrapping (I)

• Support to create a shiny, clickable MacOS X bootstrap package
• Universal build exposed link order issues in tnftp with libedit
• xargs(1) replacement for Interix: direct exit on empty input
• Split bootstrap:
  • Bootstrap bmake, pkg_admin, pkg_create (register-only) and pkg_info first
  • Rebuild using normal pkgsrc magic with WRKOBJDIR set

Bootstrapping (II)

• Conditionally build and install pdksh (OSF/1, Solaris)
• Merged mkbinary into bootstrap itself
• Saner default values: varbase and pkgdb based on prefix
• When looking for programs, don't match directories in PATH
• Dropped mtree

Bootstrapping (III)

• Preperations for further cleanup:
  • ftp is real tool now
  • pax no longer part of USE_TOOLS by default
  • fetch and extract depend on right tools
• pkgsrc itself supports case-insensitivity after repo moves

Fetching and checksumming

• Share code between checksum and patch module
• checksum target can always be called
• Verify all provided checksums, not just default list
• Don't bail out if DISTDIR is invalid and neither PATCHFILES nor DISTFILES are needed
• Separate NO_CHECKSUM from FAILOVER_FETCH
• Replace DIST_SUBDIR=. with DIST_SUBDIR=, it breaks PKG_RESUME_TRANSFER

Performance updates

• mk: Don't expand LOCALPATCHES if it isn't said — bad if /net has autofs...
• pkg_install: Bail out fast if literal part of a pattern can't match -> "pkg_admin audit" 92% faster
• wrapper: Load cache only once and use a shell function later -> 75% for long command lines
• mk: Variable substition to compute LOWER_OPSYS_VERSUFFIX instead of forks

CHECK framework (I)

• New shlib backend for ELF platforms:
  • Works for DESTDIR and cross-compiling
  • Can detect linkage against build dependency
  • WRKDIR leaks via RPATH
  • Default on NetBSD, DragonFly and FreeBSD
• CHECK_PERMS uses generated PLIST and supports DESTDIR
• No portability checks for bash scripts

CHECK framework (II)

• CHECK_INTERPRETER part of PKG_DEVELOPERS
• CHECK_INTERPRETER needs to look inside ${DESTDIR} as well
• Optional check that binaries are not stripped
• Fake ${HOME} and extend WRKDIR reference check for it

Dependency handling

• Check pkg_install first, try install/update it if out-dated
• For bulk builds record pkg_install dependency if out-dated
• Remember all dependencies and exact versions
• Prepend build and runtime dependencies directly to PLIST, not via pkg_create -P/-T
• Shorter output, but includes type of dependency

make replace and pkg_rolling_replace (I)

• unsafe_depends == a dependency might have had an ABI change
• "make replace" always sets unsafe_depends_strict on dependencies
• "make replace" has to store +REQUIRED_BY and +INSTALLED_INFO outside ${WRKDIR}/.pkgdb

make replace and pkg_rolling_replace (II)

• "pkg_admin rebuild-tree" helps to recompute +REQUIRED_BY
• Faster dependency graph logic using awk, up to 20 times faster
• "pkg_add -u" from pkg_install-renovation keeps +PRESERVE and +INSTALLED_INFO

Infrastructure additions (I)

• "make clean" with WRKOBJDIR also tries to remove the category directory
• PLIST_VARS for common task of conditional PLIST substitution
• Replace text check using file(1) with tr/wc looking for NUL bytes
• When searching for shared linkage data, skip keyword entries in PLIST: @unexec /bin/rmdir
• Fix install-sh issues with GNU configure by replacing the script with our version

Infrastructure additions (II)

• +INSTALL and +DEINSTALL are much less verbose by default
• PRIVILEGED_STAGES
• Wrapper don't print transformed command after failure
• Wrapper logging is copy&paste safe
• Fix long-standing quoting issues in the wrapper, unnoticied as -I and -L normally don't need it

Infrastructure additions (III)

• Print all files when using DESTDIR, not just those newer than the start of the build
• Call GCC with ALL_ENV when extracting version number — pkgsrc doesn't understand Brazilian Portogese
• mk/resolv.buildlink3.mk to get a thread-safe resolver
• Rebuild ldconfig-like caches where needed
• Unified emulation framework to cover NetBSD 32bit compatibility, Linux emulation etc.

Infrastructure additions (IV)

• "make package" uses temporary package + move
• Use chrpath(1) to fix up binary package instead of wrapper scripts
• GENERATE_PLIST for dynamic PLISTs, no intentionally empty PLISTs any more.
• Better group handling for unprivileged builds, can use all groups the user is in
• Ruby Gem logic
• Python Egg logic

Infrastructure additions (V)

• "make README.html" prints past and open issues using "pkg_admin audit-history"
• DESTDIR support for all Perl modules using Module::Build.
• cmake integration as tool and configure stage (USE_CMAKE=yes)
• AUTO_MKDIRS to complement INSTALLATION_DIRS
• Better diagnostic for missing CONF_FILES etc. arguments ("can't shift that many")

License work

• Started to tag packages with commented out GPL3 entries
• FOSS licenses require no -license suffix
• Make it easier to selectively remove default licenses by using a separate variable
• Improve diagnostic where the user can find mk.conf
• Al is picky about the spelling and likes the redundancy
• Ion3(tm) removal after upstream complains, questionable legal status

Cross compiling support (I)

• New variables: NATIVE_MACHINE_GNU_ARCH, NATIVE_LOWER_ARCH, NATIVE_MACHINE_ARCH, NATIVE_MACHINE_PLATFORM and NATIVE_MACHINE_GNU_PLATFORM
• mk.conf can override MACHINE_ARCH and set USE_CROSS_COMPILE
• Overrides --host and --build argument for GNU configure
• buildlink3 uses CROSS_DESTDIR hierachy as popolated e.g. by "build.sh distribution"

Cross compiling support (II)

• package-install and bin-install are unprivileged
• Mix of native tools (gmake, pkg-config) and build.sh toolchain (as, ld, strip, ...)
• Special compiler wrapper:
  • Add ${CROSS_DESTDIR}/usr/include to the include search path
  • Add ${CROSS_DESTDIR}/usr/lib to the library search path
  • Transform rpath options to rpath + rpath-link into ${CROSS_DESTDIR}
• pkg_add -m overrides uname based MACHINE_ARCH test

Q&A

Questions?