News from the VCS front

Jörg Sonnenberger

News from the VCS front

NetBSD Developer Summit, EuroBSDCon 2017

Agenda

lochley.NetBSD.org
Mercurial performance
Check list
Pending work

lochley.NetBSD.org

Former home of cvs.NetBSD.org
Lots of RAM, lots of disk space
New home of the automated CVS -> Fossil -> Git conversion
...and now home of the CVS -> Fossil -> Mercurial conversion, too

lochley and Mercurial: challenges

fastimport used one directory with one file per blob
...with uncompressed content
Duplicate commit check happens after transaction locks
fastimport assumed markers are persistent
...tagging commits with markers, breaking duplicate detection
fastimport missed support for its own mark file
...no incremental conversion without fixing it

lochley and Mercurial: status quo

Conversion fully integrated now
Similar time to the git cycle
Results pushed to Bitbucket
...beware that src is pushing some scalability limits for BB

Mercurial performance relative to Git (I)

Basic testing on E5 E3-1245v5 under Illumos with 100Mbit/s+ connectivity
hg init + unbundle from cdn.NetBSD.org: ~8min, CPU bound (zstd)
...with zstd as revlog compression engine: 6min
hg clone from https://bitbucket.org/netbsd/src: 10min (bzip2)
...with zstd as revlog compression engine: 7min
git clone: ~5min with 3min local CPU time
git clone needs 50% more bandwidth than the bundle

Mercurial performance relative to Git (II)

hg co -r trunk after initial clone: 40s (2min CPU time)
hg co -r netbsd-8 and back to trunk: 6s / 8s (16s / 22s CPU time)
git checkout netbsd-8 and back to trunk: 5s / 8s (6s / 8s CPU time)
hg commit vs hg commit BUILDING: 2s / 0.8s
git commit -a vs git commit BUILDING: 0.7s / 2s
hg blame UPDATING: 0.7s
git blame UPDATING: 2s

Check list: compared to FreeBSD's VCS wiki

CVS migration: solved
History removal: possible, but tricky with convert
Base system integration: not planned
Database recovery: double check transaction assumptions
Partial clones: needs investigation, shallow and remotefilelog extensions

Check list: core@ requirements (I)

Performance: testing on low-end hardware in progress
...but comfortable on the 90$ Pinebook under Linux
...peak RSS on Linux ~800MB
User interface and terminology mostly compatible with CVS
Concurrent use of CVS: not planned
...but preparation and testing of infrastructure against Mercurial
...expected down time for final migration: around 2 days

Check list: core@ requirements (II)

hg ↔ git bridges exists, performance and limitations need to be checked
Automatic clean-up of CVS revision references: not feasible
Repository conversion: character set conversion in CVS before final migration
...no detection of moves/copies planned
...no retro-patching of SCCS history planned

Pending work (I)

Provide documented workflows for common operations, using modern best practises
...include merge vs rebase discussion for local changes
...staging repository vs "core" repository
Work out the build and testing infrastructure
...move towards more off-the-shelf components
...somewhat easier with changesets
...currently under active investigation

Pending work (II)

Figure out new release engineering workflow
...easier with changesets
Current approach: push changes to staging repository
...CI verifies building (and maybe passing tests)

Pending work (III)

Test repository server
...automatic pushing of clone bundles to ftp.NetBSD.org
...test build host attached
...allow pushing commits for authentication testing
...provide source-changes alternative based on hg notify
Investigate Security Team ↔ Releng interaction

Q&A