Editing Cvs2svn

At the end of 2007, preparations to convert BRL-CAD's CVS repository to Subversion began.  The cvs2svn tool was used to perform the conversion, initially with version 1.5.1 but later using the latest available 2.0.1 version.  The page is provided for historic reference on how the repository was converted including the steps that were taken.

Using version 1.5.1, cvs2svn using all default options took over 7 hours of processing before it filled up available hard disk space (only 3GB was available at the time).  More space was made available and cvs2svn version 2.0.1 was installed, which provided a new more direct and faster method of extracting data from the CVS repository.  Using the same default options, version 2.0.1 took just under 3 hours.  The details of this conversion process are documented in following.

= CVS repository preparations =
As cvs2svn reported issues in the repository processing, some repairs to BRL-CAD's CVS root was required.  Working on a copy of the CVS root files, this included fixing five directory subtrees that were in the Attic:

* html/manuals/Attic/mged4.0
* html/Attic/release-notes
* Attic/libitcl
* Attic/libtcl
* Attic/libtk

These directories seemed to expose a bug in cvs2svn's Attic processing.  Initial thoughts were that someone had manually moved directories into the Attic, but upon further investigation it seemed more likely that this was perhaps actually proper behavior for an old version of CVS.  All of the files in the Attic subdirs were properly marked as dead, and they were a '''hierarchy''' of ,v files that were properly marked as Dead revisions in that Attic subdir.  Also, the directory "deletions" into the Attic were performed by multiple developers (namely butler, parker, and pjt iirc) during the late 90's, when the repository was located on an old IRIX server with an old version of cvs (pre 1.10 but unknown). 

The workaround solution used, without needing to modify cvs2svn, was to create Attic directories throughout the hierarchy subfolders, move the ,v files into their respective Attic directory for each subfolder, and then move the directory up one level out of the Attic.  The Attic/libitcl directory conflicted with an existing libitcl/ directory, so it was simply purged given it would not have been beneficial to preserve that external dependency history.

Another message reported by cvs2svn was that there was (exactly) one commit that had non-ascii characters.  Upon investigation, it was a commit that contained an ñ in utf-8 so the --encoding option   was added to use utf-8.  At that point, the options being used were:

 cvs2svn --encoding=utf_8 --cvs-revnums --auto-props=auto-props --eol-from-mime-type --mime-types=mime.types --dumpfile=/path/to/brlcad.dumpfile /path/to/cvs/repository

Upon performing a full checkout of the entire newly imported Subversion repository, the checkout failed in a branch (rel-5-1-branch) on checkout of cup.g from html/manuals/mged/. Upon review of the CVS root files, it was not apparent what was unique about cup.g,v other than the fact that there also existed a Cup.g,v (both of which existed in the Attic).  As that file was entirely a triviality as a v4 binary BRL-CAD database file used in a documentation example, both the uppercase and lowercase files were removed.

= Single project vs multiproject =

The default behavior of cvs2svn is to import all of the CVS modules (or projects) as a single project, putting module directories into each of the /trunk, /branches, and /tags directory.  For BRL-CAD, this results in a hierarchy that looks like the following:

 /trunk/CVSROOT/...srcs...
 /trunk/brlcad/...srcs...
 /trunk/...other_cvs_modules.../...srcs...
 
 /branches/ansi-branch/brlcad/...srcs...
 /branches/windows-branch/brlcad/...srcs...
 /branches/...other_brlcad_branches.../brlcad/...srcs...
 /branches/...other_cvs_module_branches.../module/...srcs...
 
 /tags/rel-5-2/brlcad/...srcs...
 /tags/rel-7-0-2/brlcad/...srcs...
 /tags/...other_brlcad_tags.../brlcad/...srcs...
 /tags/...other_module_tags.../module/...srcs...

Since BRL-CAD's modules really have been historically treated as entirely separate projects, it became apparent that a different hierarchy would be more desirable.  It was not desirable to checkout the trunk and receive all of the former CVS modules.  Similarly it was not desirable to have the extra module/project subdirectory in each of the tag and branch directories.

Instead of restructuring the layout after the import to Subversion, the BRL-CAD CVS repository conversion would be run through cvs2svn again as a "multiproject" import.  It became necessary to create a cvs2svn options file where modules are individually listed.  As a multiproject import, the following hierarchy results:

 /CVSROOT/trunk/...srcs...
 /brlcad/trunk/...srcs...
 /...other_modules.../trunk/...srcs...
 
 /brlcad/branches/ansi-branch/...srcs...
 /brlcad/branches/windows-branch/...srcs...
 /brlcad/branches/...other_brlcad_branches.../...srcs...
 /...other_modules.../branches/...other_module_branches.../...srcs...
 
 /brlcad/tags/rel-5-2/...srcs...
 /brlcad/tags/rel-7-0-2/...srcs...
 /brlcad/...other_brlcad_tags.../...srcs...
 /...other_modules.../...other_module_tags.../...srcs...

This parallels the CVS root files more directly and their nature as independent projects.  Upon conversion as a multiproject import, the repository was once again validated both manually and quantitatively.  Creation of a Subversion repository from the dumpfile was verified.  It was verified that a complete log could be extracted via "svn log" without error and that no warnings or errors existed in the cvs2svn verbose output.  Also, a complete checkout of the SVN root was performed after upload to Sourceforge to verify no errors were evident.

It was later detected that permissions were incorrect on a limited set of the CVS root files (a couple dozen), but the svn:executable property was fixed on those files after import into SVN.

= cvs2svn Options =

To perform the conversion, the following options file was used:

<pre>
import re

from cvs2svn_lib.boolean import *
from cvs2svn_lib import config
from cvs2svn_lib.common import UTF8Encoder
from cvs2svn_lib.log import Log
from cvs2svn_lib.project import Project
from cvs2svn_lib.output_option import DumpfileOutputOption
from cvs2svn_lib.output_option import ExistingRepositoryOutputOption
from cvs2svn_lib.output_option import NewRepositoryOutputOption
from cvs2svn_lib.revision_reader import RCSRevisionReader
from cvs2svn_lib.revision_reader import CVSRevisionReader
from cvs2svn_lib.checkout_internal import InternalRevisionReader
from cvs2svn_lib.symbol_strategy import AllBranchRule
from cvs2svn_lib.symbol_strategy import AllTagRule
from cvs2svn_lib.symbol_strategy import BranchIfCommitsRule
from cvs2svn_lib.symbol_strategy import ExcludeRegexpStrategyRule
from cvs2svn_lib.symbol_strategy import ForceBranchRegexpStrategyRule
from cvs2svn_lib.symbol_strategy import ForceTagRegexpStrategyRule
from cvs2svn_lib.symbol_strategy import HeuristicStrategyRule
from cvs2svn_lib.symbol_strategy import RuleBasedSymbolStrategy
from cvs2svn_lib.symbol_strategy import UnambiguousUsageRule
from cvs2svn_lib.symbol_transform import RegexpSymbolTransform
from cvs2svn_lib.property_setters import AutoPropsPropertySetter
from cvs2svn_lib.property_setters import CVSBinaryFileDefaultMimeTypeSetter
from cvs2svn_lib.property_setters import CVSBinaryFileEOLStyleSetter
from cvs2svn_lib.property_setters import CVSRevisionNumberSetter
from cvs2svn_lib.property_setters import DefaultEOLStyleSetter
from cvs2svn_lib.property_setters import EOLStyleFromMimeTypeSetter
from cvs2svn_lib.property_setters import ExecutablePropertySetter
from cvs2svn_lib.property_setters import KeywordsPropertySetter
from cvs2svn_lib.property_setters import MimeMapper
from cvs2svn_lib.property_setters import SVNBinaryFileKeywordsPropertySetter

Log().log_level = Log.VERBOSE

ctx.output_option = DumpfileOutputOption(
    dumpfile_path=r'/path/to/brlcad.dumpfile',
    )

ctx.dry_run = False
ctx.revision_reader = InternalRevisionReader(compress=True)
ctx.svnadmin_executable = r'svnadmin'
ctx.sort_executable = r'sort'
ctx.trunk_only = False

ctx.prune = True

ctx.utf8_encoder = UTF8Encoder(
    [
        'ascii',
        'utf8',
        ],
    )

ctx.filename_utf8_encoder = UTF8Encoder(
    [
        'ascii',
        ],
    )

ctx.symbol_strategy = RuleBasedSymbolStrategy()
ctx.symbol_strategy.add_rule(UnambiguousUsageRule())

ctx.username = None

ctx.svn_property_setters.extend([
    AutoPropsPropertySetter(
        r'/path/to/auto-props',
        ignore_case=True,
        ),

    MimeMapper(r'/path/to/mime.types'),

    CVSBinaryFileEOLStyleSetter(),
    CVSBinaryFileDefaultMimeTypeSetter(),

    EOLStyleFromMimeTypeSetter(),

    DefaultEOLStyleSetter(None),

    SVNBinaryFileKeywordsPropertySetter(),
    KeywordsPropertySetter(config.SVN_KEYWORDS_VALUE),
    ExecutablePropertySetter(),

    CVSRevisionNumberSetter(),

    ])

ctx.tmpdir = r'cvs2svn-tmp'
ctx.cross_project_commits = True
ctx.cross_branch_commits = True
ctx.retain_conflicting_attic_files = False

run_options.profiling = False

ctx.add_project(
    Project(
        r'/path/to/brlcad/CVSROOT',
        'CVSROOT/trunk',
        'CVSROOT/branches',
        'CVSROOT/tags',
        )
    )
ctx.add_project(
    Project(
        r'/path/to/brlcad/brlcad',
        'brlcad/trunk',
        'brlcad/branches',
        'brlcad/tags'
        )
    )
ctx.add_project(
    Project(
        r'/path/to/brlcad/jbrlcad',
        'jbrlcad/trunk',
        'jbrlcad/branches',
        'jbrlcad/tags'
        )
    )
ctx.add_project(
    Project(
        r'/path/to/brlcad/rt^3',
        'rt^3/trunk',
        'rt^3/branches',
        'rt^3/tags'
        )
    )
ctx.add_project(
    Project(
        r'/path/to/brlcad/rtcmp',
        'rtcmp/trunk',
        'rtcmp/branches',
        'rtcmp/tags'
        )
    )
ctx.add_project(
    Project(
        r'/path/to/brlcad/web',
        'web/trunk',
        'web/branches',
        'web/tags'
        )
    )
</pre>

= Automatic properties =

The following auto-props file was used:

<pre>
[auto-props]
*.[0-9] = svn:mime-type=text/plain;svn:eol-style=native
*.ac = svn:eol-style=native;svn:mime-type=text/plain
*.ai = svn:mime-type=application/illustrator
*.am = svn:eol-style=native;svn:mime-type=text/plain
*.asc = svn:eol-style=native;svn:mime-type=text/plain
*.avi = svn:mime-type=video/x-msvideo
*.bmp = svn:mime-type=image/bmp
*.c = svn:eol-style=native
*.cpp = svn:eol-style=native
*.cxx = svn:eol-style=native
*.css = svn:mime-type=text/css;svn:eol-style=native
*.doc = svn:mime-type=application/msword
*.dsp = svn:eol-style=CRLF
*.dsw = svn:eol-style=CRLF
*.eps = svn:mime-type=application/postscript
*.g = svn:mime-type=application/octet-stream
*.gif = svn:mime-type=image/gif
*.gpgkey = svn:mime-type=application/gpg-keys
*.gtar = svn:mime-type=application/x-gtar
*.gz = svn:mime-type=application/x-gtar
*.h = svn:eol-style=native
*.hpp = svn:eol-style=native
*.hxx = svn:eol-style=native
*.htm = svn:mime-type=text/html;svn:eol-style=native
*.html = svn:mime-type=text/html;svn:eol-style=native
*.igs = svn:mime-type=model/iges
*.ico = svn:mime-type=image/x-icon
*.in = svn:eol-style=native;svn:mime-type=text/plain
*.itcl = svn:eol-style=native
*.itk = svn:eol-style=native
*.java = svn:eol-style=native
*.jpeg = svn:mime-type=image/jpeg
*.jpg = svn:mime-type=image/jpeg
*.m4 = svn:eol-style=native;svn:mime-type=text/plain
*.mov = svn:mime-type=video/quicktime
*.mp3 = svn:mime-type=audio/mpeg
*.mpg = svn:mime-type=video/mpeg
*.nsi = svn:mime-type=text/plain;svn:eol-style=native
*.pdf = svn:mime-type=application/pdf
*.php = svn:mime-type=text/plain;svn:eol-style=native
*.pix = svn:mime-type=image/x-rgb
*.pl = svn:eol-style=native;svn:executable
*.plist = svn:mime-type=text/plain
*.png = svn:mime-type=image/png
*.ppt = svn:mime-type=application/vnd.ms-powerpoint
*.ps = svn:mime-type=application/postscript
*.psd = svn:mime-type=application/photoshop
*.rt = svn:eol-style=native;svn:executable;svn:mime-type=text/x-sh
*.rtf = svn:mime-type=text/rtf
*.sh = svn:eol-style=native;svn:executable;svn:mime-type=text/x-sh
*.sln = svn:eol-style=CRLF
*.smil = svn:mime-type=application/smil
*.svg  = svn:mime-type=image/svg+xml
*.svgz = svn:mime-type=image/svg+xml
*.swf = svn:mime-type=application/x-shockwave-flash
*.tcl = svn:eol-style=native
*.tex = svn:mime-type=text/x-tex
*.tgz = svn:mime-type=application/x-gtar
*.tif = svn:mime-type=image/tiff
*.tiff = svn:mime-type=image/tiff
*.txt = svn:mime-type=text/plain;svn:eol-style=native
*.vbs = svn:eol-style=CRLF;svn:executable
*.vcf = svn:mime-type=text/x-vcard
*.vcproj = svn:eol-style=CRLF
*.xbm = svn:mime-type=image/x-xbitmap
*.xls = svn:mime-type=application/vnd.ms-excel
*.xml = svn:mime-type=text/xml;svn:eol-style=native
*.zip = svn:mime-type=application/zip
AUTHORS = svn:eol-style=native;svn:mime-type=text/plain
BUGS = svn:eol-style=native;svn:mime-type=text/plain
COPYING = svn:eol-style=native;svn:mime-type=text/plain
ChangeLog = svn:eol-style=native;svn:mime-type=text/plain
HACKING = svn:eol-style=native;svn:mime-type=text/plain
INSTALL = svn:eol-style=native;svn:mime-type=text/plain
Makefile = svn:eol-style=native
NEWS = svn:eol-style=native;svn:mime-type=text/plain
README = svn:eol-style=native;svn:mime-type=text/plain
README.* = svn:eol-style=native;svn:mime-type=text/plain
TODO = svn:eol-style=native;svn:mime-type=text/plain
### catch-all for files starting and ending in all caps
[A-Z]*[A-Z] = svn:eol-style=native;svn:mime-type=text/plain
</pre>

= Results and statistics =

The final cvs2svn processing resulted in a 2.8GB dumpfile and provided the following statistics:

 cvs2svn Statistics:
 ------------------
 Total CVS Files:             20127
 Total CVS Revisions:        147514
 Total CVS Branches:          73233
 Total CVS Tags:             281841
 Total Unique Tags:              80
 Total Unique Branches:          32
 CVS Repos Size in KB:       516588
 Total SVN Commits:           29886
 First Revision Date:    Thu Dec 15 19:06:47 1983
 Last Revision Date:     Mon Dec 31 15:25:14 2007
 ------------------
 Timings (seconds) :
 ------------------
 1731   pass1    CollectRevsPass
    0   pass2    CollateSymbolsPass
  567   pass3    FilterSymbolsPass
    2   pass4    SortRevisionSummaryPass
    4   pass5    SortSymbolSummaryPass
  702   pass6    InitializeChangesetsPass
  452   pass7    BreakRevisionChangesetCyclesPass
  785   pass8    RevisionTopologicalSortPass
  276   pass9    BreakSymbolChangesetCyclesPass
  513   pass10   BreakAllChangesetCyclesPass
  486   pass11   TopologicalSortPass
  342   pass12   CreateRevsPass
    9   pass13   SortSymbolsPass
    8   pass14   IndexSymbolsPass
 3696   pass15   OutputPass
 9573   total

The result is a contiguous history of BRL-CAD development that has gone from RCS to CVS and now to SVN preserving more than 25 years of revision history.  The conversion was completed on January 10th, 2008.