Subversion to Git migration

Everything starts out of a local copy of the Subversion repository that was hosting the WebPing project since its inception:

$ rm -rf ./svn-repository-copy
$ tar xvzf ./svn-repository-copy.tar.gz
$ kill `ps -ef | grep svnserve | awk '{print $2}'`
$ svnserve --daemon --listen-port 3690 --root ./svn-repository-copy

Let’s initialize a Git repository:

$ rm -rf ./webping-git
$ mkdir ./webping-git
$ cd ./webping-git
$ git init
$ git commit --allow-empty -m 'Initial commit'
$ git tag "init"

We now migrate the code from Subversion to Git:

$ git svn init --no-metadata --username deldycke svn://localhost:3690
$ git svn fetch
$ git rebase --onto git-svn master
$ rm -rf ./.git/svn/
$ rm -rf ./.git/refs/original/
$ git reflog expire --all
$ git gc --aggressive --prune

Removing unrelated files and folders

As WebPing was not alone in the original Subversion repository, we need to clean up the latter and only keep code of the former. Worse, WebPing didn’t started its life in a dedicated subfolder, but as a tool of another project, and jumped from folders to folders. After identifying in the history all places were WebPing lived once, I came up with this big, convoluted command line to do the cleaning:

$ git filter-branch --force --prune-empty --tree-filter 'find ./ -not -ipath "*webping*" -and -not -path "./other-project/trunk/tools/web-ping*" -and -not -path "./other-project/trunk/tools" -and -not -path "./other-project/trunk" -and -not -path "./other-project" -and -not -path "./.git*" -and -not -path "./" | xargs rm -rf' -- --all

Strangely enough, my init tag went of after the command above. So I had to rebased it to get it in line:

$ git rebase init master

We can now remove SVN tags and branches, get rid of the imported git-svn branch, and clean up our Git repository:

$ git filter-branch --force --prune-empty --tree-filter 'find -path "./WebPing/tags*" | xargs rm -rf' -- --all
$ git filter-branch --force --prune-empty --tree-filter 'find -path "./WebPing/branches*" | xargs rm -rf' -- --all
$ git branch -r -D git-svn
$ rm -rf ./.git/svn/
$ rm -rf ./.git/refs/original/
$ git reflog expire --all
$ git gc --aggressive --prune

If I now only have WebPing code in the repository, it still jumps through the history between these following locations:

• other-project/trunk/tools/web-ping.py
• other-project/trunk/tools/web-ping/
• WebPing/trunk/

Using a series of git filter-branch invocations, I managed to move everything to the root of the repository:

$ git filter-branch --force --prune-empty --tree-filter 'test -d ./other-project/trunk/tools && cp -axv ./other-project/trunk/tools/* ./ && rm -rf ./other-project/trunk/tools || echo "No tools folder found"' -- --all
$ git filter-branch --force --prune-empty --tree-filter 'test -d ./other-project/trunk/tools/web-ping && cp -axv ./other-project/trunk/tools/web-ping/* ./ && rm -rf ./other-project/trunk/tools/web-ping || echo "No web-ping folder found"' -- --all
$ git filter-branch --force --prune-empty --tree-filter 'test -d ./WebPing/trunk && cp -axv ./WebPing/trunk/* ./ && rm -rf ./WebPing/trunk || echo "No trunk folder found"' -- --all

Hide and obfuscate hard-coded content

As WebPing was created for internal needs in my previous job, its original code base contains lots of references to the former infrastructure it lives in. My professional standards requires me to remove all these sensible informations before making WebPing available to the public.

For example, here is the commands which allowed me to remove all references to hostnames of our intranets:

$ git filter-branch --force --prune-empty --tree-filter 'find . -type f -exec perl -i -pe "s/([\w-.]*?)\.(company(-intranet|-extention)?)\.(fr|com|net|org)/intranet\.example\.com/g" "{}" \;' -- --all

The Perl one-liner embedded in the command above will only apply the regular expression on a line-by-line basis. If you want to have the regexp applied on the whole content of each file, you have to use Perl’s slurp mode (source of that tip):

$ git filter-branch --force --prune-empty --tree-filter 'perl -0777 -i -pe "s/MAILING_LIST\s*=\s*\[(.*?)\]/MAILING_LIST = \[\]/gs" ./web-ping.py' -- --all

The specific example above helped me removed the content of the MAILING_LIST Python list found in web-ping.py, in order to protect from spam the email addresses of my former co-workers that were unfortunately hard-coded in that variable.

Another place to hunt for sensible information is commit messages. These can be easily modified thanks to the --msg-filter option. Here is how I removed references to our internal Trac tickets:

$ git filter-branch --force --msg-filter 'sed "s/ (see ticket:666)//g"' -- --all

I also had to remove line returns introduced by abusive usage of Windows text editors (remember, WebPing was born in a corporate environment):

$ git filter-branch --force --prune-empty --tree-filter 'perl -i -pe "s/\r//" ./*' -- --all

The last useful command I use was the following, to fix author’s name and email:

$ git filter-branch --force --env-filter '
    if [ "$GIT_AUTHOR_NAME" = "deldycke" ]
      then
        export GIT_AUTHOR_NAME="Kevin Deldycke"
        export GIT_AUTHOR_EMAIL="kevin@deldycke.com"
    fi
    if [ "$GIT_AUTHOR_NAME" = "diehr" ]
      then
        export GIT_AUTHOR_NAME="Matthieu Diehr"
        export GIT_AUTHOR_EMAIL="matthieu.diehr@gmail.com"
    fi
  ' -- --all

By using a dozen variations of the commands above, and carefully reviewing the code, I was able to engineer a clean code history.

But I certainly have been a little too blunt with these regular expressions. Some of them were able to act on binary content. As a result, I had to restore static images to their original copy.

Final steps

Now that your code is clean, all you need is to recreate you tag and fix the init tag date before committing everything to GitHub:

$ git tag -f "0.0" bad4ff7fc48b8b34f6f661d75c782c7fc0d098c5
$ git tag -f "0.1" 590ac9953df0e3bc76fd02615471e36a9796a065
$ git tag -f "0.2" 33f731054042b02c6d2600e7aead5bb7c4991b12
$ git filter-branch --env-filter '
      if [ $GIT_COMMIT = 361224542bc73bba747c7ca382e992e2cdd0c356 ]
      then
          export GIT_AUTHOR_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
          export GIT_COMMITTER_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
      fi' -- --all
$ git remote add origin git@github.com:kdeldycke/webping.git
$ git push -u origin master
$ git push --tags

If you want to extract mails out of your .nsf database, this might be the tool you’re looking for. Bonus point: it’s written in Python !

Installing nlconverter and its dependencies

Here is how I installed nlconverter on a Windows 2000 (SP4) machine:

1. First I downloaded and installed the official Python builds for Windows (2.6.6 precisely):
   
   
   
   
   
2. Then Python for Windows extensions (build 214 for Python 2.6 in my case):
   
   
   
   
3. Finally I had to download the latest icalendar archive, then extract the \iCalendar-1.2\src\icalendar folder to C:\Python26\Lib\site-packages\:
   
4. Next step is to download nlconverter itself and extract it:
   

nlconverter GUI

First thing you have to do is to create an export of your mails as a .nsf database. Follow the previous link to get the instructions.

Now let’s convert this nsf to a mbox. nlconverter’s FAQ tells you to run the gui.exe program to perform the conversion.

Unfortunately it didn’t work for me:

So I tried the alternative approach by using the command line.

nlconverter command line

Again, most of the things I’m writing here are based on nlconverter’s FAQ:

1. First, we have to download the notes2mbox.py script from nlconverter’s mercurial repository, as this file is not distributed in the winnlc-alpha-1.zip archive I unzipped previously. Let’s put notes2mbox.py in C:\winnlc-alpha-1\:
   
2. Now we’ll modify the notes2mbox.py script to set the password (via the notesPasswd variable) and location (notesNsfPath variable) of the .nsf file. Here are the modifications I applied:

   --- notes2mbox.py.orig	2010-09-02 13:49:58.000000000 +0200
   +++ notes2mbox.py	2010-09-02 13:51:24.000000000 +0200
   @@ -14,8 +14,8 @@
    import NlconverterLib
   
    #Constantes
   -notesPasswd = "foobar"
   -notesNsfPath = "C:\\archive.nsf"
   +notesPasswd = "XXXXXXXXXXXXX"
   +notesNsfPath = "C:\\winnlc-alpha-1\\kevin-notes-big-backup-part-1.nsf"
   
    #Connection à Notes
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
   

3. Before running the script, we have to register a Notes DLL used by nlconverter:

   regsvr32 "C:\Program Files\Notes\nlsxbe.dll"
   

   
   And make the Python interpreter available system-wide:

   C:\winnlc-alpha-1>SET Path=%Path%;C:\Python26
   

4. Now we can run the notes2mbox.py script:

   C:\winnlc-alpha-1>C:\Python26\python.exe notes2mbox.py
   

If you’re lucky, you’ll get a nice mbox at the end of the process.

But I was not and the notes2mbox.py ended up with the following error:

Traceback (most recent call last):
  File "notes2mbox.py", line 21, in <module>
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
  File "C:\winnlc-alpha-1\NlconverterLib.py", line 43, in getNotesDb
    session = win32com.client.Dispatch(r'Lotus.NotesSession')
  File "C:\Python26\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
    dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 104, in _GetGoodDispatchAndUserName
    return (_GetGoodDispatch(IDispatch, clsctx), userName)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 84, in _GetGoodDispatch
    IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
pywintypes.com_error: (-2147221231, 'ClassFactory ne peut pas fournir la classe demand\xe9e', None, None)

As you can see, I tried hard to make nlconverter working, without any success. But this should not stop you to try. In fact I suspect the Lotus Notes installed on my machine to be crippled or corrupted (can’t really tell). So you may be more lucky than me. In any case, feel free to report any success or failure in the comment section below !

• Revert current local folder to revision 666:

  svn merge -rHEAD:666 ./
  

• Create an empty repository:

  svnadmin create ./my-repo
  

• Dump a repository (a sure way to migrate a subversion repository from one version to another):

  svnadmin dump ./my-repo > ./my-repo.dmp
  

• Migrate a remote Subversion repository without creating an intermediate dump file:

  ssh -C user@myserver.com "svnadmin dump /home/user/my-repo" | svnadmin load /home/user2/my-new-repo
  

• Launch a standalone Subversion server listening on port 3690 and serving all repositories located in ./repos/:

  svnserve --daemon --listen-port 3690 --root ./repos/
  

Local working copy hacking

• Recursive and case insensitive content search on non-binary files from the current folder, while ignoring .svn folders and their content:

  find ./ -type f -not -regex ".*\/.svn\/.*" -exec grep -Iil "string to search" {} \;
  

• Same thing as above but with an alternative approach (that don’t work with large folder content):

  grep -Ii "string to search" $(find . | grep -v .svn)
  

  Other alternative: use ack.

• Use sed to replace text in all files except in subversion metadatas:

  find ./ -type f -not -regex ".*\/.svn\/.*" -print -exec sed -i 's/str1/str2/g' "{}" \;
  

• Use svn delete to remove all files containing a tilde in their name without touching local subversion metadatas:

  find -type f -not -regex ".*\/.svn\/.*" -name "*˜*" -print -exec svn delete "{}" \;
  

• In a repository structure containing sub-projects (thinks of Plone’s collective repository as an example), get the list of all folders in all trunks, while ignoring subversion metadata folders:

  find ./ -type d -regex ".*\/trunk\/?.*" -not -regex ".*\/.svn\/?.*" -print
  

• Similarly to the command above, replace all occurrences of the string @coolcavemen.fr by @coolcavemen.com in all trunk subfolders while ignoring .svn content:

  find ./ -type f -regex ".*\/trunk\/.*" -not -regex ".*\/.svn\/.*" -print -exec sed -i 's/@coolcavemen\.fr/@coolcavemen\.com/g' "{}" \;
  

• Set a svn property to ignore all .mo files during commit in every folder of our local working copy containing .po files:

  find ./ -type f -name "*.po" -regex ".*\/trunk\/.*" -not -regex ".*\/.svn\/.*" -printf "%h\n" | uniq | xargs svn propset "svn:ignore" "*.mo"