Moving a Git sub-tree to its own repository

Coming from Subversion (and with Plone collective repository structure in mind), I’ve recently moved all my tiny software projects in a big standalone Git repository (named kev-code ). Now that I figured out that GitHub allows you to create unlimited amount of repositories, as long as they are open-source public projects, it make sense to emancipate some of my projects to their own repository. How do I move a sub-tree to its own repository? That’s what I talk about in this article.

First, there is an automated way of performing this task with git-subtree . You should try it first. For some reasons I didn’t investigate, git-subtree didn’t worked for me. So I’ll explain now how I did it by hand.

The idea is to revisit the history of my bloated Git repository and massively delete everything that is not related to the sub-folder I’m looking to export. In this case, I try to make a dedicated repository for my e107 importer for WordPress .

Let’s start by getting a local copy of my source repository:

$ git clone [email protected]:kdeldycke/kev-code.git
$ cd scripts

Then I’ll use the filter-branch action with a combination of find and rm to remove everything except the source code of my plugin:

$ git filter-branch --prune-empty --tree-filter 'find ./ -maxdepth 1 -not -path "./e107*" -and -not -path "./wordpress-e107*" -and -not -path "./.git" -and -not -path "./" -print -exec rm -rf "{}" \;' -- --all

Instead of the command above, I could have use the --subdirectory-filter option ( as suggested by jamessan on Stack Overflow ):

$ git filter-branch --prune-empty --subdirectory-filter e107-importer -- --all

But this doesn’t work in my case as my e107 Importer plugin didn’t started its life straight in a dedicated folder. So this command squash some of the history I want to preserve.

At this point I’m left with this following history:

Commit history of the extracted importer, tagged from e107-importer-0.1 in 2006 through 0.9, with one foreign commit for the Drupal 5.0 release

This looks pretty good, as all the history of my plugin is kept in order. But tags unrelated to my plugin are still there. Let’s remove them:

$ git tag -d coolkevmen-0.3 cool-blue-0.1 sapphire-0.1 sapphire-0.2 sapphire-0.3 sapphire-0.4

Now there is some commits polluting my history. These are left-overs of git-modules additions. I tried to removed them , but it didn’t worked. Also left in the history are unwanted merges and empty commits from an old CVS import . To clean this up, I started an interactive rebase:

$ git rebase --interactive init

There, using my text editor, I deleted the entries corresponding to these unrelated commits (namely c21a840 , 0dc1d76 , 37473a8 and c6f9f64 ), and hoped Git will be smart enough to reconstruct a clean history:

Interactive rebase in the editor, every commit prefixed with pick above the usual list of rebase commands

Luckily, it worked for me. If Git complain about such abuse, you may ignore warnings and force it to continue:

$ git rebase --continue

Now that we only have a clean sub-tree, let’s create a dedicated local Git repository to receive our branch:

$ cd ..
$ mkdir e107-importer
$ cd e107-importer
$ git init

Add a temporary origin hooked on our source repository:

$ git remote add origin ../kev-code

And import the master branch we carefully crafted (including tags):

$ git pull --tags origin master

Now we can create on GitHub the new repository that will receive our exported project:

GitHub's create-a-new-repository form filled in for e107-importer, described as a WordPress plugin to import from a legacy e107 CMS, set to public

It’s time to push our changes. Let’s replace our temporary origin to the new GitHub repository we just created:

$ git remote rm origin
$ git remote add origin [email protected]:kdeldycke/e107-importer.git
$ git push origin master --force --tags

So now we have a copy of the sub-tree of my plugin into its own repository. That’s great, but there is still some stuff to clean-up.

First, we will rewrite the repository to look as if the ./e107-importer sub-folder had been its project root since the beginning:

$ git filter-branch --tree-filter 'test -d ./e107-importer && mv ./e107-importer/* ./ || echo "No folder found"' -- --all

Then, I’ve altered some commit messages to fix inconsistencies due to sub-folder removal:

$ git filter-branch --msg-filter 'sed "s/Move the script to a dedicated folder/Rename script/g"' -- --all

Finally, at the bottom of the history, I still have my initial commit (a personal habit of mine when I initialize my Git repositories ). But its date was updated by the first filter-branch call. Let’s set its date back to epoch:

$ git filter-branch --force --env-filter \
    'if [ $GIT_COMMIT = a2a5c05aed893fdd10250b724eb6a54bc6e7f122 ]
        then
            export GIT_AUTHOR_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
            export GIT_COMMITTER_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
    fi' -- --all

We can now send our latest changes to the remote GitHub repository by forcing a push:

$ git push --force

Last thing we have to do, is to remove the plugin code from the fat source repository (I don’t like duplicates). But that’s another story for another article…

3 archived comments

Comments are closed. These were posted on Disqus, and are archived here.

Kevin Deldycke Author Mar. 24, 2011

About the ultimate clean-up mentionned at the very end of the post, here is how I removed the plugin code from its original repository (with some tips from Cedric):

git clone [email protected]:kdeldycke/kev-code.git
cd kev-code
git filter-branch         --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch e107-importer              ' -- --all
git filter-branch --force --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch wordpress-e107-importer.php' -- --all
git tag -d e107-importer-0.1 e107-importer-0.2 e107-importer-0.3 e107-importer-0.4 e107-importer-0.5 e107-importer-0.6 e107-importer-0.7 e107-importer-0.8 e107-importer-0.9
git remote rm origin
rm -rf .git/refs/original/
git reflog expire --all
git gc --aggressive --prune

After all these steps, I deleted my repository on GitHub and recreated it. This is quite brutal and dangerous, but was the only way I found to force all caches to be cleared on GitHub side.

After this re-initialisation, I pushed my local cleaned repository back to its new GitHub home:

git remote add origin [email protected]:kdeldycke/kev-code.git
git push origin master --force --tags

Kevin Deldycke Author Mar. 29, 2011
Practicing this process several times last week, I just found a better way to migrate a subtree to its own subdirectory.

Here is an example with my Sapphire theme:
```
git clone [email protected]:kdeldycke/kev-code.git
cd kev-code
git filter-branch --prune-empty --subdirectory-filter sapphire --tag-name-filter cat -- --all init..HEAD
git tag -d coolkevmen-0.3 cool-blue-0.1
git remote rm origin
rm -rf .git/refs/original/
git reflog expire --all
git gc --aggressive --prune
git filter-branch --force --env-filter \
  'if [ $GIT_COMMIT = 0a247ae0f4202813d6a019964d57521f8b7f67cf ]
     then
       export GIT_AUTHOR_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
       export GIT_COMMITTER_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
   fi' -- --all
git remote add origin [email protected]:kdeldycke/sapphire.git
git push -u origin master --force --tags
```
This worked well for several reasons.

First, unlike the e107-importer plugin discussed in my original article above, my theme was always living in its own sapphire folder. This mean I was able to use the straightforward --subdirectory-filter parameter instead of a convoluted --tree-filter.

Second, I was able to specify the init..HEAD range, which is only possible because of the way I initialize my repositories. And giving this range to Git has the nice effect of discarding the strange branching coming from a previous CSV import.

And lastly, I discovered that I can simultaneously apply a --tag-name-filter parameter, which saved all tags removed while specifying the branch interval as explained in the paragraph above.

Kevin Deldycke Author Mar. 29, 2011

Oh, and I stumble upon another annoying issue. Moving my cool-cavemen subfolder to its own repository was working using the method above. Alas, this sub-project was using some references to Git modules, which were lost in the process.

Like Unode from Stack Overflow, I managed to keep the .gitmodules file and rewrite its content with some sed commands. But old module's folders were not moved up to one level in the folder hierarchy.

After some experiments, I came with this simple strategy of temporarily renaming the .gitmodules file to get rid of this side effect. Here is the full detailed process which helped me keep everything clean and tidy:

git clone [email protected]:kdeldycke/kev-code.git
cd kev-code
git filter-branch --tree-filter "test -f ./.gitmodules && mv ./.gitmodules ./cool-cavemen/gitmodules || echo 'No .gitmodules file found'" -- --all
git filter-branch --force --prune-empty --subdirectory-filter cool-cavemen --tag-name-filter cat -- --all init..HEAD
git filter-branch --force --tree-filter "test -f ./gitmodules && mv ./gitmodules ./.gitmodules || echo 'No gitmodules file found'" -- --all
git filter-branch --force --tree-filter "test -f ./.gitmodules && sed -i 's/cool-cavemen\///g' ./.gitmodules || echo 'No .gitmodules file found'" -- --all
git tag -d coolkevmen-0.3 cool-blue-0.1 sapphire-0.1 sapphire-0.2 sapphire-0.3 sapphire-0.4
git remote rm origin
rm -rf .git/refs/original/
git reflog expire --all
git gc --aggressive --prune
git filter-branch --force --env-filter \
  'if [ $GIT_COMMIT = 0a247ae0f4202813d6a019964d57521f8b7f67cf ]
     then
       export GIT_AUTHOR_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
       export GIT_COMMITTER_DATE="Thu, 01 Jan 1970 00:00:00 +0000"
   fi' -- --all
git remote add origin [email protected]:kdeldycke/cool-cavemen.git
git push -u origin master --force --tags

Kevin Deldycke — 🦬🪒🐇🕳 yak-shaving the rabbit holes

Moving a Git sub-tree to its own repository

3 archived comments