Igor Alexandrov

Programming and something else…

Using Git Subtree to Share Code Between Rails Applications

| Comments

For different reasons (mainly NDA) I cannot write real project names in this post. To make all things clear, let’s say that we have one project called FooLog (Rails 3.2) and we have to create new project called FooMarket (Rails 3.2 too) that uses data and ROI algorithms from FooLog.

As it usually happens at the beginning of the development, FooMarket seemed to us a very simple project with a pair of controllers and five or eight models. We wanted to add simple API to FooLog and authorise FooMarket to load data from FooLog through this API (with OAuth for example). After a couple of conference Skype-calls with our customers we understood that we are doing something bigger then two controllers and five models and I said – “OMG! We are trying to create another FooLog here! Why we cannot just add another section to FooLog and call it FooMarket?”. The answer was: “Because of marketing. FooLog and FooMarket are two different investment projects and should work separately”.

After all technical specifications where approved we had a task to create application that uses a nearly data from FooLog database and all FooLog ROI modeling and analytical tools. There were two ways: to rewrite all these from the beginning or to share codebase from FooLog. Of course we choose the second approach, we had to share:

  • about 35 models
  • about 15 widely used concerns
  • state rules for all states in USA (one Ruby class per state)
  • even some locale files and specs

Decisions

How this could be done? First of all we could simply copy all these from FooLog and then paste to FooMarket and then projects would start to develop separately. But as you should understand this is road to nowhere, you would always do double work, first change something in one project then make the same changes in another project. Then somebody will forget to copy some code, or copy it wrong or something and BOOOOM you are getting HTTP 500 status in production… bad.

Ok, I remembered that a couple of years ago we used git-submodule to track code of separate library in our project. But after a couple of test commits we understood that git-submodule works well only in one direction. If you only pull code from remote repository and do not push to it, then git-submodule is for you. The most common example can be a third-author library that you use with your code and always want to have the latest version. But if you want to push your changes to shared repository then submodules stop to work correctly: you should remember to commit push both main repository and submodule, you will get conflicts nearly on every commit, you cannot handle remote branches correctly and so on… Also git-submodule special constructions like .gitmodule files in your repository. Bad.

After some additional hours of searching we found it – git-subtree. If you are using git-1.7.11 and higher then you have nothing additional to do, if older, then you have to install git-subtree plugin.

Here is the synopsis of git-subtree command:

1
2
3
4
5
git subtree add   -P <prefix> <commit>
git subtree pull  -P <prefix> <repository> <refspec...>
git subtree push  -P <prefix> <repository> <refspec...>
git subtree merge -P <prefix> <commit>
git subtree split -P <prefix> [OPTIONS] [<commit>]

How git-subtree works with Rails?

As you can remember, we have two Rails applications: foolog and foomarket; foolog is an “old” application and has all code that we want to share, foomarket is new, just created Rails app. At first, let’s create a folder where all shared code will be stored and move needed classes to it.

1
2
3
4
5
6
saturn:~/workspace/foolog (dev)$ mkdir shared
saturn:~/workspace/foolog (dev)$ mkdir shared/models
saturn:~/workspace/foolog (dev)$ mv app/models/bar.rb shared/models
saturn:~/workspace/foolog (dev)$ mv app/models/bar_holder.rb shared/models
# a lot of other commands here
saturn:~/workspace/foolog (dev)$ mv app/models/portfolio.rb shared/models

Of course don’t forget to add shared folder to GIT.

1
saturn:~/workspace/foolog (dev)$ git add shared

And tell Rails to autoload your news paths. To do this, edit config/application.rb.

1
2
3
4
5
6
7
8
9
10
11
module FooLog
  class Application < Rails::Application
    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration should go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded.

    # Custom directories with classes and modules you want to be autoloadable.
    config.autoload_paths += %W(#{config.root}/shared/models #{config.root}/shared/concerns)

  end
end

Don’t forget to commit your changes.

1
2
saturn:~/workspace/foolog (dev)$ git commit -a -m "Moved models to shared"
saturn:~/workspace/foolog (dev)$ git push origin dev

Now you should be able to work as usual, start your application, modify your classes and so on. But what about sharing code? At first we need a third GIT repository (we already have one for FooLog and one for FooMarket), let’s call this repository foosoftware. To use git-subtree, we should add this third repository as a remote:

1
2
saturn:~/workspace/foolog (dev)$ git remote add foosoftware git@jetrockets.ru:foosoftware.git
saturn:~/workspace/foolog (dev)$ git fetch foosoftware

In 99% of cases now you have two remotes configured: origin and foosoftware.

Splitting

If you make any commits to the subproject inside master, you can use git subtree split to backport the commits to the right timeline. Let’s split commits that were made to shared folder and push them to foosoftware repository.

1
2
saturn:~/workspace/foolog (dev)$ git subtree split --prefix shared -b backport
saturn:~/workspace/foolog (dev)$ git push foosoftware backport:dev

What we’ve done? We splitted all out commits, that changed ‘shared’ folder to a separate branch called backport (use git branch to find it) and then push all these commits to foosoftware/dev branch.

Merging

Now let’s switch to FooMarket and add foosoftware repository to it.

1
2
3
saturn:~/workspace/foomarket (dev)$ git remote add foosoftware git@jetrockets.ru:foosoftware.git
saturn:~/workspace/foomarket (dev)$ git fetch foosoftware
saturn:~/workspace/foomarket (dev)$ git subtree add --prefix shared foosoftware/dev

Of course modify your config/application.rb.

1
2
3
4
5
6
7
8
9
10
11
module FooMarket
  class Application < Rails::Application
    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration should go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded.

    # Custom directories with classes and modules you want to be autoloadable.
    config.autoload_paths += %W(#{config.root}/shared/models #{config.root}/shared/concerns)

  end
end

Now all code from foosoftware repo is placed to shared folder, Rails is configured and we are ready to commit it and push to FooMarket origin.

1
2
saturn:~/workspace/foomarket (dev)$ git commit -a -m "Add shared folder"
saturn:~/workspace/foomarket (dev)$ git push origin dev

After the initial merge with git subtree add, for subsequent merges you use git subtree merge (for any command, you need to remember to use the prefix option to tell it the location of the subdirectory). For example if I want to update FooMarket with latest code from FooLog, then I first do git subtree split in FooLog and then git subtree merge in FooMarket.

1
2
3
saturn:~/workspace/foomarket (dev)$ git fetch foosoftware
saturn:~/workspace/foomarket (dev)$ git subtree merge --prefix shared foosoftware/dev
saturn:~/workspace/foomarket (dev)$ git push origin dev

Also it is important to know that subtree merging works in both sides. If I changed shared code in FooMarket and want to share is with FooLog then I first do git subtree split in FooMarket, push to foosoftware and then do git subtree merge in FooLog.

Conclusion

We are using git-subtree for more then two months and have only good emotions:

  • with subtree merging you don’t have any limitations on your shared repository, you can use all GIT features like branches, tags and so on;
  • it works in both sides;
  • unlike the git submodule command, git-subtree doesn’t produce any special constructions (like .gitmodule files or gitlinks) in your repository, and doesn’t require end-users of your repository to do anything special or to understand how subtrees work.

Comments