Multi-repository build plans with Atlassian Bamboo

There are two ways to combine multiple repositories in an Atlassian Bamboo build plan. One, you add all the required repositories to the build plan, which implies that one repository becomes the main repository. Two, you chose a main repository and include the other repositories as Git submodules in the main repository. Both approaches have downsides due to limitations in Bamboo and Git. But, with the workaround described in this post, the second approach works best for me.

Multi-repository build limitations

The first approach mentioned above, where all required repositories are added to the Bamboo build plan, has a serious limitation. Automatic builds or custom builds for specific commits only relates to the main repository, which implies that the build will be based on the head of the master (or a selected branch when adding the repository) of all the other included repositories. In other words, you cannot specify the commits to be used in a custom build for other than the main repository. Also, changes to any of the non-main repositories are not automatically detected and scheduled for a new build by Bamboo.

The second approach relies on Git submodules, a feature that probably should be avoided. Bamboo does in principle support this feature, but in practice it is problematic. The main problem is a long standing issue that the access credentials used for checking out the main repository is not used when updating the submodule(s). Therefore, Bamboo requires you to upload and configure SSH keys (or other access credentials) on each Bamboo Agent for the agent to gain access to the submodule repositories. Uploading credentials to the agent server can be done as a task in a build job before checking out the source code, as shown below, but it may go against the security policy of your company to expose the key text directly in the Bamboo build plan interface. Another problem with the submodule approach, is that it requires you to create build jobs that performs automatic commits to the main repository each time a submodule is updated, otherwise Bamboo will not detect changes to the submodules of the main repository.

Give the agent access

If you chose to use the Git submodules approach you need to provide the agent with access to the submodules. Here follows scripts to include as tasks in the build plan for copying an SSH private key to the Agent server. The Atlassian documentation provides details on how to provide credentials for submodule access via HTTPS

Linux:

mkdir -p ~/.ssh
chmod 0700 ~/.ssh
rm -f ~/.ssh/id_rsa
cat <<EOT >~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
EOT
chmod 0600 ~/.ssh/id_rsa

 

Windows:

@echo off

if exist "%USERPROFILE%" (
if not exist "%USERPROFILE%\.ssh" mkdir "%USERPROFILE%\.ssh"
)

call :data1 >"%USERPROFILE%\.ssh\id_rsa"

exit /b

:data1
echo -----BEGIN RSA PRIVATE KEY-----
echo ...
echo -----END RSA PRIVATE KEY-----

exit /b

 

Bonus: An alternativ approach

There is an alternative way to construct a project that requires code from multiple "sub" repositories, which involves copying all the code to a single repository. Hence, the main repository consolidates code from all the other repositories. So, instead of including repositories as Git submodule directories, you copy the code of each repository into subdirectories of the main repository. You have to repeat this process (manually or automatically) for each update (or major revision) of the "sub" repositories. This approach is somewhat equivalent to the submodule approach, but it bypasses some of the downsides of using Git submodules.

In the end, I found that the submodule approach works fine for me (when uploading credentials to the Agent server), but your mileage may vary.