Pull latest changes for all git submodules

Asked 2023-09-20 20:28:47 View 809,141

We're using git submodules to manage a couple of large projects that have dependencies on many other libraries we've developed. Each library is a separate repo brought into the dependent project as a submodule. During development, we often want to just go grab the latest version of every dependent submodule.

How do I pull the latest changes for all git submodules?

  • git-deep should help with this. - anyone
  • @Brad do you want to update your copies of submodules to the commit revs named in the master project; or do you want to pull the latest HEAD commit from every submodule? Most of the answers here address the former; many people want the latter. - anyone
  • You can find this answer with Git flow with submodules helpful stackoverflow.com/a/70910397/9566462 - anyone

Answers

If it's the first time you check-out a repo you need to use --init first:

git submodule update --init --recursive

For git 1.8.2 or above, the option --remote was added to support updating to latest tips of remote branches:

git submodule update --recursive --remote

This has the added benefit of respecting any "non default" branches specified in the .gitmodules or .git/config files (if you happen to have any, default is origin/master, in which case some of the other answers here would work as well).

For git 1.7.3 or above you can use (but the below gotchas around what update does still apply):

git submodule update --recursive

or:

git pull --recurse-submodules

if you want to pull your submodules to latest commits instead of the current commit the repo points to.

See git-submodule(1) for details

Answered   2023-09-20 20:28:47

  • Probably you should use git submodule update --recursive nowadays. - anyone
  • Performance improvement: git submodule foreach "(git checkout master; git pull)&" - anyone
  • update will update each submodule to the specified revision, not update it to the latest for that repository. - anyone
  • Just to add, blindly sticking origin master at the end of this command might have unexpected results if some of your submodules are tracking a different branch or location name of that particular submodule. Obvious to some, but probably not to everyone. - anyone
  • Just to clarify for everyone. git submodule update --recursive looks to see which revision the parent repository has stored for each submodule, then checks out that revision in each submodule. It does NOT pull the latest commits for each submodule. git submodule foreach git pull origin master or git pull origin master --recurse-submodules is what you want if you intend to update each submodule to the latest from their origin repositories. Only then will you get pending changes in the parent repo with updated revision hashes for submodules. Check those in and you're good. - anyone
git pull --recurse-submodules --jobs=10

a feature git first learned in 1.8.5.

Until the bug is fixed, for the first time you do need to run

git submodule update --init --recursive

Answered   2023-09-20 20:28:47

  • upvoted, i use this: alias update_submodules='git pull --recurse-submodules && git submodule update' - anyone
  • This works if the submodules have already been pulled at least once but for submodules that have never been checked out, see gahooa's answer below. - anyone
  • This will pull up to the version the top repo specifies; it does NOT pull HEAD. For example if TopRepo specifies a version 2 behind HEAD for SubRepo, this will pull SubRepo with that version that's 2 behind. Other answers here pull HEAD in SubRepo. - anyone
  • Note that neither git pull --recurse-submodules nor git submodule update --recursive does not initialize newly added submodules. To initialize them you need run git submodule update --recursive --init. Quote from manual: If the submodule is not yet initialized, and you just want to use the setting as stored in .gitmodules, you can automatically initialize the submodule with the --init option. - anyone
  • maybe add a hint to git submodule update --recursive --remote which also updates the submodules to the remote latest revision instead of the stored SHA-1. - anyone

On init running the following command:

git submodule update --init --recursive

from within the git repo directory, works best for me.

This will pull all latest including submodules.

Explained

git - the base command to perform any git command
    submodule - Inspects, updates and manages submodules.
        update - Update the registered submodules to match what the superproject
        expects by cloning missing submodules and updating the working tree of the
        submodules. The "updating" can be done in several ways depending on command
        line options and the value of submodule.<name>.update configuration variable.
            --init without the explicit init step if you do not intend to customize
            any submodule locations.
            --recursive is specified, this command will recurse into the registered
            submodules, and update any nested submodules within.

After this you can just run:

git submodule update --recursive

from within the git repo directory, works best for me.

This will pull all latest including submodules.

Answered   2023-09-20 20:28:47

  • Yes -- the highest voted answer was the best way to do it in '09, but this is definitely simpler and more intuitive now. - anyone
  • @MichaelScottCuthbert thanks, i'm sure in another 3 years this command will be crazy too - anyone
  • Nevertheless, this does not checkout the latest revision from the submodule, only the latest revision that the parent is tracking. - anyone
  • @NathanOsman which is what you want...you will end up with broken code by not following the parents revision tracking. If you are the maintainer of the parent you can update those yourself and commit them. - anyone
  • Yes, but from my understanding, that isn't what the OP wanted. - anyone

Note: This is from 2009 and may have been good then but there are better options now.

We use this. It's called git-pup:

#!/bin/bash
# Exists to fully update the git repo that you are sitting in...

git pull && git submodule init && git submodule update && git submodule status

Just put it in a suitable bin directory (/usr/local/bin). If on Windows, you may need to modify the syntax to get it to work :)

Update:

In response to the comment by the original author about pulling in all of the HEADs of all of the submodules -- that is a good question.

I am pretty sure that git does not have a command for this internally. In order to do so, you would need to identify what HEAD really is for a submodule. That could be as simple as saying master is the most up to date branch, etc...

Following this, create a simple script that does the following:

  1. check git submodule status for "modified" repositories. The first character of the output lines indicates this. If a sub-repo is modified, you may NOT want to proceed.
  2. for each repo listed, cd into it's directory and run git checkout master && git pull. Check for errors.
  3. At the end, I suggest you print a display to the user to indicate the current status of the submodules -- perhaps prompt them to add all and commit?

I'd like to mention that this style is not really what git submodules were designed for. Typically, you want to say "LibraryX" is at version "2.32" and will stay that way until I tell it to "upgrade".

That is, in a sense, what you are doing with the described script, but just more automatically. Care is required!

Update 2:

If you are on a windows platform, you may want to look at using Python to implement the script as it is very capable in these areas. If you are on unix/linux, then I suggest just a bash script.

Need any clarifications? Just post a comment.

Answered   2023-09-20 20:28:47

  • I don't think that's what I want. Won't that pull the version of the submodules that the super-project was last committed with. I want to pull the head version of all the submodules. - anyone
  • This works great, and works not only to update the submodules but also to fetch them for the first time if that's what you need. - anyone
  • I'm just getting "There is no tracking information for the current branch. Please specify which branch you want to merge with." No matter what I try :/ - anyone
  • Why not create an alias for it? git config --global alias.pup '!git pull && git submodule init && git submodule update && git submodule status' and then use it as git pup without any scripting. - anyone
  • Thank you, for some reason even though I have git 1.9.1 I had to perform git submodule init after first pull that had submodules included, so that everything would start working properly. - anyone

Henrik is on the right track. The git submodule foreach command can execute any arbitrary shell script. Two options to pull the very latest might be:

git submodule foreach git pull origin master

and:

git submodule foreach /path/to/some/cool/script.sh

That will iterate through all initialized submodules and run the given commands.

Answered   2023-09-20 20:28:47

  • Nice! This is perhaps the best answer! If your subrepo contains subrepos, however, you'll want to add --recusrive too, like this: git submodule foreach --recursive git pull origin master. See man git submodule and search for the section which says foreach [--recursive] <command>, for details. - anyone

The following worked for me on Windows.

git submodule init
git submodule update

Answered   2023-09-20 20:28:47

  • This clearly is not what the OP asked for. It will only update to the associated submodule commit and not the latest one. - anyone
  • This is however the only thing on this page that got git to pull submodules the first time I checked out a repo - anyone
  • Can also use: git submodule update --init --recursive (particularly if the submodule in question is RestKit from a fresh clone) - anyone
  • I needed to init first. Thanks - anyone

For me, git 2.24.03, get updated to latest commit of remote branches defined in .gitmodules.

git submodule update --recursive --init

git submodule update --recursive --remote

git version 2.24.3 (Apple Git-128)

Please Note: Someone said that git pull --recurse-submodulesis the same as git submodule update --recursive --remote. But from my test, git pull --recurse-submodules may not get updated to latest commit of remote branches defined in .gitmodules.

Answered   2023-09-20 20:28:47

  • do you know why the 2nd command is needed with --remote? - anyone
  • this works with the --init thanks - anyone

First time

Clone and Init Submodule

git clone git@github.com:speedovation/kiwi-resources.git resources
git submodule init

Rest

During development just pull and update submodule

git pull --recurse-submodules  && git submodule update --recursive

Update Git submodule to latest commit on origin

git submodule foreach git pull origin master

Preferred way should be below

git submodule update --remote --merge

note: last two commands have same behaviour

Answered   2023-09-20 20:28:47

  • I did a git clone with no submodules by mistake and all other options didn't worked, no one did clone submodules. Using yours, git submodule update did the trick. Now I'm downloading submodules data missing from the clone first step. Thank you. I'm not good at git :C - anyone
  • This anser is actually a very good answer to ask a question here on top: why do I have to ".. --recursive-submodules.." and then additionally the "... update ..." and even "...foreach..." later to get latest commit? All this does not look GIT like at all! What is "update" doing and why do I have to manually go to each module to pull? Isn't that what "... --recurse-submodules .." is doing? Any hints? - anyone

Edit:

In the comments was pointed out (by philfreo ) that the latest version is required. If there is any nested submodules that need to be in their latest version :

git submodule foreach --recursive git pull

-----Outdated comment below-----

Isn't this the official way to do it ?

git submodule update --init

I use it every time. No problems so far.

Edit:

I just found that you can use:

git submodule foreach --recursive git submodule update --init 

Which will also recursively pull all of the submodules, i.e. dependancies.

Answered   2023-09-20 20:28:47

  • Your answer doesn't answer the OP's question, but to do what you've proposed you can just say git submodule update --init --recursive - anyone
  • I see, latest version is needed. Well this might be usefull if there is nested submodules: git submodule foreach --recursive git pull - anyone
  • I couldn't make any of these actually download anything -- "git submodule update --init --recursive" worked for me however. - anyone

As it may happens that the default branch of your submodules is not master, this is how I automate the full Git submodules upgrades:

git submodule init
git submodule update
git submodule foreach 'git fetch origin; git checkout $(git rev-parse --abbrev-ref HEAD); git reset --hard origin/$(git rev-parse --abbrev-ref HEAD); git submodule update --recursive; git clean -dfx'

Answered   2023-09-20 20:28:47

  • out of the many answers to the many questions, this one worked for me (2019, github error with specific hash ids) - anyone
  • This is what I've been looking for as I have submodules with master and main. Thanks! - anyone

I don't know since which version of git this is working, but that's what you're searching for:

git submodule update --recursive

I use it with git pull to update the root repository, too:

git pull && git submodule update --recursive

Answered   2023-09-20 20:28:47

The above answers are good, however we were using git-hooks to make this easier but it turns out that in git 2.14, you can set git config submodule.recurse to true to enable submodules to to updated when you pull to your git repository.

This will have the side effect of pushing all submodules change you have if they are on branches however, but if you have need of that behaviour already this could do the job.

Can be done by using:

git config submodule.recurse true

Answered   2023-09-20 20:28:47

  • Gotta love this option, unfortunately still need to use git submodule init before hand though if your submodule isn't yet initialized. - anyone

To clarify a few things based on already available answers of pulling "latest" code of each submodule from remote.

If "latest" means the submodule pointers that were checked in, then by all means use:

git submodule update --recursive
  - or -
git pull --recurse-submodules --jobs=X

If "latest" means the latest of main, then something like this can work:

git submodule foreach "git checkout main && git pull"

Unfortunately, this means there's no "--jobs" option, so we cannot run it in parallel. The closest I've seen to running this in parallel is by using the pfs python code.

Answered   2023-09-20 20:28:47

I often use this commands, it works so far.

git pull
git submodule foreach --recursive git checkout master
git submodule foreach --recursive git pull

Hope this faster.

Answered   2023-09-20 20:28:47

As noted in antitoxic's answer, a simple git submodule foreach --recursive git pull can be enough.

Francis Bacon's answer notes git pull --recurse-submodules can differ.

You could test out and see what is going on with a verbose option:

git pull -v --recurse-submodules

But for that, you will need Git 2.40 (Q1 2023).

"git pull -v --recurse-submodules"(man) attempted to pass -v down to underlying git submodule update(man), which did not understand the request and barfed: this has been corrected with Git 2.40 (Q1 2023).

See commit 6f65f84 (10 Dec 2022) by Sven Strickroth (csware).
(Merged by Junio C Hamano -- gitster -- in commit b3b9e5c, 28 Dec 2022)

submodule: accept -v for the update command

Signed-off-by: Sven Strickroth

Since a56771a ("builtin/pull: respect verbosity settings in submodules", 2018-01-25, Git v2.17.0-rc0 -- merge listed in batch #3), "git pull -v --recurse-submodules"(man) propagates the -v to the submodule command, but because the latter command does not understand the option, it barfs.

Teach "git submodule update"(man) to accept the option to fix it.

No more "barfing" (IE. displaying the usage 'git submodule foreach [--quiet] [--recursive] [--] <command>') because the -v was an unknown option for git submodule.

Answered   2023-09-20 20:28:47

Git for windows 2.6.3:

git submodule update --rebase --remote

Answered   2023-09-20 20:28:47

  • That's the only one that worked for me. I wasn't even able to init or update as the submodule pointer was pointing to a version that wasn't in the remote anymore - anyone

From the top level in the repo:

git submodule foreach git checkout develop
git submodule foreach git pull

This will switch all branches to develop and pull latest

Answered   2023-09-20 20:28:47

  • Do you have something like an Everything sln file which adds all the project references in the tree? Also what error do you see? Can you check your gitignore file too - anyone
  • git submodule foreach git pull origin master Had to append the branch I wanted to fetch. other than that, worked perfectly. - anyone
  • You may want to add --recursive as well in order to recurse down into submodules within submodules. Ex: git submodule foreach --recursive git pull - anyone

I did this by adapting gahooa's answer above:

Integrate it with a git [alias] ...

If your parent project has something like this in .gitmodules:

[submodule "opt/submodules/solarized"]
    path = opt/submodules/solarized
    url = git@github.com:altercation/solarized.git
[submodule "opt/submodules/intellij-colors-solarized"]
    path = opt/submodules/intellij-colors-solarized
    url = git@github.com:jkaving/intellij-colors-solarized.git

Add something like this inside your .gitconfig

[alias]
    updatesubs = "!sh -c \"git submodule init && git submodule update && git submodule status\" "

Then to update your submodules, run:

git updatesubs

I have an example of it in my environment setup repo.

Answered   2023-09-20 20:28:47

All you need to do now is a simple git checkout

Just make sure to enable it via this global config: git config --global submodule.recurse true

Answered   2023-09-20 20:28:47

Here is the command-line to pull from all of your git repositories whether they're or not submodules:

ROOT=$(git rev-parse --show-toplevel 2> /dev/null)
find "$ROOT" -name .git -type d -execdir git pull -v ';'

If you running it in your top git repository, you can replace "$ROOT" into ..

Answered   2023-09-20 20:28:47

I worked on this simple shell script which works fine for me.

#!/bin/bash

#git init
git init

#git clone repo including submodules
git clone --recurse-submodules https://github.com/username/project.git

#change directory -repo
cd project

#update the remote ie tag/commits
git submodule update --remote

#add commit 
git commit -a -m "commit in submodule"

#git push 
git push -u origin 

Answered   2023-09-20 20:28:47

Remark: not too easy way, but workable and it has its own unique pros.

If one want to clone only HEAD revision of a repository and only HEADs of all the its submodules (i.e. to checkout "trunk"), then one can use following Lua script. Sometimes simple command git submodule update --init --recursive --remote --no-fetch --depth=1 can result in an unrecoverable git error. In this case one need to clean up subdirectory of .git/modules directory and clone submodule manually using git clone --separate-git-dir command. The only complexity is to find out URL, path of .git directory of submodule and path of submodule in superproject tree.

Remark: the script is only tested against https://github.com/boostorg/boost.git repository. Its peculiarities: all the submodules hosted on the same host and .gitmodules contains only relative URLs.

-- mkdir boost ; cd boost ; lua ../git-submodules-clone-HEAD.lua https://github.com/boostorg/boost.git .
local module_url = arg[1] or 'https://github.com/boostorg/boost.git'
local module = arg[2] or module_url:match('.+/([_%d%a]+)%.git')
local branch = arg[3] or 'master'
function execute(command)
    print('# ' .. command)
    return os.execute(command)
end
-- execute('rm -rf ' .. module)
if not execute('git clone --single-branch --branch master --depth=1 ' .. module_url .. ' ' .. module) then
    io.stderr:write('can\'t clone repository from ' .. module_url .. ' to ' .. module .. '\n')
    return 1
end
-- cd $module ; git submodule update --init --recursive --remote --no-fetch --depth=1
execute('mkdir -p ' .. module .. '/.git/modules')
assert(io.input(module .. '/.gitmodules'))
local lines = {}
for line in io.lines() do
    table.insert(lines, line)
end
local submodule
local path
local submodule_url
for _, line in ipairs(lines) do
    local submodule_ = line:match('^%[submodule %"([_%d%a]-)%"%]$')
    if submodule_ then
        submodule = submodule_
        path = nil
        submodule_url = nil
    else
        local path_ = line:match('^%s*path = (.+)$')
        if path_ then
            path = path_
        else
            submodule_url = line:match('^%s*url = (.+)$')
        end
        if submodule and path and submodule_url then
            -- execute('rm -rf ' .. path)
            local git_dir = module .. '/.git/modules/' .. path:match('^.-/(.+)$')
            -- execute('rm -rf ' .. git_dir)
            execute('mkdir -p $(dirname "' .. git_dir .. '")')
            if not execute('git clone --depth=1 --single-branch --branch=' .. branch .. ' --separate-git-dir ' .. git_dir .. ' ' .. module_url .. '/' .. submodule_url .. ' ' .. module .. '/' .. path) then
                io.stderr:write('can\'t clone submodule ' .. submodule .. '\n')
                return 1
            end
            path = nil
            submodule_url = nil
        end
    end
end

Answered   2023-09-20 20:28:47

I think you'll have to write a script to do this. To be honest, I might install python to do it so that you can use os.walk to cd to each directory and issue the appropriate commands. Using python or some other scripting language, other than batch, would allow you to easily add/remove subprojects with out having to modify the script.

Answered   2023-09-20 20:28:47