Integrating documentation from multiple sources: Using Git Submodules, an alternative to MadCap Flare’s Global project linking (GPL)

Creating comprehensive documentation often requires combining content from various sources. For technical writers, this might mean pulling in documentation specific to different products, components, or even teams, all while maintaining a single, unified output. The challenge lies in doing this efficiently, ensuring content stays updated, and avoiding the pitfalls of manual copying.

Imagine needing to build a master documentation set that includes content from two or more independently developed product documentation projects. The ideal solution allows you to reuse existing content, add new information specific to the combined set, and automatically incorporate updates from the source projects.

Exploring Content Reuse Features: Global Project Linking

When considering how to achieve content reuse within a documentation tool like MadCap Flare, a feature that often comes to mind is Global Project Linking. This functionality is designed to allow you to share and reuse content and resources across different Flare projects.

The idea is appealing: designate certain projects as “parent” projects containing reusable content (topics, snippets, stylesheets, images, etc.), and then link “child” projects to them, importing the necessary files. This offers centralized management for shared resources.

Initially, one might explore Global Project Linking with the hope that it can solve the problem of integrating full documentation sets from independent product projects. You could, in theory, treat each product’s documentation as a “parent” project and your combined documentation as a “child,” importing all required content.

Encountering Limitations with Large-Scale Integration

While Global Project Linking works effectively for sharing smaller, more controlled sets of resources like stylesheets, templates, or common snippets, applying it to integrate entire documentation projects, especially large ones, often reveals significant practical limitations.

Through trying this approach for integrating content from multiple large source projects, several challenges become apparent:

Performance Issues with Large Data: Importing and managing a vast number of files from large source projects via Global Project Linking can severely impact the performance of your documentation tool. Loading, building, and even basic editing within the combined project can become slow, unresponsive, or unstable, particularly when dealing with tens of thousands of files or very large individual files. This significantly hinders the productivity and reliability of the authoring environment.
Complexity with Multiple Writers and Git Repositories: In a collaborative environment where multiple writers work on the combined documentation, and the source documentation is managed in separate Git repositories, coordinating updates through Global Project Linking becomes cumbersome. The linking mechanism might not seamlessly integrate with the standard Git workflow across multiple independent repositories. Ensuring every writer has the correct, latest versions of all imported files from their respective source repositories, and handling conflicts or updates, can become a manual and error-prone process.
Reliability Concerns: With intricate linking structures involving numerous files and projects, Global Project Linking can sometimes be less reliable for maintaining file integrity compared to a Git-native solution. Issues like broken links or difficulty consistently resolving updates can occur, especially when the underlying source files are frequently changing across different Git branches and repositories.

While Global Project Linking is a valuable tool for specific reuse scenarios, these practical limitations, particularly concerning performance with large projects and the complexities of managing updates across multiple users and Git repositories, highlight the need for a different approach when the goal is full content integration from independently version-controlled sources.

The Git Submodule Solution: A Robust Alternative

For scenarios involving integrating documentation from multiple Git repositories where the content volume is substantial and collaboration involves multiple writers, Git Submodules offer a more robust and Git-centric solution.

Git submodules allow one Git repository (your main, combined documentation project) to include, as a subdirectory, another Git repository (each source documentation project). Instead of copying the source files, your main repository simply records a reference to a specific commit in the external source repository.

This approach aligns perfectly with a Git-based workflow and provides precise control over which version of the source content is included in your combined documentation.

Getting Started with Git Submodules

Implementing this solution involves setting up your main documentation project to include the source documentation projects as submodules.

Clone Your Main Documentation Project: Begin by cloning the Git repository for your main, combined documentation project:

Bash
git clone [url-of-your-main-repo]

Navigate into the project directory:

Bash
cd [your-main-project-folder]
Initialize and Clone Submodules: If submodules are already configured in your main repository (typically defined in a .gitmodules file), you can initialize and clone them using a single command:

Bash
git submodule update --init --recursive

This command reads the .gitmodules file, initializes the submodules defined there, and clones the content from their respective repositories at the specific commits referenced by your main project. If you are setting up submodules manually for the first time, you would add each source documentation repository as a submodule to your main project. Choose a path within your main project where the submodule content should reside:

Bash
git submodule add --depth 1 [url-of-source-repo-1] [path-for-repo-1]
git submodule add --depth 1 [url-of-source-repo-2] [path-for-repo-2]
# Add more submodules as needed

The --depth 1 option performs a shallow clone, which is faster for initial setup as it fetches only the latest commit. After adding them, run git submodule update --init to finalize the cloning. After these steps, your main project directory contains the content from the source documentation projects in the specified subdirectories.

Keeping Submodule Content Current

Your main documentation project now holds references to specific versions of the submodule content. To pull in the latest changes from the independent source documentation repositories, you need to update these references.

Recommended Workflow: Regular Synchronization

To ensure you are always working with the most current source content, incorporate a step at the beginning of your workday to synchronize your main project and its submodules.

Pull Latest from Main Repository: Always start by getting the latest changes for your main documentation project:

Bash
git pull origin main
(Replace main with the default branch name of your main repository).
Update Submodules: After updating the main repository, update the submodules to pull in the latest content from their source repositories. While you can perform the update manually using Git commands (shown below), you might choose to use an automated script to streamline this process. Here are examples of batch scripts (for Windows) that illustrate how you could automate the submodule update process.

Remember to replace placeholders like [main-branch-name], [submodule-branch-name], and [submodule-path] with your actual names and paths.

Example Script 1: Update Main and Submodules (and record new references)

This script pulls the latest from the main repository, updates each submodule to the latest commit on a specified branch (e.g., their main or master branch), and then stages, commits, and pushes the updated submodule references in the main repository.

Code snippet

Example Script 2: Pull Main and Synchronize Submodules (to main repo’s state) This script pulls the latest from the main repository and then updates the submodules to match the specific commit that the latest commit in the main repository is configured to reference for each submodule. This is useful for ensuring your local working copy of the submodules aligns with the state intended by the team’s latest work on the main project.

Code snippet

Place the chosen script in the root of your main project and execute it from your terminal each morning.
Open Your Documentation Tool: Launch your documentation authoring tool (like MadCap Flare) and open your main project.
Begin Your Work: You are now ready to work on the combined documentation project, incorporating the most recent content from the source projects.

Manual Submodule Update

If you prefer not to use a script, or if a script fails, you can update submodules manually.

Pull Main Repository: Ensure your main project is up-to-date: Bashgit pull origin main (Replace main with your default branch name).
Manually Fetch and Reset Submodules: Use the git submodule foreach command to run commands in each submodule directory:

Bash
git submodule foreach 'git fetch origin [submodule-branch-name] --depth=1 && git reset --hard origin/[submodule-branch-name]'

(Replace [submodule-branch-name] with the branch you want to track). This fetches the latest from the specified branch in each submodule and resets the submodule to that commit.
Record Changes in Main Repository: Stage, commit, and push the updated submodule references in your main project:

Bash
git add [path-to-submodule-1] [path-to-submodule-2] ...
# Add all submodule paths
git commit -m "Update submodules to latest"
git push origin main
(Replace paths and main accordingly).

Run these commands whenever you need to synchronize your local copy of the submodules with the latest content from their source repositories and record that state in your main project’s history.

Understanding Submodule Commits

It’s important to understand what happens when you update submodules and then commit and push in your main repository. You are not making changes to the submodule repositories themselves.

Instead, your main repository tracks which specific commit from each submodule repository it is currently using. When you update a submodule and then commit and push in the main repository, you are updating the reference in the main repository’s history to point to the new commit in the submodule. This ensures that when other collaborators clone or pull the main project, Git knows which specific versions of the submodules to provide them.

Troubleshooting Automated Updates

Automated scripts can sometimes encounter issues. Common reasons include:

Network Connectivity: Unstable or dropped internet prevents access to remote repositories.
Authentication: Incorrect or expired credentials for accessing the source repositories.
Changes to Remote URLs: If the URLs of the source repositories change.
Local Modifications in Submodules: Unintended changes within a submodule directory can sometimes interfere with hard resets.
Git Installation Issues: Git not installed correctly or not in the system’s PATH.
Main Repository State: Issues within the main project’s .git directory or a detached HEAD state.

Review any error messages carefully. The manual commands provide a way to isolate and troubleshoot issues step-by-step.

Key Considerations

Avoid Direct Modification: Never directly edit files within the directories that contain your submodules in the main project. All changes to the source documentation must be made and committed in the source documentation repositories.
Main Project for Integration/New Content: The main documentation project is where you add content specific to the combined documentation set, structure the output, and integrate the submodule content.

The Impact of Large Submodules

Be mindful that the size of your source documentation repositories impacts clone and update times for the corresponding submodules. Very large submodules can make cloning and daily updates take longer. They can also potentially affect the performance of your documentation tool when loading the combined project. If source repositories are excessively large, consider strategies for optimizing their content or structure independently to improve overall efficiency.

Using Git submodules provides a powerful, version-control-native solution for integrating documentation from multiple independent sources, offering better scalability and reliability for large projects and collaborative teams compared to methods like Global Project Linking in such specific scenarios.