Automatically Syncing Files with rsync

Automatically synchronize files between your local development environment and Workspace Manager workspaces using rsync.

Overview

rsync provides efficient file synchronization, allowing you to work with local files while keeping them in sync with your remote workspace. This is ideal for developers who prefer local editing with remote execution.

Prerequisites

• rsync installed on your local machine
• SSH access configured (see SSH documentation)
• Workspace running and accessible

Basic Usage

Once your SSH configuration is set up, you can use rsync to synchronize files between your local machine and workspace.

Push Local Changes to Workspace

rsync -avz --progress ~/local/project/ wsm.my_workspace:~/project/

Pull Workspace Changes to Local

rsync -avz --progress wsm.my_workspace:~/project/ ~/local/project/

Common rsync Options

• -a - Archive mode (preserves permissions, timestamps, symbolic links)
• -v - Verbose output
• -z - Compress data during transfer
• --progress - Show progress during transfer
• --delete - Delete files in destination that don’t exist in source
• --dry-run - Preview changes without actually syncing

Sync Strategies

One-Way Sync (Local to Remote)

Push your local changes to the workspace. This is useful when you edit locally and want to test/run code remotely.

rsync -avz --delete --progress \
  --exclude '.git' \
  --exclude 'node_modules' \
  ~/local/project/ wsm.my_workspace:~/project/

The --delete flag ensures files deleted locally are also removed from the workspace.

One-Way Sync (Remote to Local)

Pull workspace changes to your local machine. Useful for retrieving generated files or changes made directly on the workspace.

rsync -avz --progress \
  --exclude '.git' \
  --exclude 'node_modules' \
  wsm.my_workspace:~/project/ ~/local/project/

Data Loss Warning

Be careful with --delete when pulling from remote. It will remove local files that don’t exist on the workspace.

Automatic Synchronization

Using fswatch (macOS)

Install fswatch and create a sync script:

brew install fswatch

Create sync-to-workspace.sh:

#!/bin/bash
PROJECT_DIR="$HOME/local/project"
WORKSPACE="wsm.my_workspace:~/project/"

fswatch -o "$PROJECT_DIR" | while read f; do
  echo "Changes detected, syncing..."
  rsync -avz --delete \
    --exclude '.git' \
    --exclude 'node_modules' \
    "$PROJECT_DIR/" "$WORKSPACE"
done

Make it executable and run:

chmod +x sync-to-workspace.sh
./sync-to-workspace.sh

Using inotifywait (Linux)

Install inotify-tools:

# Ubuntu/Debian
sudo apt-get install inotify-tools

# Fedora/RHEL
sudo dnf install inotify-tools

Create a sync script:

#!/bin/bash
PROJECT_DIR="$HOME/local/project"
WORKSPACE="wsm.my_workspace:~/project/"

while inotifywait -r -e modify,create,delete,move "$PROJECT_DIR"; do
  echo "Changes detected, syncing..."
  rsync -avz --delete \
    --exclude '.git' \
    --exclude 'node_modules' \
    "$PROJECT_DIR/" "$WORKSPACE"
done

Using WSL (Windows)

For Windows users, we recommend using Windows Subsystem for Linux (WSL) for the best rsync experience:

# Install WSL (PowerShell as Administrator)
wsl --install

# Inside WSL, install rsync
sudo apt-get update
sudo apt-get install rsync

# Use the same Linux scripts above
# Access Windows files via /mnt/c/Users/YourName/project

You can then use the inotifywait script above, adjusting paths to access your Windows files:

#!/bin/bash
PROJECT_DIR="/mnt/c/Users/YourName/project"
WORKSPACE="wsm.my_workspace:~/project/"

while inotifywait -r -e modify,create,delete,move "$PROJECT_DIR"; do
  echo "Changes detected, syncing..."
  rsync -avz --delete \
    --exclude '.git' \
    --exclude 'node_modules' \
    "$PROJECT_DIR/" "$WORKSPACE"
done

Windows Native Alternatives

While WSL is recommended, Windows users can also consider:

• Mutagen - Has native Windows support with excellent performance
• WinSCP - GUI-based sync tool with scripting capabilities
• cwRsync - Windows port of rsync (less maintained)

Bidirectional Sync with Unison

For true two-way synchronization, consider using Unison:

# Install Unison
brew install unison  # macOS
sudo apt-get install unison  # Ubuntu/Debian

# Create sync profile
unison ~/local/project ssh://wsm.my_workspace//home/web/project \
  -ignore 'Name .git' \
  -ignore 'Name node_modules' \
  -auto -batch

Excluding Files

Using —exclude Flags

Exclude specific files or directories:

rsync -avz \
  --exclude '.git' \
  --exclude 'node_modules' \
  --exclude '*.log' \
  --exclude '.env' \
  ~/local/project/ wsm.my_workspace:~/project/

Using —exclude-from File

Create an exclude file (similar to .gitignore):

.rsyncignore

.git
node_modules
*.log
.env
.DS_Store
dist/
build/
*.pyc
__pycache__/

Use it with rsync:

rsync -avz --exclude-from='.rsyncignore' \
  ~/local/project/ wsm.my_workspace:~/project/

Using Existing .gitignore

You can leverage your existing .gitignore:

rsync -avz \
  --filter=':- .gitignore' \
  ~/local/project/ wsm.my_workspace:~/project/

Performance Optimization

Use Compression Wisely

For fast networks, compression overhead may slow things down:

# Skip compression on fast networks
rsync -av ~/local/project/ wsm.my_workspace:~/project/

Limit Bandwidth

Prevent rsync from saturating your connection:

# Limit to 1000 KB/s
rsync -avz --bwlimit=1000 \
  ~/local/project/ wsm.my_workspace:~/project/

Incremental Transfers

rsync is already incremental by default, but you can optimize further:

# Use checksum instead of timestamp for change detection
rsync -avzc ~/local/project/ wsm.my_workspace:~/project/

Parallel Transfers

For many small files, consider using parallel rsync:

# Install parallel
brew install parallel  # macOS
sudo apt-get install parallel  # Ubuntu/Debian

# Parallel sync
find ~/local/project -type f | parallel -j 4 \
  rsync -avz {} wsm.my_workspace:~/project/{}

Troubleshooting

Permission Denied

Ensure you have write permissions on the workspace:

# Check permissions
ssh wsm.my_workspace "ls -la ~/project"

# Fix permissions if needed
ssh wsm.my_workspace "chmod -R u+w ~/project"

Connection Issues

If rsync fails to connect:

1. Verify SSH access works: ssh wsm.my_workspace
2. Update SSH config: wsm-cli ssh config-update
3. Check workspace is running in WSM admin

Slow Transfers

If syncing is slow:

1. Use --progress to identify bottlenecks
2. Exclude large binary files or build artifacts
3. Consider using --compress-level=1 for faster compression
4. Check network connectivity

Files Not Syncing

If changes aren’t syncing:

1. Verify paths are correct (note trailing slashes matter!)
2. Check exclude patterns aren’t too broad
3. Use --dry-run to preview what would sync
4. Add -vv for verbose debugging output

Alternative Tools

Mutagen

Mutagen provides real-time, bidirectional synchronization with better performance for large projects:

# Install Mutagen
brew install mutagen-io/mutagen/mutagen

# Create sync session
mutagen sync create ~/local/project wsm.my_workspace:~/project \
  --ignore=.git --ignore=node_modules

# List sessions
mutagen sync list

# Monitor session
mutagen sync monitor

Unison

Unison offers true bidirectional sync with conflict detection:

unison ~/local/project ssh://wsm.my_workspace//home/web/project \
  -ignore 'Name .git' \
  -ignore 'Name node_modules'

SSHFS

Mount the remote workspace as a local filesystem:

# Install SSHFS
brew install macfuse sshfs  # macOS
sudo apt-get install sshfs  # Ubuntu/Debian

# Mount workspace
mkdir ~/workspace-mount
sshfs wsm.my_workspace:~/project ~/workspace-mount

# Unmount when done
umount ~/workspace-mount

Related Documentation

• SSH Access - Configure SSH access to workspaces
• Connecting JetBrains IDEs - Use JetBrains remote development
• Connecting Visual Studio Code - Use VS Code Remote-SSH