How to efficiently copy files between compute servers using ssh and rsync

What You'll Learn:
Gain practical knowledge on securely copying files between compute servers using SSH and rsync within CoCalc. This tutorial demonstrates setting up SSH keys, configuring server access, and using rsync for data transfer—including handling large directories and incremental updates.

Discover More:
📄 CoCalc Documentation: https://doc.cocalc.com/index.html
🌐 CoCalc Website: https://cocalc.com/
🛍 CoCalc Store: https://cocalc.com/store

Stay Connected:
Facebook: https://www.facebook.com/CoCalcOnline
Twitter: https://x.com/cocalc_com
LinkedIn: https://www.linkedin.com/company/sagemath-inc./

#cocalc #ssh #rsync #datatransfer #linux

Transcription:

Hello everyone, in this video I’ll show you how to copy files between two compute servers using SSH and rsync in CoCalc.

1. Create two compute servers in your CoCalc project. For example, one server could be in Europe and the other in Japan.
2. Open a terminal for each server. To transfer files between them, you first need to set up SSH keys for passwordless authentication.
3. On your source server, generate a new SSH key:

```bash
ssh-keygen -t ed25519
```

4. Copy the contents of `/home/user/.ssh/id_ed25519.pub` and add it under SSH Keys in your project settings.
5. After adding the key, you can SSH between your compute servers. Test this by connecting from one server to the other using their names or IP addresses.
6. Add your data to a directory, such as `scratch/`, on the first server. For example, clone a repository to populate the directory.
7. Use rsync to copy files from one server to another. The following command synchronizes the contents, preserving permissions and handling interruptions:

```bash
rsync -axvH scratch/ compute-server-n:scratch/
```

This command uses archive mode (`-a`), excludes crossing file system boundaries (`-x`), shows progress (`-v`), and preserves hard links (`-H`). If a file changes, only the differences are copied during subsequent transfers, making it faster and saving bandwidth.

8. To synchronize incremental changes, simply modify a file and rerun the rsync command—the updated file alone is transferred.

If you need to transfer files between servers in different projects, add the SSH key to each project and use the server's IP address instead of its name.

This tutorial covers copying files securely and incrementally between compute servers in CoCalc. For more content, please like, subscribe, and follow. Thanks for watching!

Видео How to efficiently copy files between compute servers using ssh and rsync канала CoCalc by SageMath, Inc.