Incremental Backup Issue With MinIO

by ADMIN 36 views

Introduction

In this article, we will delve into the issue of incremental backup with MinIO as the S3 storage. We will explore the problem, its causes, and provide a step-by-step guide to resolve the issue. The goal is to ensure that incremental backups are successfully created and stored in the remote MinIO bucket.

Environment and Setup

Our environment consists of MinIO as the S3 storage and ClickHouse Backup as the backup tool. The versions of MinIO and ClickHouse Backup are as follows:

  • MinIO: RELEASE.2025-02-28T09-55-16Z
  • ClickHouse Backup: 2.6.5

Full Backup Creation

To create a full backup, we use the following command:

clickhouse-backup create_remote full_backup

This command creates a full backup and stores it in the remote MinIO bucket.

Local Full Backup Deletion

After creating the full backup, we delete the local full backup using the following command:

clickhouse-backup delete local full_backup

This command deletes the local full backup, leaving only the remote full backup in the MinIO bucket.

Incremental Backup Creation

To create an incremental backup, we use the following command:

clickhouse-backup create_remote --diff-from-remote=full_backup incremental_backup1

This command creates an incremental backup based on the full backup stored in the remote MinIO bucket.

Error Message

However, when running the incremental backup command, we encounter the following error message:

ERR pkg/backup/watch.go:157 > create_remote --diff-from-remote=full_backup return error: b.getTablesDiffFromRemote return error: backup-{shard}-full-20250312044255 not found on remote storage

This error message indicates that the full backup cannot be found in the remote MinIO bucket, preventing the creation of the incremental backup.

Causes of the Issue

There are several possible causes for this issue:

  • Incorrect Bucket Name: The bucket name specified in the --diff-from-remote option may be incorrect, leading to the full backup not being found in the remote MinIO bucket.
  • Full Backup Not Stored: The full backup may not have been stored in the remote MinIO bucket, either due to a previous error or a misconfiguration.
  • MinIO Bucket Configuration: The MinIO bucket configuration may be incorrect, preventing the full backup from being stored or retrieved.

Troubleshooting Steps

To resolve the issue, we can follow these troubleshooting steps:

Step 1: Verify Bucket Name

Verify that the bucket name specified in the --diff-from-remote option is correct. Check the MinIO bucket configuration to ensure that the bucket name matches the one specified in the option.

Step 2: Check Full Backup Storage

Check if the full backup has been stored in the remote MinIO bucket. Use the MinIO console or CLI to verify that the full backup is present in the bucket.

Step 3: Review MinIO Bucket Configuration

Review the MinIO bucket configuration to ensure that it is correct. Check the bucket policies, access controls, and storage settings to ensure that they are not preventing the full backup from being stored or retrieved.

Step 4: Run Full Backup Again

Run the full backup command again to recreate the full backup in the remote MinIO bucket.

Step 5: Run Incremental Backup Again

Run the incremental backup command again, using the correct bucket name and full backup name.

Resolution

By following these troubleshooting steps, we can resolve the issue of incremental backup with MinIO. The key is to verify the bucket name, check the full backup storage, review the MinIO bucket configuration, and recreate the full backup if necessary.

Conclusion

In this article, we have explored the issue of incremental backup with MinIO and provided a step-by-step guide to resolve the issue. By following these troubleshooting steps, we can ensure that incremental backups are successfully created and stored in the remote MinIO bucket.

Best Practices

To avoid this issue in the future, follow these best practices:

  • Verify Bucket Name: Always verify the bucket name specified in the --diff-from-remote option to ensure that it matches the one in the MinIO bucket configuration.
  • Check Full Backup Storage: Regularly check if the full backup has been stored in the remote MinIO bucket to ensure that it is present.
  • Review MinIO Bucket Configuration: Regularly review the MinIO bucket configuration to ensure that it is correct and not preventing the full backup from being stored or retrieved.

Introduction

In our previous article, we explored the issue of incremental backup with MinIO and provided a step-by-step guide to resolve the issue. In this article, we will answer some frequently asked questions (FAQs) related to incremental backup with MinIO.

Q: What is the cause of the "not found on remote storage" error?

A: The "not found on remote storage" error is caused by the full backup not being stored in the remote MinIO bucket. This can be due to a previous error, misconfiguration, or incorrect bucket name.

Q: How do I verify the bucket name in MinIO?

A: To verify the bucket name in MinIO, you can use the MinIO console or CLI. Log in to the MinIO console and navigate to the bucket settings. Check the bucket name to ensure it matches the one specified in the --diff-from-remote option.

Q: What are the best practices for incremental backup with MinIO?

A: The best practices for incremental backup with MinIO include:

  • Verify Bucket Name: Always verify the bucket name specified in the --diff-from-remote option to ensure that it matches the one in the MinIO bucket configuration.
  • Check Full Backup Storage: Regularly check if the full backup has been stored in the remote MinIO bucket to ensure that it is present.
  • Review MinIO Bucket Configuration: Regularly review the MinIO bucket configuration to ensure that it is correct and not preventing the full backup from being stored or retrieved.

Q: How do I resolve the issue if the full backup is not stored in the remote MinIO bucket?

A: To resolve the issue if the full backup is not stored in the remote MinIO bucket, follow these steps:

  1. Verify Bucket Name: Verify that the bucket name specified in the --diff-from-remote option is correct.
  2. Check Full Backup Storage: Check if the full backup has been stored in the remote MinIO bucket.
  3. Review MinIO Bucket Configuration: Review the MinIO bucket configuration to ensure that it is correct and not preventing the full backup from being stored or retrieved.
  4. Run Full Backup Again: Run the full backup command again to recreate the full backup in the remote MinIO bucket.
  5. Run Incremental Backup Again: Run the incremental backup command again, using the correct bucket name and full backup name.

Q: What are the common mistakes to avoid when using incremental backup with MinIO?

A: The common mistakes to avoid when using incremental backup with MinIO include:

  • Incorrect Bucket Name: Using an incorrect bucket name in the --diff-from-remote option.
  • Full Backup Not Stored: Not storing the full backup in the remote MinIO bucket.
  • MinIO Bucket Configuration: Incorrect MinIO bucket configuration preventing the full backup from being stored or retrieved.

Q: How do I ensure that my incremental backups are successful with MinIO?

A: To ensure that your incremental backups are successful with MinIO, follow these best practices:

  • Verify Bucket Name: Always verify the bucket name specified in the --diff-from-remote option to ensure that it matches the one in the MinIO bucket configuration.
  • Check Full Backup Storage: Regularly check if the full backup has been stored in the remote MinIO bucket to ensure that it is present.
  • Review MinIO Bucket Configuration: Regularly review the MinIO bucket configuration to ensure that it is correct and not preventing the full backup from being stored or retrieved.

Conclusion

In this article, we have answered some frequently asked questions (FAQs) related to incremental backup with MinIO. By following the best practices and avoiding common mistakes, you can ensure that your incremental backups are successful with MinIO.