Endpoint_url Global Option From AWS Config Is Not Working

by ADMIN 58 views

Introduction

AWS Config is a powerful tool that allows you to manage your AWS resources and services. However, sometimes you may encounter issues with the endpoint_url global option. In this article, we will explore a bug with AWS Config and endpoint_url option, and provide a solution to fix it.

The Bug

The bug occurs when you set the endpoint_url option with DNS records and port in the ~/.aws/config file. For example:

[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4

You will see buckets, but cannot open them and get an error. The error log will show:

2025-03-13 12:44:34.360836 +03:00 Failed to load objects: ServiceError(ServiceError { source: Unhandled(Unhandled { source: XmlDecodeError { kind: Custom("encountered invalid XML root: expected ListBucketResult but got StartEl { name: Name { prefix: \"\", local: \"ListAllMyBucketsResult\" }, attributes: [Attr { name: Name { prefix: \"\", local: \"xmlns\" }, value: \"http://s3.amazonaws.com/doc/2006-03-01/\" }], closed: false, depth: 0 }. This is likely a bug in the SDK.") }, meta: ErrorMetadata { code: None, message: None, extras: None } }), raw: Response { status: StatusCode(200), headers: Headers { headers: {"accept-ranges": HeaderValue { _private: H0("bytes") }, "content-length": HeaderValue { _private: H0("465") }, "content-type": HeaderValue { _private: H0("application/xml") }, "server": HeaderValue { _private: H0("MinIO") }, "strict-transport-security": HeaderValue { _private: H0("max-age=31536000; includeSubDomains") }, "vary": HeaderValue { _private: H0("Origin") }, "vary": HeaderValue { _private: H0("Accept-Encoding") }, "x-amz-id-2": HeaderValue { _private: H0("dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8") }, "x-amz-request-id": HeaderValue { _private: H0("182C53996FBE08C5") }, "x-content-type-options": HeaderValue { _private: H0("nosniff") }, "x-ratelimit-limit": HeaderValue { _private: H0("1045") }, "x-ratelimit-remaining": HeaderValue { _private: H0("1045") }, "x-xss-protection": HeaderValue { _private: H0("1; mode=block") }, "date": HeaderValue { _private: H0("Thu, 13 Mar 2025 09:44:34 GMT") }} }, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListAllMyBucketsResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName>minio</DisplayName></Owner><Buckets><Bucket><Name>test-bucket</Name><CreationDate>2025-03-12T11:33:23.176Z</CreationDate></Bucket><Bucket><Name>test-bucket-2</Name><CreationDate>2025-03-12T11:35:32.468Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>")), retryable: true }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } })

The error message indicates that the endpoint_url option is ignored and the SDK is using the public AWS instance s3.amazonaws.com.

How to Reproduce

To reproduce the bug, you can follow these steps:

  1. Create a docker-compose.yaml file to setup MinIO:
---
version: "3.8"

services:
  minio:
    image: minio/minio
    container_name: minio
    restart: unless-stopped
    ports:
      - "9000:9000" 
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
      MINIO_DEFAULT_BUCKETS: test-bucket
    command: server /data --console-address ":9001"
    volumes:
      - minio_data:/data

  create-bucket:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      sleep 2;
      /usr/bin/mc alias set local http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc mb local/test-bucket 2>/dev/null;
      /usr/bin/mc mb local/test-bucket-2 2>/dev/null;
      echo 'Test file' > /tmp/test.txt;
      /usr/bin/mc cp /tmp/test.txt local/test-bucket/test.txt;
      /usr/bin/mc cp /tmp/test.txt local/test-bucket-2/test.txt;
      exit 0;
      "

volumes:
  minio_data:
  1. Run docker-compose up to start the MinIO service.
  2. Create an ~/.aws/config file with the following content:
[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4
  1. Create an ~/.aws/credentials file with the following content:
[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin
  1. Run stu with the --endpoint-url argument:
stu --endpoint-url http://localhost:9000

Solution

To fix the bug, you can use the --endpoint-url argument when running stu. This will override the endpoint_url option in the ~/.aws/config file.

Alternatively, you can modify the ~/.aws/config file to use the --endpoint-url argument:

[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4
  endpoint_url = http://localhost:9000

This will set the endpoint_url option to the value specified in the --endpoint-url argument.

Conclusion

Q: What is the endpoint_url global option in AWS Config?

A: The endpoint_url global option in AWS Config is used to specify the URL of the AWS service endpoint. This option is used to override the default endpoint URL for a specific service.

Q: Why is the endpoint_url option not working?

A: The endpoint_url option is not working because the SDK is ignoring the specified endpoint URL and using the public AWS instance instead.

Q: How can I reproduce the bug?

A: To reproduce the bug, you can follow these steps:

  1. Create a docker-compose.yaml file to setup MinIO.
  2. Run docker-compose up to start the MinIO service.
  3. Create an ~/.aws/config file with the following content:
[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4
  1. Create an ~/.aws/credentials file with the following content:
[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin
  1. Run stu without the --endpoint-url argument.

Q: How can I fix the bug?

A: To fix the bug, you can use the --endpoint-url argument when running stu. This will override the endpoint_url option in the ~/.aws/config file.

Alternatively, you can modify the ~/.aws/config file to use the --endpoint-url argument:

[default]
region = us-east-1
output = json
endpoint_url = http://localhost:9000

s3 =
  signature_version = s3v4
  endpoint_url = http://localhost:9000

This will set the endpoint_url option to the value specified in the --endpoint-url argument.

Q: What are the benefits of using the endpoint_url option?

A: The endpoint_url option provides several benefits, including:

  • Flexibility: The endpoint_url option allows you to specify a custom endpoint URL for a specific service.
  • Security: The endpoint_url option enables you to use a secure endpoint URL for a specific service.
  • Performance: The endpoint_url option can improve performance by allowing you to use a faster endpoint URL for a specific service.

Q: Are there any limitations to using the endpoint_url option?

A: Yes, there are several limitations to using the endpoint_url option, including:

  • Compatibility: The endpoint_url option may not be compatible with all AWS services.
  • Security: The endpoint_url option may not provide the same level of security as using a secure endpoint URL.
  • Performance: The endpoint_url option may not improve performance in all cases.

Q: How can I troubleshoot issues with the endpoint_url option?

A: To troubleshoot issues with the endpoint_url option, you can follow these steps:

  1. Check the ~/.aws/config file to ensure that the endpoint_url option is correctly specified.
  2. Check the ~/.aws/credentials file to ensure that the credentials are correctly specified.
  3. Run stu with the --endpoint-url argument to test the endpoint URL.
  4. Check the AWS service logs to ensure that the endpoint URL is being used correctly.

By following these steps, you can troubleshoot issues with the endpoint_url option and ensure that it is working correctly.