Args passed to the container

Oct 13, 2025 by sorusgentrification.com 25 views

Issue: Product-name information missing for MI300 GPUs in AKS

Problem Description

We are using AMD MI300X GPUs in AKS. We use the node-labeller Daemonset to discover the product-name however on this amdgpu-labeller.

Args passed to the container

    - -vram
    - -cu-count
    - -simd-count
    - -device-id
    - -family
    - -product-name
    - -driver-version
    - -driver-src-version
    - -firmware

SKU of the node Standard_ND96isr_MI300X_v5

Label added by AMD on the node.

    amd.com/gpu.cu-count: "304"
    amd.com/gpu.device-id: 74b5
    amd.com/gpu.driver-src-version: 0A82C3819E55873715679EA
    amd.com/gpu.driver-version: 6.8.5
    amd.com/gpu.family: AI
    amd.com/gpu.simd-count: "1216"
    amd.com/gpu.vram: 191G
    beta.amd.com/gpu.cu-count: "304"
    beta.amd.com/gpu.cu-count.304: "8"
    beta.amd.com/gpu.device-id: 74b5
    beta.amd.com/gpu.device-id.74b5: "8"
    beta.amd.com/gpu.family: AI
    beta.amd.com/gpu.family.AI: "8"
    beta.amd.com/gpu.firmware.CE.feat.0: "8"
    beta.amd.com/gpu.firmware.CE.fw.0: "8"
    beta.amd.com/gpu.firmware.MC.feat.0: "8"
    beta.amd.com/gpu.firmware.MC.fw.0: "8"
    beta.amd.com/gpu.firmware.ME.feat.0: "8"
    beta.amd.com/gpu.firmware.ME.fw.0: "8"
    beta.amd.com/gpu.firmware.MEC.feat.49: "8"
    beta.amd.com/gpu.firmware.MEC.fw.160: "8"
    beta.amd.com/gpu.firmware.PFP.feat.0: "8"
    beta.amd.com/gpu.firmware.PFP.fw.0: "8"
    beta.amd.com/gpu.firmware.RLC.feat.1: "8"
    beta.amd.com/gpu.firmware.RLC.fw.64: "8"
    beta.amd.com/gpu.firmware.SDMA0.feat.44: "8"
    beta.amd.com/gpu.firmware.SDMA0.fw.21: "8"
    beta.amd.com/gpu.firmware.SMC.feat.0: "8"
    beta.amd.com/gpu.firmware.SMC.fw.5600514: "8"
    beta.amd.com/gpu.firmware.UVD.feat.0: "8"
    beta.amd.com/gpu.firmware.UVD.fw.0: "8"
    beta.amd.com/gpu.firmware.VCE.feat.0: "8"
    beta.amd.com/gpu.firmware.VCE.fw.0: "8"
    beta.amd.com/gpu.simd-count: "1216"
    beta.amd.com/gpu.simd-count.1216: "8"
    beta.amd.com/gpu.vram: 191G
    beta.amd.com/gpu.vram.191G: "8"

This is from a different SKU in azure Standard_nd96asr_mi200_v4

    amd.com/gpu.cu-count: "110"
    amd.com/gpu.device-id: 740c
    amd.com/gpu.driver-src-version: 0A82C3819E55873715679EA
    amd.com/gpu.driver-version: 6.8.5
    amd.com/gpu.family: AI
    amd.com/gpu.product-name: AMD_INSTINCT_MI250X_MCM_OAM_AC_MBA_MSFT
    amd.com/gpu.simd-count: "440"
    amd.com/gpu.vram: 64G
    beta.amd.com/gpu.cu-count: "110"
    beta.amd.com/gpu.cu-count.110: "16"
    beta.amd.com/gpu.device-id: 740c
    beta.amd.com/gpu.device-id.740c: "16"
    beta.amd.com/gpu.family: AI
    beta.amd.com/gpu.family.AI: "16"
    beta.amd.com/gpu.firmware.CE.feat.0: "16"
    beta.amd.com/gpu.firmware.CE.fw.0: "16"
    beta.amd.com/gpu.firmware.MC.feat.0: "16"
    beta.amd.com/gpu.firmware.MC.fw.0: "16"
    beta.amd.com/gpu.firmware.ME.feat.0: "16"
    beta.amd.com/gpu.firmware.ME.fw.0: "16"
    beta.amd.com/gpu.firmware.MEC.feat.46: "16"
    beta.amd.com/gpu.firmware.MEC.fw.83: "16"
    beta.amd.com/gpu.firmware.PFP.feat.0: "16"
    beta.amd.com/gpu.firmware.PFP.fw.0: "16"
    beta.amd.com/gpu.firmware.RLC.feat.1: "16"
    beta.amd.com/gpu.firmware.RLC.fw.17: "16"
    beta.amd.com/gpu.firmware.SDMA0.feat.44: "16"
    beta.amd.com/gpu.firmware.SDMA0.fw.8: "16"
    beta.amd.com/gpu.firmware.SMC.feat.0: "16"
    beta.amd.com/gpu.firmware.SMC.fw.4471808: "16"
    beta.amd.com/gpu.firmware.UVD.feat.0: "16"
    beta.amd.com/gpu.firmware.UVD.fw.0: "16"
    beta.amd.com/gpu.firmware.VCE.feat.0: "16"
    beta.amd.com/gpu.firmware.VCE.fw.0: "16"
    beta.amd.com/gpu.product-name: AMD_INSTINCT_MI250X_MCM_OAM_AC_MBA_MSFT
    beta.amd.com/gpu.product-name.AMD_INSTINCT_MI250X_MCM_OAM_AC_MBA_MSFT: "16"
    beta.amd.com/gpu.simd-count: "440"
    beta.amd.com/gpu.simd-count.440: "16"
    beta.amd.com/gpu.vram: 64G
    beta.amd.com/gpu.vram.64G: "16"

When I use rocm-smi --showproductname it works on both GPUs (MI200 and MI300x).

Output on MI300.

GPU[0]          : Card Series:          Aqua Vanjaram [Instinct MI300X VF]
GPU[0]          : Card Model:           0x74b5
GPU[0]          : Card Vendor:          Advanced Micro Devices, Inc. [AMD/ATI]
GPU[0]          : Card SKU:             M3000100
GPU[0]          : Subsystem ID:         0x74a1
GPU[0]          : Device Rev:           0x00
GPU[0]          : Node ID:              2
GPU[0]          : GUID:                 65402
GPU[0]          : GFX Version:          gfx942

On MI200

GPU[0]          : Card Series:          AMD INSTINCT MI250X (MCM) OAM AC MBA MSFT
GPU[0]          : Card Model:           0x740c
GPU[0]          : Card Vendor:          Advanced Micro Devices, Inc. [AMD/ATI]
GPU[0]          : Card SKU:             D65205
GPU[0]          : Subsystem ID:         0x0b0c
GPU[0]          : Device Rev:           0x01
GPU[0]          : Node ID:              4
GPU[0]          : GUID:                 10595
GPU[0]          : GFX Version:          gfx9010

The expectation is that product-name should be available on all GPUs, both MI200 and MI300

Operating System

NAME="Ubuntu" VERSION="22.04.5 LTS (Jammy Jellyfish)"

CPU

model name : Intel(R) Xeon(R) Platinum 8480C

GPU

Name: Intel(R) Xeon(R) Platinum 8480C Marketing Name: Intel(R) Xeon(R) Platinum 8480C Name: Intel(R) Xeon(R) Platinum 8480C Marketing Name: Intel(R) Xeon(R) Platinum 8480C Name: gfx942 Marketing Name: AMD Instinct MI300X VF Name: amdgcn-amd-amdhsa--gfx942:srame
Q&A: Product-name information missing for MI300 GPUs in AKS

Q: What is the issue with the product-name information for MI300 GPUs in AKS?

A: The product-name information is missing for MI300 GPUs in AKS, which is a problem because it is expected to be available on all GPUs, including MI200 and MI300.

Q: What is the expected behavior for the product-name information?

A: The expected behavior is that the product-name information should be available on all GPUs, including MI200 and MI300.

Q: What is the current behavior for the product-name information?

A: The current behavior is that the product-name information is missing for MI300 GPUs in AKS.

Q: What are the possible causes of this issue?

A: The possible causes of this issue are:

The node-labeller Daemonset is not configured correctly to discover the product-name information.
The args passed to the container are not correct.
The SKU of the node is not correct.
The label added by AMD on the node is not correct.

Q: How can I troubleshoot this issue?

A: To troubleshoot this issue, you can:

Check the node-labeller Daemonset configuration to ensure it is correct.
Verify the args passed to the container are correct.
Check the SKU of the node to ensure it is correct.
Verify the label added by AMD on the node is correct.

Q: What are the steps to reproduce this issue?

A: The steps to reproduce this issue are:

Create a node with a MI300 GPU in AKS.
Run the node-labeller Daemonset to discover the product-name information.
Verify that the product-name information is missing.

Q: What is the expected output for the product-name information?

A: The expected output for the product-name information is:

Card Series: Aqua Vanjaram [Instinct MI300X VF]
Card Model: 0x74b5
Card Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Card SKU: M3000100
Subsystem ID: 0x74a1
Device Rev: 0x00
Node ID: 2
GUID: 65402
GFX Version: gfx942

Q: What is the actual output for the product-name information?

A: The actual output for the product-name information is:

Card Series: AMD INSTINCT MI250X (MCM) OAM AC MBA MSFT
Card Model: 0x740c
Card Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Card SKU: D65205
Subsystem ID: 0x0b0c
Device Rev: 0x01
Node ID: 4
GUID: 10595
GFX Version: gfx9010

Q: What is the difference between the expected and actual output?

A: The difference between the expected and actual output is that the expected output shows the product-name information for the MI300 GPU, while the actual output shows the product-name information for the MI200 GPU.

Q: How can I resolve this issue?

A: To resolve this issue, you can:

Update the node-labeller Daemonset configuration to include the correct args for the product-name information.
Verify the SKU of the node to ensure it is correct.
Verify the label added by AMD on the node is correct.

Q: What are the benefits of resolving this issue?

A: The benefits of resolving this issue are:

The product-name information will be available for all GPUs, including MI200 and MI300.
The node-labeller Daemonset will be able to discover the product-name information correctly.
The SKU of the node will be verified to ensure it is correct.
The label added by AMD on the node will be verified to ensure it is correct.