background image

DU-10189-001 _v5.0.2 

   |   

 March   2021

DGX Station A100

User Guide

Summary of Contents for DGX Station A100

Page 1: ...DU 10189 001 _v5 0 2 March 2021 DGX Station A100 User Guide ...

Page 2: ... Using the BMC 17 3 1 Understanding the BMC Controls 18 3 2 Configuring a Static IP Address for the BMC 19 3 3 Configuring a BMC Static IP Address Using ipmitool 19 3 3 1 Configuring a BMC Static IP Address Using the System BIOS 20 3 4 Logging into the BMC 20 3 5 Changing Your Default BMC Password 21 3 6 Logging in After Entering an Incorrect Password 22 3 7 Common BMC Tasks 22 3 7 1 Configuring t...

Page 3: ...omputing 35 5 9 1 Enabling the TPM 35 5 9 2 Clearing the TPM 35 5 10 Changing Disk Passwords Adding Disks or Replacing Disks 36 5 11 Recovering a Lost Key 36 Chapter 6 Unpacking and Repacking the DGX Station A100 37 6 1 Unpacking the DGX Station A100 37 6 2 Repacking the DGX Station A100 for Shipment 40 Appendix A Security 44 Appendix B Safety 45 B 1 Intended Application Uses 45 B 2 General Precau...

Page 4: ... 58 D 12 South Korea 59 D 13 Taiwan 59 D 14 United Kingdom 60 D 15 United States 61 D 16 United States Canada 62 Appendix E DGX Station A100 Hardware Specifications 63 E 1 Environmental Conditions 63 E 2 Component Specifications 63 E 3 Mechanical Specifications 64 E 4 Power Specifications 64 Appendix F Customer Support for the NVIDIA DGX Station A100 65 ...

Page 5: ...DGX Station A100 DU 10189 001 _v5 0 2 v List of Tables Table 1 BMC Navigation Controls 18 Table 2 Fields to Generate an SSL Certificate 25 ...

Page 6: ...oftware for Red Hat Enterprise Linux or CentOS For additional information to help you use the DGX Station A100 see the following table Task Additional Information Use the Ubuntu Desktop Linux OS Ubuntu 20 04 Desktop Guide https help ubuntu com 20 04 ubuntu help index html Use the DGX Station A100 to download and run containers for deep learning frameworks NGC Container Registry for DGX User Guide ...

Page 7: ...in an easy to place workstation form factor It s the only system with four fully interconnected and Multi Instance GPU MIG capable NVIDIA A100 Tensor Core GPUs with up to 320 gigabytes GB of total GPU memory that can plug into a standard power outlet in the office or at home resulting in a powerful AI appliance that you can place anywhere 1 1 Registering Your DGX Station A100 To obtain support for...

Page 8: ...aration of Conformity Wheel locks Wheel locks are provided to prevent the unit from rolling out of place To install these locks from the side slide them under two of the wheels Inspect each piece of equipment in the packing box If anything is missing or damaged contact your supplier Before you get started you need to first unpack DGX Station A100 See Unpacking your DGX Station A100 for more inform...

Page 9: ...PU 1 Single AMD 7742 64 cores and 2 25 GHz base 3 4 GHz max boost GPU 4 NVIDIA A100 with 80 GB per GPU 320 GB total of GPU memory System Memory and Storage Component Qty Unit Capacity Total Capacity Description System memory 8 64 GB 512 GB System DDR4 RAM Data storage 1 7 68 TB 7 68 TB Cache Data U 2 NVME drive OS storage 1 1 92 TB 1 92 TB Boot M 2 NVME drive ...

Page 10: ...ower on the DGX Station A100 2 1 Connecting and Powering on the DGX Station A100 To complete this task you need the following items which are not supplied with the DGX Station A100 Mini DisplayPort 1 2 to DisplayPort USB keyboard USB mouse Ethernet cable 1 Connect a display to any DisplayPort connector and a keyboard and mouse to any two USB ports ...

Page 11: ... display to the DGX Station A100 After you complete the initial Ubuntu OS configuration you can configure the DGX Station A100 to use multiple displays Refer to the NVDIA DGX OS 5 User Guide for more information 2 Use any of the two Ethernet ports to connect the DGX Station A100 to your LAN with Internet connectivity ...

Page 12: ...for DHCP If both the ports are connected simultaneously each port will get its own IP address The IP address that the Linux operating system OS uses will then alternate between these addresses causing the OS and applications to malfunction 3 Make sure that the power supply rocker switch is in the OFF position 4 Connect the supplied power cable from the power socket at the back of the unit to an ap...

Page 13: ...e and do not use this power cable with any other products or for any other purpose Not all power cables have the same current ratings Do not use household extension cables with your product Household extension cables do not have overload protection and are not intended for use with computer systems 5 Connect the display to a suitable AC outlet and power on the display 6 Move the DGX Station A100 p...

Page 14: ...ton on the front of the unit to power on the DGX Station A100 2 2 Using DGX Station A100 as a Server Without a Monitor By default DGX Station A100 is shipped with the DP port automatically selected in the display To enter the SBIOS setup see Configuring a BMC Static IP Address Using the System BIOS ...

Page 15: ... BMC remote console will not show a display In this case return to the BIOS and complete the following steps Important If you do not change your SBIOS settings after the machine has booted in the Desktop GUI the BMC remote console will not show a display To change your SBIOS settings Tip The SBIOS screen will show up on any monitor that is connected to the DP port the VGA port or the BMC remote co...

Page 16: ...is because nvidia smi is including the DGX Display GPU that is used to drive the monitor and high quality graphics output When running an application or workload the DGX Display GPU can get in the way because it does not have direct NVlink connectivity sufficient memory or the performance characteristics of the NVIDIA A100 GPUs that are installed on the system As a result you should ensure that th...

Page 17: ...st lab ro dvt 058 80gb usr local cuda 11 2 samples 1_Utilities p2pBandwidthLatencyTest sudo make usr local cuda bin nvcc ccbin g I common inc m64 threads 0 gencode arch compute_35 code sm_35 gencode arch compute_37 code sm_37 gencode arch compute_50 code sm_50 gencode arch compute_52 code sm_52 gencode arch compute_60 code sm_60 gencode arch compute_61 code sm_61 gencode arch compute_70 code sm_70...

Page 18: ...ice 4 CANNOT Access Peer Device 0 Device 4 CANNOT Access Peer Device 1 Device 4 CANNOT Access Peer Device 2 Device 4 CANNOT Access Peer Device 3 NOTE In case a device doesn t have P2P access to other one it falls back to normal memcopy procedure So you can see lesser Bandwidth GB s and unstable Latency us in those cases P2P Connectivity Matrix D D 0 1 2 3 4 0 1 1 1 1 0 1 1 1 1 1 0 2 1 1 1 1 0 3 1 ...

Page 19: ...across all five GPUs including the DGX Display GPU The application also shows that there is no peer to peer connectivity between any GPU and GPU 4 This indicates that GPU 4 should not be used for high performance workloads Run the example one more time by using the CUDA_VISIBLE_DEVICES variable which limits the number of GPUs that the application can see Note All GPUs can communicate with all othe...

Page 20: ... Enabled Bandwidth Matrix GB s D D 0 1 2 3 0 1592 76 184 88 185 21 185 30 1 184 99 1589 52 185 19 185 32 2 185 28 185 30 1585 49 185 01 3 185 45 185 39 184 84 1587 91 P2P Disabled Latency Matrix us GPU 0 1 2 3 0 2 38 21 56 21 61 21 56 1 21 70 2 34 21 54 21 56 2 21 55 21 56 2 41 21 06 3 21 57 21 34 21 56 2 39 CPU 0 1 2 3 0 4 22 11 99 12 71 12 09 1 11 86 4 09 12 00 11 71 2 12 52 11 98 4 27 12 24 3 1...

Page 21: ... 0f2dff15 7c85 4320 da52 d3d54755d182 GPU 2 Graphics Device UUID GPU dc598de6 dd4d 2f43 549f f7b4847865a5 GPU 3 DGX Display UUID GPU 91b9d8c8 e2b9 6264 99e0 b47351964c52 GPU 4 Graphics Device UUID GPU e32263f2 ae07 f1db 37dc 17d1169b09bf In Docker enter the MIG UUID from this output in which GPU 0 and Device 0 have been selected If you are running on DGX Station A100 restart the nv docker gpus and...

Page 22: ...th Matrix GB s D D 0 0 190 77 Bidirectional P2P Enabled Bandwidth Matrix GB s D D 0 0 190 53 P2P Disabled Latency Matrix us GPU 0 0 3 57 CPU 0 0 4 07 P2P Enabled Latency P2P Writes Matrix us GPU 0 0 3 55 CPU 0 0 4 07 NOTE The CUDA Samples are not meant for performance measurements Results may vary when GPU Boost is enabled 2 4 Completing the Initial Ubuntu OS Configuration When you power on the DG...

Page 23: ...r Depending on the BIOS settings you can direct the OS X display to either adapter These settings can be adjusted to use one of the three modes listed before from the BIOS The nvidia conf xconfig service manages this function for the X Windowing system Here is a list of the display select modes Auto Automatically configures the X windowing system to use whichever compatible display adapter is curr...

Page 24: ...trols Control Description Dashboard Displays the overall information about the status of the device Sensor Provides status and readings for system sensors such as SSD PSUs voltages CPU temperatures DIMM temperatures and fan speeds System Inventory Displays inventory information of the following system modules System Processor Memory Controller BaseBoard Power Thermal PCIE Device PCIE Function and ...

Page 25: ...BMC web UI 3 2 Configuring a Static IP Address for the BMC Here is some information about how to configure a static IP address for the BMC 3 3 Configuring a BMC Static IP Address Using ipmitool Here is some information about how to set a static IP address for the BMC from the Ubuntu command line Note If you cannot access the DGX Station A100 remotely connect a display 1440x900 or lower resolution ...

Page 26: ...resolution to the DGX A100 System and power on the DGX Station A100 2 When you see the SBIOS version screen to enter the BIOS Setup Utility screen press Del or F2 3 In the BIOS Setup Utility screen on the Server Mgmt tab scroll to BMC Network Configuration and press Enter 4 Scroll to Configuration address source and press Enter 5 In the Configuration address source dialog box select Static and the...

Page 27: ...n 3 Click Sign in Here is an example of the BMC dashboard 3 5 Changing Your Default BMC Password When you complete one of the following tasks In the BMC web UI click Restore Factory Defaults and do not click Save to preserve the settings Run the sudo ipmitool raw 0x32 0x66 command the passwords for the admin and Administrator users revert to the following default passwords admin for the admin role...

Page 28: ...rrect password again the lockout period is extended The lockout period is calculated in the following way 10 minutes failure count 5 tolerances For example if you entered incorrect passwords 7 times you must wait for 20 minutes 10 7 5 before you can log in again Important The start of the current lockout period is calculated from the latest incorrect password that you entered and not from the firs...

Page 29: ...ove KVM and access the DGX A100 Station console Important If you select the Media Boost checkbox the processes that are related to media redirection will have a higher priority than other processes 1 In the left navigation click Remote Control 2 Click Launch KVM 3 7 3 Setting Up Active Directory or LDAP E Directory Here is some information about setting up Active Directory or LDAP E Directory in B...

Page 30: ... UnConfigured in the upper left corner of of the page To delete an event filter from the list click the x icon 3 7 5 Uploading or Generating SSL Certificates Here is some information about how you can upload or generate SSL certificates You can set up a new SSL certificate in one of the following ways Generate a self signed SSL Upload an SSL for example to use a Trusted CA signed certificate In th...

Page 31: ...d Maximum length of 64 alpha numeric characters The special characters and are not allowed Organization O The name of the organization for which the certificate is generated Maximum length of 64 alpha numeric characters The special characters and are not allowed Organization Unit OU Overall organization section unit name for which the certificate is generated Maximum length of 64 alpha numeric cha...

Page 32: ...ngth The key length bit value of the certificate for example 2048 bits 3 To generate the certificate click Save 3 7 5 3 Uploading the SSL Certificate In BMC you can upload your SSL certificate Make sure the certificate and key meet the following requirements SSL certificates and keys must both use the pem file extension Private keys must not be encrypted SSL certificates and keys must each be less...

Page 33: ...nvidia smi i GPU IDs mig 1 command 3 Select the GPUs by using comma separated GPU indexes PCI Bus Ids or UUIDs Here is some information to remember If you do not specify a GPU ID the MIG mode is applied to all the GPUs on the system sudo nvidia smi i 0 mig 1 Enabled MIG Mode for GPU 00000000 36 00 0 All done nvidia smi i 0 query gpu pci bus_id mig mode current format csv pci bus_id mig mode curren...

Page 34: ...t the system to make MIG mode effective All done 4 Stop the nvsm dcgm and gdm3 services enable MIG mode on the desired GPU and restore the monitoring services sudo systemctl stop nvsm sudo systemctl stop dcgm sudo systemctl stop gdm3 sudo nvidia smi i 0 mig 1 Enabled MIG Mode for GPU 00000000 07 00 0 All done The examples use super user privileges When you grant read access to mig config capabilit...

Page 35: ...rives are self encrypting drives Only SEDs used as data drives are supported The software will not manage SEDs that are OS drives The software provides the following functionality Identifies eligible drives on the system Allows you to assign authentication keys passwords for each SED as part of the initialization process The software can also generate random passwords for each drive The passwords ...

Page 36: ...ut how you can initialize your DGX system for drive encryption Note Before you initialize drive encryption see Configuring Trusted Computing and if necessary complete the configuration instructions Initialize the system for drive encryption using the nv disk encrypt command sudo nv disk encrypt init k your vault password f path to json file g r Here are the options k lets you create the vault pass...

Page 37: ...SON File This section provides information about a method that specifies the drive password mapping ahead of time By using the method in this example you can simultaneously initialize several drives at a time and after issuing the initialization command and avoid entering a password for each drive 5 5 1 1 Determining the Drives Can be Managed as Self Encrypting Review the storage layout of the DGX...

Page 38: ... be used for encryption are indicated by the following sed_capable true used_for_boot false The drives that t be used for encryption are indicated by both of the following options sed_capable true used_for_boot true sed_capable false 5 5 1 2 Creating the Drive Password Mapping JSON File You can create the drive password mapping JSON file and use this file to initialize the system 1 Create a JSON f...

Page 39: ...ach drive sudo nv disk encrypt init k your vault password g r sudo nv disk encrypt lock The vault password must consist of only upper case letters lower case letters digits and or these special characters _ and 5 5 3 Example 3 Specifying Passwords One at a Time When Prompted This example provides information about how you can specify passwords one at a time If there are a small number of drives or...

Page 40: ... task in 50 words or fewer Stop cachefilesd and unmount the RAID array CAUTION When you complete this task all data will be lost On DGX Station A100 systems these drives generally form a RAID 0 array which will also be destroyed when you performe an erase After you initialize the system for SED management use the nv disk encrypt command to erase data on your drives 1 To completely stop the RAID is...

Page 41: ...nter the BIOS Setup in the NVIDIA splash screen press Del or F2 3 On the Advanced tab scroll to Trusted Computing and press Enter 4 To enable TPM scroll to Security Device and switch the setting to Enabled 5 To continue the boot process save and exit the BIOS Setup After the system boots you can initialize the drive encryption 5 9 2 Clearing the TPM If you lost your TPM password you cannot access ...

Page 42: ...e Initializing the System for Drive Encryption 5 11 Recovering a Lost Key NVIDIA recommends that you back up your keys and store the keys in a secure location If you lose the key that was used to initialize and lock your drives you cannot unlock the drive The only way to recover is to perform a factory reset which will result in data loss SED drives come with a PSID printed on the label This value...

Page 43: ... A100 and repacking it for shipment 6 1 Unpacking the DGX Station A100 After you receive your DGX Station A100 carefully unpack it CAUTION The DGX Station A100 weighs 91 lbs 43 1 kg Do not attempt to lift the DGX Station A100 Instead move it into position by rolling it on its fitted casters 1 Open the top flap and remove the Accessory box and the power cord box ...

Page 44: ...100 DGX Station A100 DU 10189 001 _v5 0 2 38 2 Disengage and remove the packing clasps from the cutouts in the shipping carton Do not use excessive force when removing the clasps to prevent them from becoming jammed inside the shipping carton ...

Page 45: ... DGX Station A100 DU 10189 001 _v5 0 2 39 3 Raise the top cover of the shipping carton 4 Fold down the ramp at the front of the bottom tray of the DGX Station A100 shipping carton and remove the packing material from the top 5 Swing the door open ...

Page 46: ... the DGX Station A100 for Shipment If you are returning the DGX Station A100 to NVIDIA under an RMA repack it in the packaging in which the replacement unit was advanced shipped to prevent damage during shipment CAUTION The DGX Station A100 weighs 91 lbs 43 1 kg Do not attempt to lift the DGX Station A100 Instead move it into position by rolling it on its fitted casters 1 Carefully roll the DGX St...

Page 47: ...cking the DGX Station A100 DGX Station A100 DU 10189 001 _v5 0 2 41 2 Swing the door closed 3 Fold up the ramp at the front of the bottom tray of the DGX Station A100 shipping carton and place the packing material on top ...

Page 48: ... 10189 001 _v5 0 2 42 4 Lower the top cover of the shipping carton 5 Re engage and insert the packing clasps into the cutouts in the shipping carton Do not use excessive force when inserting the clasps to prevent them from becoming jammed inside the shipping carton ...

Page 49: ...Unpacking and Repacking the DGX Station A100 DGX Station A100 DU 10189 001 _v5 0 2 43 6 Close the top flap and place the Accessory box and the power cord box ...

Page 50: ...les the default user and scramble its password so it cannot be used to access the BMC The username for the new BMC user will be the same as the username that you entered for the operating system administrative user The password will be unique because it is entered separately during the operating system configuration process Plugging in the Network Cable After you complete the initial configuration...

Page 51: ...he product Fire Overload temperature material flammability Mechanical Sharp edges moving parts instability Energy Circuits with high energy levels 240 volt amperes or potential as burn hazards Heat Accessible parts of the product at high temperatures Chemical Chemical fumes and vapors Radiation Noise ionizing laser ultrasonic waves Retain and follow all product safety and operating instructions Al...

Page 52: ...ot attempt to defeat safety interlocks where provided Operate the DGX Station A100 in a place where the temperature is always in the range 10 C to 35 C 50 F to 95 F B 3 Electrical Precautions Power Cable To reduce the risk of electric shock fire or damage to the equipment Use only the supplied power cable and do not use this power cable with any other products or for any other purpose Not all powe...

Page 53: ...outlet is near the equipment and is readily accessible for disconnection To help protect your system from sudden transient increases and decreases in electrical power consider using a surge suppressor or line conditioner Never force a connector into a port Check for obstructions on the port If the connector and port don t join with reasonable ease they probably don t match Make sure that the conne...

Page 54: ...ork contains some nickel The metal foam is not intended for direct and prolonged skin contact While nickel exposure is unlikely to be a problem you should be aware of the possibility in case you re susceptible to nickel related reactions CAUTION There is the danger of explosion if the battery is incorrectly replaced When replacing the battery use only the battery recommended by the equipment manuf...

Page 55: ...ation A100 DU 10189 001 _v5 0 2 49 Appendix C Connections Controls and Indicators C 1 Front Panel Connections and Controls ID Type Qty Description 1 Power Button 1 Press to turn the DGX Station A100 on or off ...

Page 56: ... 1 Type C port 2 Ethernet 2 RJ45 10G LAN ports Remote Management 1 RJ45 1G LAN dedicated BMC Management MGMT 3 USB 3 1 x 2 Type A 1 2 x USB 3 1 Type A port 6 AC Input 1 Power supply input 7 Reset Button 1 Press to reboot the system without turning off the system power 10 DisplayPort 3 Ports for connecting up to 3 displays 11 Power Supply Switch 1 Turn the power supply on and off ...

Page 57: ...s and Indicators DGX Station A100 DU 10189 001 _v5 0 2 51 C 3 LAN Port Indicators LEDs on each Ethernet LAN port indicate the connection status as illustrated in the following figure and described in the following tables ...

Page 58: ...s DGX Station A100 DU 10189 001 _v5 0 2 52 Speed LED Status Description Off 100 Mbps connection Orange 1 Gbps connection Green 10 Gbps connection Activity Link LED Status Description Off No link Green Linked Green blinking Data activity ...

Page 59: ...e regulations listed in this section D 1 DGX Station A100 Model Number Model P3487 D 2 Australia New Zealand Australian Communications and Media Authority RCM This product meets the applicable EMC requirements for Class A I T E equipment D 3 Brazil Instituto Nacional de Metrologia Qualidade e Tecnologia INMETRO ...

Page 60: ...pment Canada ISED CAN ICES 003 A NMB 003 A The Class A digital apparatus meets all requirements of the Canadian Interference Causing Equipment Regulation Cet appareil numérique de la classe A respecte toutes les exigences du Règlement sur le matériel brouilleur du Canada D 5 China RoHS Material Content ...

Page 61: ...as been marked with the CE Mark to illustrate its compliance This device complies with the following Directives EMC Directive 2014 30 EU for Class A I T E equipment Low Voltage Directive 2014 35 EU for electrical safety RoHS Directive 2011 65 EU for hazardous substances ErP Directive 2009 125 EC for European Ecodesign A copy of the Declaration of Conformity to the essential requirements may be obt...

Page 62: ...related consumables and spares complies with the reduction in hazardous substances provisions of the India E waste Management and Handling Rule 2016 It does not contain lead mercury hexavalent chromium polybrominated biphenyls or polybrominated diphenyl ethers in concentrations exceeding 0 1 weight and 0 01 weight for cadmium except for where allowed pursuant to the exemptions set in Schedule 2 of...

Page 63: ...Compliance DGX Station A100 DU 10189 001 _v5 0 2 57 D 8 Japan Voluntary Control Council for Interference VCCI Japan RoHS Material Content Declaration ...

Page 64: ...ХНИЧЕСКИЙ РЕГЛАМЕНТ ТАМОЖЕННОГО СОЮЗА Электромагнитная совместимость технических средств ТР ТС 020 2011 Технический регламент Евразийского экономического союза Об ограничении применения опасных веществ в изделиях электротехники и радиоэлектроники ТР TC 037 2016 D 11 South Africa South African Bureau of Standards SABS This device complies with the following SABS Standards SANS 2332 2017 CISPR 32 20...

Page 65: ...Compliance DGX Station A100 DU 10189 001 _v5 0 2 59 SANS IEC 60950 1 D 12 South Korea Korean Certification KC D 13 Taiwan Bureau of Standards Metrology Inspection BSMI ...

Page 66: ...Compliance DGX Station A100 DU 10189 001 _v5 0 2 60 D 14 United Kingdom UK Conformity Assessed UKCA ...

Page 67: ...device may not cause harmful interference and 2 this device must accept any interference received including any interference that may cause undesired operation of the device NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference ...

Page 68: ...Compliance DGX Station A100 DU 10189 001 _v5 0 2 62 D 16 United States Canada TUV Rheinland Energy Star Energy Star qualified server ...

Page 69: ...on CPU 1 Single AMD 7742 64 cores 2 25 GHz base 3 4 GHz max boost and 2 25 GHz base 3 4 GHz max boost GPU current units 4 4x NVIDIA A100 80 GB GPUs 5 petaOPS INT8 4 80 GB 320 GB total GPU memory 2 5 petaFLOPS AI GPU earlier units 4 NVIDIA Tesla V100 DGXS 16GB featuring 4 125 TeraFLOPS 500 TeraFLOPS total FP16 4 16 GB 64 GB total GPU memory 4 640 2 560 total NVIDIA Tensor Cores 4 5 120 20 480 total...

Page 70: ...m Width 10 1 256 mm Depth 20 4 518 mm Gross weight 91 lbs 43 1 kg E 4 Power Specifications RESTRICTION The power source for DGX Station A100 must be 100V and cannot fall below 90V Input Comments 100 115VAC 15A 115 120VAC 12A 200 240VAC 10A and 50 60Hz The DGX Station A100 power consumption can reach 1 500 W ambient temperature 30 C with all system resources under a heavy load Be aware of your elec...

Page 71: ... problems with your DGX Station A100 system Also contact NVIDIA Enterprise Support for assistance in installing or moving the DGX Station A100 system For details on how to obtain support visit the NVIDIA Enterprise Support web site https www nvidia com en us support enterprise Our support team can help collect appropriate information about your issue and involve internal resources as needed ...

Page 72: ...omer s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and or requirements beyond those contained in this document NVIDIA accepts no liability related to any default damage costs or problem which may be based on or attributable to i the use of the NVIDIA product in any manner that is contrary to this document or ii c...

Reviews: