diff --git a/work/Nexus_1_Build.md b/work/Nexus_1_Build.md new file mode 100644 index 0000000..42f67e7 --- /dev/null +++ b/work/Nexus_1_Build.md @@ -0,0 +1,715 @@ +--- +title: C9300GX-1 Build +description: +published: true +date: 2026-02-19T20:45:10.926Z +tags: +editor: markdown +dateCreated: 2026-02-19T20:45:10.926Z +--- + +# AT1EU-NEXUS-1 — Cisco Nexus 9300 Configuration + +## Overview + +AT1EU-NEXUS-1 is the **primary** switch in a vPC pair (role priority 10, lower = preferred). It runs NX-OS 10.3(7) and forms a vPC domain with AT1EU-NEXUS-2. The two switches share a vPC peer-link (Po10) across Eth1/47–48, and use out-of-band management (mgmt0 at 192.168.0.1) for the vPC peer-keepalive path. + +**Key roles of this switch:** +- vPC primary (role priority 10) +- STP root bridge for management/native VLANs (priority 8192 for VLANs 1, 66) +- Layer 3 gateway for Vlan502 (Atom VRF, IP 15.0.2.121/24) +- NTP master (stratum 3) +- Upstream connections: 500e-X1 (Po3), 500e-X2 (Po4), 9300 (Po124) +- Storage connections: AFF300-A (Po127), AFF300-B (Po128), FAS2750-A (Po129), FAS2750-B (Po130), A70-A (Po131), A70-B (Po132) +- Compute connections: UCS-A (Po125), UCS-B (Po126) + +--- + +## Cut-and-Paste Configuration + +``` +version 10.3(7) Bios:version 07.71 +switchname AT1EU-NEXUS-1 + +! --- QoS: Jumbo Frame Policy --- +policy-map type network-qos JUMBO + class type network-qos class-default + mtu 9216 + +! --- VDC Resource Limits --- +vdc AT1EU-NEXUS-1 id 1 + limit-resource vlan minimum 16 maximum 4094 + limit-resource vrf minimum 2 maximum 4096 + limit-resource port-channel minimum 0 maximum 511 + limit-resource m4route-mem minimum 58 maximum 58 + limit-resource m6route-mem minimum 8 maximum 8 + +! --- Features --- +feature nxapi +feature bash-shell +feature scp-server +cfs eth distribute +feature udld +feature interface-vlan +feature lacp +feature vpc +feature lldp +feature telemetry + +! --- RBAC --- +role name network-ro + rule 2 permit read + rule 1 permit command show running-config + +! --- Users --- +username admin password 5 $5$MFJCIC$AJyskD7vdoVFKK5cTS2lO20omFL4XFrgqNB94qDA5Z2 role network-admin +ssh key rsa 2048 + +! --- Banner --- +banner motd ^ +********************* DOD NOTICE AND CONSENT BANNER ************************* +* You are accessing a U.S. Government (USG) Information System (IS) that is * +* provided for USG-authorized use only. By using this IS (which includes any* +* device attached to this IS), you consent to the following conditions: * +*-The USG routinely intercepts and monitors communications on this IS for * +* purposes including, but not limited to, penetration testing, COMSEC * +* monitoring, network operations and defense, personnel misconduct (PM), * +* law enforcement (LE), and counterintelligence (CI) investigations. * +*-At any time, the USG may inspect and seize data stored on this IS. * +*-Communications using, or data stored on, this IS are not private, are * +* subject to routine monitoring, interception, and search, and may be * +* disclosed or used for any USGauthorized purpose. * +*-This IS includes security measures (e.g., authentication and access * +* controls) to protect USG interests--not for your personal benefit or * +* privacy. * +*-Notwithstanding the above, using this IS does not constitute consent to * +* PM, LE or CI investigative searching or monitoring of the content of * +* privileged communications, or work product, related to personal * +* representation or services by attorneys, psychotherapists, or clergy, and * +* their assistants. Such communications and work product are private and * +* confidential. See User Agreement for details. * +************************ POC: SIL Network Team **************************** +^ + +! --- SSH --- +ssh ciphers aes256-gcm + +! --- DNS & Domain --- +ip domain-lookup +ip name-server 15.0.2.128 15.0.2.129 15.32.2.128 +ip domain-name atom.dev use-vrf Atom +ip name-server 15.0.2.128 15.0.2.129 15.32.2.128 use-vrf Atom + +! --- RADIUS --- +radius-server host 15.0.11.68 key 7 "V1P-jaynmv" authentication accounting +radius-server host 15.32.11.68 key 7 "V1P-jaynmv" authentication accounting +aaa group server radius NETMAN_RADIUS + server 15.0.11.68 + server 15.32.11.68 + use-vrf Atom + +! --- Management ACL --- +ip access-list SWITCH_MGMT + 10 permit ip 15.0.11.150/32 any log + 20 permit ip 15.0.11.151/32 any log + 30 permit ip 15.32.2.154/32 any log + 40 permit ip 15.0.2.154/32 any log + 50 permit ip 15.32.2.1/32 any log + 60 permit ip 15.0.2.1/32 any log + 70 permit ip 15.0.2.2/32 any log + 80 permit ip 15.0.11.47/32 any log + 90 permit ip 15.32.11.45/32 any log + 93 permit ip 15.32.11.150/32 any log + 100 deny ip any any log + +! --- System QoS --- +system qos + service-policy type network-qos JUMBO +copp profile strict + +! --- SNMP --- +snmp-server user admin network-admin auth sha 042F64DB5D2E0D40DF543D6A00495F1F18F9DD5FED7B priv aes-128 00540CF9793F282ED96D666B110B00753FC3F269E964 localizedV2key +snmp-server host 15.0.2.188 traps version 3 priv at-sw-svc +snmp-server enable traps config ccmCLIRunningConfigChanged +rmon event 1 log trap public description FATAL(1) owner PMON@FATAL +rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL +rmon event 3 log trap public description ERROR(3) owner PMON@ERROR +rmon event 4 log trap public description WARNING(4) owner PMON@WARNING +rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO + +! --- NTP --- +ntp server 15.0.0.9 prefer use-vrf Atom key 123 +ntp server 15.32.0.9 prefer use-vrf Atom key 125 +ntp source-interface Vlan502 +ntp authenticate +ntp authentication-key 123 md5 pz5yamz 7 +ntp trusted-key 123 +ntp logging +ntp master 3 + +! --- AAA --- +aaa authentication login default group NETMAN_RADIUS local +aaa authentication login console group NETMAN_RADIUS local +aaa accounting default group NETMAN_RADIUS local +system default switchport +no ip source-route + +! --- VLANs --- +vlan 1-2,8,10,12,66,85,100-103,107-108,121-124,129-130,142-143,145-146,148-150,153,157-158,188,305,321,323,340,342,349,353,374,382,501-502,504-505,549,551,559,562-563,600,611,660-661,667-668,672-673,697-698,701-702,704-710,720-722,724,727,740,750-751,772,777,800-802,804,814,820-823,905,1051,1127,1129,1160-1161,1551,1559-1560,1670-1674,1720-1722,1800-1802,1814-1817,1862,1865,1870-1871 +vlan 1882-1883,1885,1905,3563,3965 +vlan 2 + name TEST_CLUS_COMM +vlan 8 + name FP_Test1 +vlan 10 + name NESS_BOX_TRANSIT +vlan 12 + name FP_Test2 +vlan 66 + name NATIVE_VLAN +vlan 85 + name NESS_Temp +vlan 100 + name migration +vlan 101 + name iscsi_csv +vlan 102 + name iscsi_boot +vlan 103 + name Netapp_XFER +vlan 107 + name Test +vlan 108 + name NET_TEST_NET +vlan 121 + name Atom_Backup +vlan 123 + name storage +vlan 124 + name Admin_iSCSI +vlan 130 + name SIL_SNAPMIRROR +vlan 143 + name Secman_Storage +vlan 146 + name Foxhound_Storage +vlan 150 + name iscsi +vlan 153 + name Javelin(L4) +vlan 157 + name GNext_Storage +vlan 158 + name Ness_Storage +vlan 188 + name JASON_NFS +vlan 321 + name ATOM_Backup +vlan 323 + name AT-vServer +vlan 340 + name ucs_test +vlan 342 + name MadHatter_SVM_Mgmt +vlan 349 + name Rock_SVM3_Mgmt +vlan 353 + name Javlin_SVM +vlan 374 + name Rock_Backup_Mgmt +vlan 382 + name Darrin_User +vlan 501 + name MGMT +vlan 502 + name Atom_User2 +vlan 504 + name Commvault_Test +vlan 505 + name NETAPP_SNAP +vlan 549 + name WDS +vlan 551 + name L4_User +vlan 559 + name Victory_WS_L4 +vlan 562 + name Brace(L3)_User +vlan 667 + name Britt_Test +vlan 668 + name RockTesters(L4)_User +vlan 672 + name GTRI_User +vlan 673 + name VDI(L5) +vlan 701 + name MH_L3_DATA_HLCI +vlan 702 + name MH_L4_DATA_HLCI +vlan 704 + name Legacy-704 +vlan 705 + name Legacy-705 +vlan 706 + name Legacy-706 +vlan 707 + name Legacy-707 +vlan 708 + name Legacy-708 +vlan 709 + name Legacy-709 +vlan 710 + name Legacy-710 +vlan 721 + name GTRI_JAVELIN_L4-721 +vlan 740 + name NETMAN +vlan 750 + name l4_secman +vlan 751 + name Secman_DMP-751 +vlan 777 + name FTD1010_TSHOOT +vlan 804 + name FH_L4_HLCI +vlan 814 + name Rock_L4 +vlan 820 + name GNext_User +vlan 821 + name GNext_Sentris +vlan 822 + name GNext_VPX +vlan 823 + name GNext_VDA +vlan 905 + name Rock_(L4) +vlan 1051 + name IP_SEC_1010 +vlan 1127 + name Vic_Storage +vlan 1551 + name Services(L3)_User +vlan 1559 + name Victory(L3)_User +vlan 1670 + name BigTen_User +vlan 1671 + name Victory_DMP-1671 +vlan 1672 + name VIC_VDI +vlan 1673 + name Victory_Sentris +vlan 1720 + name Javelin(L3)_User +vlan 1721 + name GTRI_JAVELIN_L3-1721 +vlan 1722 + name Victory_VDI-1722 +vlan 1800 + name Foxhound(L3)_User +vlan 1801 + name FH_L3_DATA_HLCI +vlan 1814 + name ROCK_L3_MLS +vlan 1815 + name ServMan_User +vlan 1870 + name AT1EU-JavelinCoop(L3)_User +vlan 1883 + name NESS_User +vlan 1885 + name NESS_Client +vlan 1905 + name Rock(L3)_User +vlan 3563 + name Brace_User +vlan 3965 + name V3E_DEV_HOST + +! --- Spanning Tree --- +spanning-tree port type edge bpduguard default +spanning-tree port type edge bpdufilter default +spanning-tree port type network default +spanning-tree vlan 1,66 priority 8192 +spanning-tree vlan 2,100-102,107-108,121-123,129,142,145,148-150,153,305,323,340,353,382,501-502,505,549,551,562-563,600,611,660-661,667-668,672,697-698,701-702,704-710,720-722,724,727,750,772,800-802,804,814,905,1127,1129,1160-1161,1551,1559-1560,1670,1672-1673,1720-1721,1800-1802,1814-1817,1862,1865,1870-1871,1882,1905,3563,3965 priority 24576 +spanning-tree vlan 3-65,67-99,103-106,109-120,124-128,130-141,143-144,146-147,151-152,154-304,306-322,324-339,341-352,354-381,383-500,503-504,506-548,550,552-561,564-599,601-610,612-659,662-666,669-671,673-696,699-700,703,711-719,723,725-726,728-749,751-771,773-799,803,805-813,815-904,906-1126,1128,1130-1159,1162-1550,1552-1558,1561-1669,1671,1674-1719,1722-1799,1803-1813,1818-1861,1863-1864,1866-1869,1872-1881,1884-1904,1906-3562,3564-3964,3966-3967 priority 0 +spanning-tree vlan 1883 priority 4096 + +! --- VRF --- +vrf context Atom + ip domain-name atom.dev + ip name-server 15.0.2.128 15.0.2.129 15.32.2.128 + ip route 0.0.0.0/0 15.0.2.254 +vrf context management + +! --- Port-Channel Load Balance --- +port-channel load-balance src-dst ip-l4port-vlan + +! --- vPC Domain --- +vpc domain 1 + peer-switch + role priority 10 + peer-keepalive destination 192.168.0.2 source 192.168.0.1 + delay restore 150 + peer-gateway + auto-recovery + +! --- SVI --- +interface Vlan1 + +interface Vlan502 + no shutdown + vrf member Atom + no ip redirects + ip address 15.0.2.121/24 + no ipv6 redirects + +! --- Port-Channels --- +interface port-channel3 + description //Trunk 500e X1 + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + vpc 3 + +interface port-channel10 + description //Trunk Peer - Allow STP + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type network + vpc peer-link + +interface port-channel124 + description //Trunk 9300 + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type normal + spanning-tree bpduguard disable + spanning-tree guard root + mtu 9216 + no lacp suspend-individual + vpc 124 + +interface port-channel125 + description //Trunk UCS-A + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard disable + spanning-tree guard root + mtu 9216 + vpc 125 + +interface port-channel126 + description //Trunk UCS-B + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard disable + spanning-tree guard root + mtu 9216 + vpc 126 + +interface port-channel127 + description //Trunk AFF300-A + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree guard root + mtu 9216 + vpc 127 + +interface port-channel128 + description //Trunk AFF300-B + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree guard root + mtu 9216 + vpc 128 + +interface port-channel129 + description //Trunk FAS 2750-A + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + storm-control broadcast level 99.00 + storm-control unicast level 99.00 + switchport block unicast + vpc 129 + +interface port-channel130 + description //Trunk Fas 2750-B + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + storm-control broadcast level 99.00 + storm-control unicast level 99.00 + switchport block unicast + vpc 130 + +interface port-channel131 + description //Trunk A70-A + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree guard root + mtu 9216 + vpc 131 + +interface port-channel132 + description //Trunk A70-B + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree guard root + mtu 9216 + vpc 132 + +! --- Breakout Ports (100G -> 4x25G) --- +int e1/1 - 26 + shutdown +exit +interface breakout module 1 port 1 map 25g-4x +interface breakout module 1 port 5 map 25g-4x + + +! --- Physical Interfaces: Breakout (UCS/A70) --- +interface Ethernet1/1/1 + description //Trunk 6554-1:25 + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 125 mode active + no shutdown + +interface Ethernet1/1/2 + description //Trunk 6554-1:26 + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 125 mode active + no shutdown + +interface Ethernet1/1/3 + description //Trunk 6554-2:27 + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 126 mode active + no shutdown + +interface Ethernet1/1/4 + description //Trunk 6554-2:28 + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 126 mode active + no shutdown + +interface Ethernet1/5/1 + description //Trunk A70-A + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 131 mode active + no shutdown + +interface Ethernet1/5/2 + description //Trunk A70-A + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 131 mode active + no shutdown + +interface Ethernet1/5/3 + description //Trunk A70-B + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 132 mode active + no shutdown + +interface Ethernet1/5/4 + description //Trunk A70-B + switchport mode trunk + switchport access vlan 67 + switchport trunk native vlan 66 + switchport trunk allowed vlan 2-66,68-4094 + spanning-tree port type edge trunk + spanning-tree bpduguard enable + spanning-tree guard root + mtu 9216 + channel-group 132 mode active + no shutdown + + + +! +! --- Bulk Disabled Ports --- +int e1/3/1-4,e1/7/1-4,e1/11/1-4,e1/13-23 + description //Disabled access + switchport access vlan 67 + switchport trunk native vlan 66 + spanning-tree port type edge + spanning-tree bpduguard enable + spanning-tree guard root + storm-control broadcast level 99.00 + storm-control unicast level 99.00 + switchport block unicast + udld enable + shutdown + +! --- Management Interface --- +interface mgmt0 + vrf member management + ip address 192.168.0.1/24 + +icam monitor scale + +! --- Console & VTY --- +line console + exec-timeout 5 +line vty + session-limit 4 + exec-timeout 0 + access-class SWITCH_MGMT in + + +! --- Logging --- +logging ip access-list cache entries 8001 +logging logfile LOG_FILE 6 size 4096 +logging server 15.0.2.146 2 +logging server 15.0.2.222 6 +logging level authpri 6 + +intersight use-vrf Atom +``` + +--- + +## Configuration Explanation + +### Platform & Global Settings +Running NX-OS 10.3(7) with a Jumbo MTU QoS policy (9216 bytes) applied globally via `system qos`. IP source-route is disabled. SSH is restricted to AES256-GCM ciphers. CoPP is set to strict for control-plane protection. + +### VDC Resource Limits +Standard resource limits for a single-VDC 9300 — up to 4094 VLANs, 4096 VRFs, and 511 port-channels. + +### Features Enabled +`nxapi`, `bash-shell`, `scp-server`, `udld`, `interface-vlan`, `lacp`, `vpc`, `lldp`, `telemetry`, and CFS Ethernet distribution for vPC. + +### Authentication & Access Control +RADIUS authentication via two servers (15.0.11.68 and 15.32.11.68) in the `NETMAN_RADIUS` group, using the `Atom` VRF. AAA fallback is local. VTY access is restricted to the `SWITCH_MGMT` ACL (specific management host IPs only, with a deny-all default). VTY timeout is 0 (no timeout — note this differs from NEXUS-2 which uses 5 minutes). + +### NTP +Two NTP servers in the Atom VRF (preferred) with MD5 authentication. NTP source is Vlan502. This switch acts as NTP master stratum 3. + +### SNMP +SNMPv3 with SHA auth and AES-128 privacy. Traps sent to 15.0.2.188. RMON events configured for severity levels 1–5. + +### VLANs +Approximately 200 VLANs are defined, covering storage (iSCSI, NFS, SnapMirror), compute (UCS, HLCI workloads), management, user, and VDI segments. VLAN 66 is the native VLAN; VLAN 67 is the unused/quarantine access VLAN for disabled ports. + +### Spanning Tree +STP is configured with global edge/bpduguard and bpdufilter defaults for access ports, and network type for uplinks. This switch holds STP root priority 8192 for VLANs 1 and 66, making it the root for those VLANs. Most production VLANs are set to priority 24576 (secondary root). Unused VLANs are set to priority 0 (disabled from becoming root). + +### VRF & Routing +A single non-default VRF `Atom` carries the management/user traffic with a default route to 15.0.2.254. Vlan502 (`Atom_User2`) is the L3 gateway SVI at 15.0.2.121/24. + +### vPC Domain +- **Domain:** 1 +- **Role Priority:** 10 (primary) +- **Peer-link:** Po10 (Eth1/47–48), `spanning-tree port type network` +- **Peer-keepalive:** mgmt0, destination 192.168.0.2, source 192.168.0.1 +- **Options:** `peer-switch`, `peer-gateway`, `auto-recovery`, 150-second restore delay +- **vPC members:** Po3 (500e-X1), Po4 (500e-X2), Po124 (9300), Po125 (UCS-A), Po126 (UCS-B), Po127 (AFF300-A), Po128 (AFF300-B), Po129 (FAS2750-A), Po130 (FAS2750-B), Po131 (A70-A), Po132 (A70-B) + +### Port-Channel Load Balancing +`src-dst ip-l4port-vlan` — distributes traffic based on source/destination IP, L4 port, and VLAN for optimal flow distribution. + +### Physical Interfaces +- **Ports 1/1–1/26:** Shut down as a group first, then individual interfaces are re-configured. Ports 1, 5, and 9 are broken out as 4x25G sub-interfaces. +- **Eth1/1/1–1/1/4:** 25G breakout ports to UCS 6554 FIs → Po125/Po126 +- **Eth1/5/1–1/5/4:** 25G breakout ports to A70 storage arrays → Po131/Po132 +- **Eth1/24–1/25, 1/45–1/46:** 9300 uplink → Po124 (4-link LACP) +- **Eth1/26:** 500e-X1 → Po3 +- **Eth1/18:** 500e-X2 → Po4 +- **Eth1/47–1/48:** vPC peer-link → Po10 +- **Eth1/53–1/54:** AFF300-A/B → Po127/Po128 +- **Eth1/2–1/3:** FAS2750 → Po129/Po130 +- **Disabled ports:** Placed in VLAN 67, bpduguard enabled, storm-control, UDLD, unicast block — shutdown + + +### Logging +Syslog to 15.0.2.146 (severity 2) and 15.0.2.222 (severity 6). Local log file `LOG_FILE` at severity 6. ACL hit caching configured for 8001 entries. +