来自频道: @DMIT_INC
来自频道: @DMIT_INC
DMIT already aware about LAX EB packet loss on China direction.

We observe there is no packet loss on PtP segment.

The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue.

DMIT will keep you updated.
来自频道: @DMIT_INC
来自频道: @DMIT_INC
Edge Router hardware swap.

Time: 10AM-1PM JST, 12/18/2024

Expected maximum downtime: 60mins
来自频道: @DMIT_INC
Adjusting TYO network arch, some product experienced downtime, and it has been fixed now.
There are couple works need to be done before return to stable.
来自频道: @DMIT_INC
/var/log/DMIT-NOC.log
Content:
- Network Upgrade
- Hardware Upgrade
- Storage Upgrade
- Network Upgrade in progress.
-- Adding: RETN 100G, NTT 100G, BBIX 100G;
-- Remove: Equinix IX 10G, Cogent 10G; NTT 10G;
-- ETA: within a week.

- Hardware Upgrade completed
-- 7402P > 7443P
-- 10G * 2 per Node > 100G * 2 per Node
-- 20G uplink per rack > 100G * 2 uplink per rack.

- Storage Upgrade completed
-- SATA 10G SDS Cluster > NVMe 100G*2 SDS Cluster.
来自频道: @DMIT_INC
Observed TOR issue on TYO, dispatching engineer.
Done
来自频道: @DMIT_INC
/var/log/DMIT-NOC.log
TYO: Pending reboot action in an hour, expect VM reboot, and upto 1hr.
- Extend 30 mins.
- Extend 45 mins again. Due to the initial RH does not record the label correct; it took many time to trace and re-cable.
- Ready in 11:30 JST
来自频道: @DMIT_INC
来自频道: @DMIT_INC
LAX:
Emergency Rebooting one of hypervisor; replace NICs.
Done.
来自频道: @DMIT_INC
Schedule Maintenance:
Location: Tokyo;
Date: Nov 18 ~ Dec 18, 2024

Content:
- Network Upgrade
- Hardware Upgrade
- Storage Upgrade

Impact:
- Network jitter, instability and downgrade;
- Possible SSH Key change due to hardware migrate;
- Experience downtime.
来自频道: @DMIT_INC
LAX:
Emergency Rebooting one of hypervisor
来自频道: @DMIT_INC
/var/log/DMIT-NOC.log
Traffic switching in progress; Routing Engine reboot in progress.
Done; upgrade the another RE;
Should expect no traffic stop.
来自频道: @DMIT_INC
来自频道: @DMIT_INC
There is a pending upgrade on LAX's MX Edge Router.
Expect maximum 30 mins down time and 60 mins network downgrade.

Time: 3PM ~ 6PM PDT.

DMIT.io will also be impacted at the same time.
来自频道: @DMIT_INC
/var/log/DMIT-NOC.log
Photo
Interface configuration error. Revised.
来自频道: @DMIT_INC
来自频道: @DMIT_INC
来自频道: @DMIT_INC