There is an uncommon, and undocumented bug on JunOS that make unpredictable error in the routing and rdp.
We’ve ran many test past 2 months to ident the root cause.
The issue on Los Angeles RIB/FIB desync has been resolved.
Like some China Unicom prefer to use CMIN2 even the best route is 9929 in the RIB, and backup in CMIN2.
Also some GTT route will not go to GTT IP Transit even it’s the best and no other comparable selection. (Failover to default, use GSL..)
There is a pending change to fix this problem entirely that might be bring internal BGP reset. DMIT will perform this action at super off-peak time. The estimate impact time is less than 10 minus.
We’ve ran many test past 2 months to ident the root cause.
The issue on Los Angeles RIB/FIB desync has been resolved.
Like some China Unicom prefer to use CMIN2 even the best route is 9929 in the RIB, and backup in CMIN2.
Also some GTT route will not go to GTT IP Transit even it’s the best and no other comparable selection. (Failover to default, use GSL..)
There is a pending change to fix this problem entirely that might be bring internal BGP reset. DMIT will perform this action at super off-peak time. The estimate impact time is less than 10 minus.
来自频道: @DMIT_INC
CoreSite Edge Router encounter an unexpected issue bring the rpd process crash; rollback and it was resovled.
来自频道: @DMIT_INC
There are couple fiber cut inside APAC region;
GSL is now losing both short path connection between HK and TY.
RETN is also experience congestion.
NTT won't be good, as our experience.
DMIT is now configuring router to adapter the MPLS Ethernet connectivity.
After that, we can delivery the PNI and ASE connection.
* The TPE-TYO of GSL shows available but 0% IN/OUT; might need wait for GSL furtuer action.
来自频道: @DMIT_INC
📢 [Network Update – China Unicom Packet Loss in LAX]
We have observed packet loss on EB and Pro routes via China Unicom (AS9929), primarily caused by a DDoS attack targeting AS9929 customers.
China Unicom does not drop attack traffic within China when AS10099 customer announces RTBH (Blackhole). Instead, traffic is blocked only at overseas Provider Edge (PE) routers. If the attack volume exceeds their backbone capacity, packet loss and jitter are unavoidable.
🔒 DMIT has already implemented second-level DDoS detection and automatic RTBH to minimize impact, with victim IPs blackholed in under few seconds. This protects our uplink with AS10099 from congestion.
However, please note:
This cannot prevent packet loss if the attack targets other AS10099 customers.
It is also ineffective if the attack exceeds China Unicom's international backbone capacity (AS9929/AS10099).
China Unicom does not support RTBH natively on AS4837 and instead sells it as a premium service. Even with this feature, we cannot mitigate attacks aimed at unrelated IPs. (like if there is attack to other AS10099 customers.)
Unfortunately, resolution is not possible unless China Unicom changes its policies—which appears unlikely, as the current structure incentivizes selling RTBH as a product.
We appreciate your understanding and we will continue to monitor and mitigate as much as possible on our end.
We have observed packet loss on EB and Pro routes via China Unicom (AS9929), primarily caused by a DDoS attack targeting AS9929 customers.
China Unicom does not drop attack traffic within China when AS10099 customer announces RTBH (Blackhole). Instead, traffic is blocked only at overseas Provider Edge (PE) routers. If the attack volume exceeds their backbone capacity, packet loss and jitter are unavoidable.
🔒 DMIT has already implemented second-level DDoS detection and automatic RTBH to minimize impact, with victim IPs blackholed in under few seconds. This protects our uplink with AS10099 from congestion.
However, please note:
This cannot prevent packet loss if the attack targets other AS10099 customers.
It is also ineffective if the attack exceeds China Unicom's international backbone capacity (AS9929/AS10099).
China Unicom does not support RTBH natively on AS4837 and instead sells it as a premium service. Even with this feature, we cannot mitigate attacks aimed at unrelated IPs. (like if there is attack to other AS10099 customers.)
Unfortunately, resolution is not possible unless China Unicom changes its policies—which appears unlikely, as the current structure incentivizes selling RTBH as a product.
We appreciate your understanding and we will continue to monitor and mitigate as much as possible on our end.
来自频道: @DMIT_INC
来自频道: @DMIT_INC
[HK Maintenance]
Date: June 27 - July 1, 2025
Content:
- Replacement of edge router outdated 10Gbps line cards(2x);
---- CMI expected upto 1hr downgrade;
---- CN2 GIA has redundnacy circuits expected graceful switch over;
- Replacement of main switch;
---- To Juniper QFX5200-32C (from Arista 7280CR)
- Replacement of TOR switch;
---- To Juniper QFX5200-32C (from QFX5100-24Q)
- Replacement of on-site DDoS mitigation facility;
- Repair error virtualization node;
- Adding virtualization nodes;
- Adding 480G edge router line card.
- Adding 2x100G peering PNI;
- Adding ASE connectivity to Tokyo.
Date: June 27 - July 1, 2025
Content:
---- CMI expected upto 1hr downgrade;
---- CN2 GIA has redundnacy circuits expected graceful switch over;
- Replacement of main switch;
---- To Juniper QFX5200-32C (from Arista 7280CR)
- Replacement of TOR switch;
---- To Juniper QFX5200-32C (from QFX5100-24Q)
- Replacement of on-site DDoS mitigation facility;
- Repair error virtualization node;
- Adding virtualization nodes;
- Adding 480G edge router line card.
来自频道: @DMIT_INC
[APAC Network Notice]
DMIT has connect with GSL in HKG; due to GSL has more peers than RETN, most network are favor GSL rather than RETN.
We cannot force make any networks favor RETN in EU, because
- AS3257 is RETN's IP Transit; they only connect it at EU. This make any AS3356 customer where ever they at, go to EU. (Like APAC/US>EU>RENT>HK); any they only connect it via Frankfurt.
- RETN also only peer with LUMEN(AS3356) in EU, the same reason, and RETN won't pass our communities to AS3356, if in EU, AS3356 is favored thru RETN; The LUMEN Hong Kong will also go to EU.
- same for some peers.
- GSL does not offer a regional prepend community.
It is likely that only some RETN customers are currently connected via RETN, most European targets will now be connected via GSL.
Due to this reason, DMIT NOC needs extra time to build anohter approches. Keep you posted.
DMIT has connect with GSL in HKG; due to GSL has more peers than RETN, most network are favor GSL rather than RETN.
We cannot force make any networks favor RETN in EU, because
- AS3257 is RETN's IP Transit; they only connect it at EU. This make any AS3356 customer where ever they at, go to EU. (Like APAC/US>EU>RENT>HK); any they only connect it via Frankfurt.
- RETN also only peer with LUMEN(AS3356) in EU, the same reason, and RETN won't pass our communities to AS3356, if in EU, AS3356 is favored thru RETN; The LUMEN Hong Kong will also go to EU.
- same for some peers.
- GSL does not offer a regional prepend community.
It is likely that only some RETN customers are currently connected via RETN, most European targets will now be connected via GSL.
Due to this reason, DMIT NOC needs extra time to build anohter approches. Keep you posted.
来自频道: @DMIT_INC
[Network Update] DMIT Now Receiving IP Transit from GSL (AS137409)
DMIT is pleased to confirm IP Transit from GSL (AS137409) is now available at all three of our core locations.
✅ Los Angeles is fully operational.
🔜 Tokyo and Hong Kong will be brought online soon.
Routing Overview:
Los Angeles:
- GSL serves as the primary upstream for the inbound traffic.
- AS3257, AS1299, and AS174 will be used for inbound backup, their customer routes, and outbound. (Selected by BGP)
- This maximize the Internet direction Anti-DDoS capacity for Pro and EB customer.
- T1's inbound will only go thru DMIT's AS3257, AS1299, and AS174 only at this time, but T1's outbound will have access to GSL
Tokyo & Hong Kong (APAC):
- GSL and RETN will operate in a dual-primary configuration with mutual failover.
- RETN remains the preferred path for Europe-bound traffic optimization.
- DMIT will try to make AS58453 primarily go thru GSL(via AS3356, LUMEN) for better connectivity; but DMIT does ***NOT*** give any guarantee for connectivity between T1 and China. Whatever the quality, performance or anything else.
====
DMIT also has on site mitigation capacity for any traffic that comes in from other ISPs. (400Gbps in LAX, 200G in HKG, 200G in Tokyo).
DMIT is pleased to confirm IP Transit from GSL (AS137409) is now available at all three of our core locations.
✅ Los Angeles is fully operational.
🔜 Tokyo and Hong Kong will be brought online soon.
Routing Overview:
Los Angeles:
- GSL serves as the primary upstream for the inbound traffic.
- AS3257, AS1299, and AS174 will be used for inbound backup, their customer routes, and outbound. (Selected by BGP)
- This maximize the Internet direction Anti-DDoS capacity for Pro and EB customer.
- T1's inbound will only go thru DMIT's AS3257, AS1299, and AS174 only at this time, but T1's outbound will have access to GSL
Tokyo & Hong Kong (APAC):
- GSL and RETN will operate in a dual-primary configuration with mutual failover.
- RETN remains the preferred path for Europe-bound traffic optimization.
- DMIT will try to make AS58453 primarily go thru GSL(via AS3356, LUMEN) for better connectivity; but DMIT does ***NOT*** give any guarantee for connectivity between T1 and China. Whatever the quality, performance or anything else.
====
DMIT also has on site mitigation capacity for any traffic that comes in from other ISPs. (400Gbps in LAX, 200G in HKG, 200G in Tokyo).
来自频道: @DMIT_INC
Partial complete;
No service impact right now.
June 3rd night finish the pre-patch work;
June 4th 7pm hot cut rest of fiber, and bring it into network.
No service impact right now.
June 3rd night finish the pre-patch work;
June 4th 7pm hot cut rest of fiber, and bring it into network.
来自频道: @DMIT_INC
Los Angeles Maintenance Emergency Notice
- Time: June 2, 2025, 7 p.m. Pacific Time
- Details: Deploy CoreSite Edge Router, move some IP Transit on it.
- Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted
- Duration: 3 hours
- Time: June 2, 2025, 7 p.m. Pacific Time
- Details: Deploy CoreSite Edge Router, move some IP Transit on it.
- Impact: Some IP Transit customer services will be down, AS58807 and AS9929 will be temporarily interrupted
- Duration: 3 hours
来自频道: @DMIT_INC
📢 [Incident Report] L1-135 Power Outage
On May 6, 2025, our upstream data center partner experienced two separate power-related incidents affecting Data Hall L1-135, where your VM(s) be hosted:
1️⃣ Primary Incident – Facility Power Bus Failure
A major arc flash occurred in a 4000A bus duct due to a downstream ground fault, causing a complete loss of utility and generator power. This led to downtime for both critical power and cooling. Power was restored at 5:33 PM, and UPS was fully recovered by 9:00 PM.
2️⃣ Secondary Issue – Faulty Customer PDU
A customer-provided PDU (not part of our equipment) caused intermittent ground faults, which tripped the UPS Static Bypass Module and delayed recovery. The PDU was removed on May 14.
⚠️ DMIT Position:
While we appreciate the transparency of the facility’s investigation, DMIT Cloud Service does not fully agree with the root cause attribution. A single faulty PDU should not be able to take down an entire UPS system, and we believe better electrical isolation should have been in place to localize such failures.
✅ DMIT Cloud Service Response & Improvements:
🔁 Network Resilience: We will relocate all Layer 3 network devices out of W7C, ensuring that IP transit and routing customers across other sites are not impacted by any single-site failure.
🧭 New Availability Zone: We are adding a secondary zone in Los Angeles, expected to come online by the end of 2025.
🔄 Dual-Zone Support: VM customers will soon be able to choose between availability zones or enable dual-zone deployment for improved redundancy.
🤝 Ongoing Coordination: We are working closely with the IDC provider to improve power infrastructure and failure containment.
We apologize for the disruption and thank you for your continued trust. All systems are fully restored and operating normally.
On May 6, 2025, our upstream data center partner experienced two separate power-related incidents affecting Data Hall L1-135, where your VM(s) be hosted:
1️⃣ Primary Incident – Facility Power Bus Failure
A major arc flash occurred in a 4000A bus duct due to a downstream ground fault, causing a complete loss of utility and generator power. This led to downtime for both critical power and cooling. Power was restored at 5:33 PM, and UPS was fully recovered by 9:00 PM.
2️⃣ Secondary Issue – Faulty Customer PDU
A customer-provided PDU (not part of our equipment) caused intermittent ground faults, which tripped the UPS Static Bypass Module and delayed recovery. The PDU was removed on May 14.
⚠️ DMIT Position:
While we appreciate the transparency of the facility’s investigation, DMIT Cloud Service does not fully agree with the root cause attribution. A single faulty PDU should not be able to take down an entire UPS system, and we believe better electrical isolation should have been in place to localize such failures.
✅ DMIT Cloud Service Response & Improvements:
🔁 Network Resilience: We will relocate all Layer 3 network devices out of W7C, ensuring that IP transit and routing customers across other sites are not impacted by any single-site failure.
🧭 New Availability Zone: We are adding a secondary zone in Los Angeles, expected to come online by the end of 2025.
🔄 Dual-Zone Support: VM customers will soon be able to choose between availability zones or enable dual-zone deployment for improved redundancy.
🤝 Ongoing Coordination: We are working closely with the IDC provider to improve power infrastructure and failure containment.
We apologize for the disruption and thank you for your continued trust. All systems are fully restored and operating normally.
来自频道: @DMIT_INC
DMIT observed there are packet loss on the NTT and RETN network;
There are submarine fiber cut in APAC, which caused capacity shrink in Asia.
We will keep monitoring; due to this is a fiber issue with our transit provider, there is limited option we can do.
来自频道: @DMIT_INC