All systems are operational

Past Incidents

31st March 2023

Ticket#10196883 |January 2023 to March 2023 | All Sites| Intermittent Wireless Authentication Issue, scheduled 2 years ago

• Event : Unplanned Wireless Network Outage - Remediated

• Schedule : January 2023 to March 2023 (1st Quarter) - Ongoing

• Activity Details : Our wireless authentication services in the AWS and TIM datacenters are experiencing intermittent problems with users connecting to the E and E-NET SSIDs.

Though the issue has been mitigated by installing a patch provided by the vendor, there have been reports of the issue recurring at random times of day with a limited scope.

Short-Term Remedy: While the two authentication servers are operational at this time, we have provisioned a third server in AWS for intrusive testing. Once we have completed testing and achieved the expected results, this will be used as the primary server. Production implementation is scheduled for late next week, February 3.

Damage Management IT will provide silver E-connect codes that will expire in 7 days to affected users. Please note that these codes are for single device use only and should only be used on company provided workstations.

Long Term Solution : IT Infrastructure team is developing an improved security solution and is in the middle of provisioning a new platform that will include the replacement of our current authentication service. ETA is by the end of 1st quarter.

Service Impact: All onsite users connecting wirelessly using E and E-NET SSID

8th February 2023

Ticket#10197981 |February 08, 2023| NTB and SFS |IPLC Connectivity Maintenance, scheduled 2 years ago

• Event : IPLC Connectivity Maintenance

• Schedule : February 08, 2023 (Wednesday) 6:59 AM to February 09, 2023 (Thursday) 1:00 PM (PHT)

• Activity Details : One of our internet service provider will be performing a corrective maintenance on one of their optical cable. During the maintenance window, an increase latency of 50ms switch hits plus loss of protection is expected.

• Service Impact : Latency Increase and Loss of Protection on NCP IPLC Connectivity

• Status: Completed

4th February 2023

Ticket#10196886 | February 4, 2023 | TIM Data Center | Data Center Rack and Equipment Cleaning, scheduled 2 years ago

• Event: Data Center Rack and Equipment Cleaning - TIM Carmona

• Schedule: February 4, 2023 - 4:00 PM (PHT) onwards

• Activity Details: One of our data centers will perform rack and equipment cleaning. This maintenance aims to keep the data center clean and prevent dust accumulation.

• Service Impact: None Service Impacting

• Status: Completed Successfully

30th January 2023

No incidents reported

29th January 2023

Unplanned Third-Party Provider Network Provider Outage - Alabang

• Event : Unplanned Third-Party Provider Network Provider Outage - Alabang

• Time of Incident: January 29, 2023 2:05AM (PH Time)

• Time of Restoration: January 29, 2023 4:08PM (PH Time)

• Details : Uplink to Data Center impacted by third-party provider outage.

• Service Impact: No network and internet connectivity onsite.

• Resolution: Waiting for updates from Third Party Provider.

Ticket#10197590 | January 29, 2023| PNB| Power Outage, scheduled 2 years ago

• Event : Power Outage at 11th Floor PNB Site

• Schedule: January 29, 2023 (Sunday) | 4:00 PM to 10:00 PM (PHT) - Cancelled

• Activity Details : Our facilities admin informed us that their contractor will be having a shutdown and lightnings at PNB 11th Floor which will last for 5 to 6 hours to complete their punch list at EE Room. During the maintenance window, We strongly advise to save all of your necessary and important files prior to the start of the maintenance to avoid any such data loss as there will be a total power outage at 11th Floor.

Service Impact: PNB 11th Floor Power Supply

• Event : Power Outage at 11th Floor PNB Site

• Schedule: January 29, 2023 (Sunday) | 4:00 PM to 10:00 PM (PHT)

• Activity Details : We received an update from our facilities admin that this maintenance has been cancelled due to a health emergency by their contractor person..

Service Impact: PNB 11th Floor Power Supply

28th January 2023

No incidents reported

27th January 2023

No incidents reported

26th January 2023

No incidents reported

25th January 2023

Ticket#10196976 | January 25, 2023 | Microsoft 365 Services Global Outage

• Event : Microsoft 365 Services Issue

• Unscheduled : January 25, 2023 (Wednesday) | 3:30 PM (PHT) - 8:43 PM (PHT)

• Activity Details: Please see below update details from Microsoft.

Title: Users may be unable to access multiple Microsoft 365 services.

Impacted Services: Microsoft 365 Suites

More information: We’ve received reports that the following services are impacted.

Microsoft Teams Exchange Online Outlook SharePoint Online OneDrive for Business Microsoft Graph

• Final Status: We've confirmed, after a period of monitoring, that the majority of impacted services have been recovered and remain stable.

• Service Impact: Any user serviced by the affected infrastructure may be unable to access multiple Microsoft 365 services.

• Preliminary root cause: A wide-area networking (WAN) routing change resulted in users being unable to access multiple Microsoft 365 services.

24th January 2023

No incidents reported

23rd January 2023

Ticket#10192091 | January 24-25, 2023 | All Sites | Internet Service Provider Network Maintenance, scheduled 2 years ago

• Event : Planned 3rd Party Maintenance - Scheduled

• Schedule: January 24, 2023 (Tuesday) – January 25, 2023 (Wednesday) | 8:00 PM – 5:00 AM (PHT) - Completed

• Activity Details :We have received a confirmation from our internet service provider that this maintenance activity has been successfully completed. No issues encountered and the circuit internet uplink is stable and operational as usual after the completion of the maintenance.

• Summary Result : Successfully Completed

• Event : Planned 3rd Party Maintenance - Scheduled

• Schedule: January 24,2023 to January 25,2023

• Activity Details : Rehoming of circuit to an upgraded device for service improvement.

• Service Impact: Partial to no Impact on the Network.

22nd January 2023

No incidents reported

21st January 2023

No incidents reported

19th January 2023

Ticket#10195631|January 29, 2023| TIM Data Center| Monthly Load Testing of Generator Set, scheduled 2 years ago

Event : Data Center Monthly Load Testing of Generator Sets - TIM Data Center - Cavite

Schedule : January 29, 2023 (Sunday) | 8:40 AM to 5:25 PM (PHT)

Activity Details : One of our data centers will be performing their monthly load testing for their generator sets. The purpose of this activity testing is to meet the National Fire Protection Association (NFPA) standards for security and reliability of their power generators. There will be no impact during the maintenance window.

Service Impact: Non-Impacting

Update: Monthly load testing has been successfully completed. No issues monitored during the said test.