Past Incidents

• Event : Planned Internet Service Provider Network Maintenance – All Sites

• Schedule : August 10, 2024 (Saturday) | 10:00 PM to August 11, 2024 (Sunday) | 1:00 AM (PH Time Zone)

• Activity Details : One of our service providers will be performing a migration as part of their continuous improvement to their facility and to avoid future outages that may be encountered.

During their maintenance activity, a maximum of fifteen (15) minutes downtime and intermittency on one of our internet uplink circuits going to our data center is expected before it will fully failover to the alternative path that serves as a backup route.

Please feel free to contact us if you have additional questions or clarifications.

• Service Impact : Intermittency on Network and Internet Connectivity for both Wired and Wireless Connection – All Sites

• Event : Migration of Services to New Platform

• Schedule : July 28, 2024 (Sunday) | 4:00 AM to 9:00 AM (PH Time Zone)

• Activity Details : Hi Team,

One of our data center's internet uplink providers will be conducting a migration of services to a new platform to enhance their service availability and internet quality.

During the maintenance, the affected internet circuit will be unavailable however due to our way the network is setup, a partial impact on network and internet connectivity is expected as we have another available internet uplink that serve as a backup.

• Service Impact : Partial Impact on Internet and Network Connectivity – ALL SITES

• Event : Active Directory subnet level traffic isolation testing on JMT Site.

• Schedule : July 27, 2024 (Saturday) | 6:30 PM (PH Time)

• Activity Details : Dear Team,

This is to formally inform you that the Systems Engineering and Administration team will be conducting Active Directory Subnet level traffic isolation testing on the JMT Site.

• Service Impact : No impact as this is tested on test subnet in JMT Site.

• Event : Partial Network Outage

• Incident Date/Time : July 26, 2024 (Friday) | 4:45 AM (PHT) - 5:55 AM (PHT)

• Activity Details : Hi Team,

Please be informed that the affected circuit has recovered.

All circuits are now fully operational with redundancy. We are closely monitoring for stability.

• Service Impact : Partial Impact on Internet and Network Connectivity – All Sites

• Event : Partial Network Outage

• Incident Date/Time : July 26, 2024, 2024 (Friday) | 4:45 AM - ongoing

• Activity Details : Hi Team,

Please be informed that one of our Datacenter Internet Uplink is currently experiencing an outage and we are currently coordinating with our circuit provider to more about the situation.

Due to the configuration of our network having backup circuits, we expect only partial impact on internet and network connectivity. This may manifest as slow application performance, while remote desktop sessions and VOIP calls may experience intermittent disconnects within a 4-5 minute timeframe while automatic failovers/failback activate.

• Service Impact : Partial Impact on Internet and Network Connectivity – All Sites

• Event: Unplanned Partial Network and Internet Interruption at IBM Building

• Incident: Instance 1: July 24, 2024 (Wednesday) | 4:28 PM – 11:25 PM (PHT Time Zone) Instance 2: July 25, 2024 (Thursday) | 3:38 AM – 10:13 AM (PHT Time Zone)

• Activity Details: Hi Team,

We wish to notify you that the circuit has recovered around 10:13 AM PHT and is now being monitored for stability. Our service provider has verified the successful replacement of the UPS and the connection should now be fully operational.

• Service Impact: Partial Network and Internet Connectivity Interruption for both Wired and Wireless Connection – IBM Site

• Event: Unplanned Partial Network and Internet Interruption at IBM Building

• Incident: Instance 1: July 24, 2024 (Wednesday) | 4:28 PM – 11:25 PM (PHT Time Zone) Instance 2: July 25, 2024 (Thursday) | 3:38 AM – ongoing (PHT Time Zone)

• Activity Details: Hi Team,

We've noticed the circuit has gone down again at 3:38 AM. We are in contact with our service provider for further investigation.

During the incident, only partial impact on network and internet connectivity is expected as we have another site circuit uplink going to other data center connection is available for redundancy.

• Service Impact: Partial Network and Internet Connectivity Interruption for both Wired and Wireless Connection – IBM Site

• Event : Unplanned Partial Network and Internet Interruption at IBM Building

• Incident : July 24, 2024 (Wednesday) | 4:28 PM – Ongoing (PHT Time Zone)

• Activity Details : Hi Team,

We would like to inform you that the circuit was restored at exactly 11:25 PM and has been stable for 30 minutes.

• Service Impact : Partial Network and Internet Connectivity Interruption for both Wired and Wireless Connection – IBM Site

• Event : Unplanned Partial Network and Internet Interruption at IBM Building

• Incident : July 24, 2024 (Wednesday) | 4:28 PM – Ongoing (PHT Time Zone)

• Activity Details : Per our service provider they have detected a loss of signal (LOS) on their end and already informed their field engineers to further check the cause of the issue. No ETR yet provided.

• Service Impact : Partial Network and Internet Connectivity Interruption for both Wired and Wireless Connection – IBM Site

• Event : Unplanned Partial Network and Internet Interruption at IBM Building

• Incident : July 24, 2024 (Wednesday) | 4:28 PM – Ongoing (PHT Time Zone)

• Activity Details : Hi Team,

We have observed that one of our site circuit uplinks going to one of our data centers is currently down. We already coordinating this with our service provider for further investigation.

During the incident, only partial impact on network and internet connectivity is expected as we have another site circuit uplink going to other data center connection is available for redundancy.

Please feel free to reach out to us if you have additional questions or clarifications.

• Service Impact : Partial Network and Internet Connectivity Interruption for both Wired and Wireless Connection – IBM Site

• Event : Power Service Interruption at NTB

• Incident Date/Time : July 23, 2024 (Tuesday) | 10:10 PM – July 24, 2024 (Wednesday) | 12:10 AM (PH-Time)

• Activity Details : Hi Team,

At 12:10 AM earlier, normal power was restored to the power line. Upon verification, network devices and internet connectivity are functioning normally.

• Service Impact :NTB Building Main Power Supply

• Event : Power Service Interruption at NTB

• Incident Date/Time : July 23, 2024 (Tuesday) | 10:10 PM - Ongoing

• Activity Details : Hi Team,

We are currently experiencing unscheduled power interruption at the Neutrinos Building Baguio Site.

As of 9:03PM, the building’s electricity is being suppled by a generator set. We will continue to monitor the situation and provide updates, as necessary.

There is no expected impact on the network and internet connectivity due to the availability of generator set and Uninterruptible Power Supplies (UPS) serving as backup power for our network devices. However, devices without battery support, such as desktops and other equipment, may experience fluctuations.

• Service Impact :NTB Building Main Power Supply

• Event : Facilities and Building Admin Generator maintenance and testing

• Schedule : July 20, 2024 (Saturday) | 8:00 AM to 11:00 AM (PHT)

• Activity Details : Hi Team,

The Building Administration at our Baguio site has made a request for an additional time for load retesting of the 250KVA generator on Saturday, July 20, 2024, Between 8AM- 11AM.

During the maintenance activity, please expect less than a minute shifting from main power to generator and vice versa.

• Service Impact : Building Power Supply – NTB Site

• Event : Partial Outage on Network and Internet Connectivity - NTB

• Schedule : July 18, 2024 (Thursday) | 10:46 AM – On-going (PH-Time)

• UPDATE 1 : We have observed that one of our Site to Datacenter Circuit is down at our NTB site and have coordinated this with our service provider for further checking.

Due to the way our network is setup, only partial impact on internet and network connectivity is expected as we have another internet circuit uplink available that serve as a backup.

• UPDATE 2: We have received an update from our provider. Per their initial checking, the circuit is affected by a network fault and is currently being handled by their fiber restoration team for rectification and validation. They have provided an ETR of 10 PM.

• Service Impact : Partial Impact on Internet and Network Connectivity – NTB

• Event : Third-party provider Essential Maintenance

• Schedule : July 17, 2024 (Wednesday) | 7:00 AM to 1:00 PM (PHT) - COMPLETED

• Activity Details : Our service provider has successfully completed this maintenance activity.

• Service Impact : Loss of Service on IPLC Connectivity

• Event : Third-party provider Essential Maintenance

• Schedule : July 17, 2024 (Wednesday) | 7:00 AM to 1:00 PM (PHT)

• Activity Details : Our service provider will be re-routing the traffic of the circuit due to cable fault.

• Service Impact : Loss of Service on IPLC Connectivity