Networking Incident Impacting SA Systems and Services
Resolved
After monitoring systems for several hours, Engineers are now confident that the issue is fully resolved. The root cause was due to network equipment failure on a fiber connection. If you continue to experience any issues, please contact the Help Desk at 805-893-7962 or at support@sa.ucsb.edu.
Posted Jan 16, 2020 - 15:51 PST
Monitoring
A fiber connection was identified to be causing network instability. The affected connection was disconnected from the network and multiple systems immediately became more responsive. Engineers will continue to monitor the environment for several hours to ensure the issue is resolved.
Posted Jan 16, 2020 - 09:43 PST
Update
Engineers are continuing to investigate. Microsoft Premier support was engaged at approximately 5:09PM, and coordination with Premier support continued throughout the night and into the morning. At around 2:22AM Microsoft recommended further investigation on network infrastructure as logs and network captures indicated many packet RETRANSMITS and ACK RESETs. SIS&T Network Engineers continue to be engaged and are continuing to investigate.
Posted Jan 16, 2020 - 07:32 PST
Investigating
Multiple colleagues are investigating reports of timeouts and slowness in SIS&T supported applications and services. Additionally, several reports have come in of slowness and timeouts for non-SIS&T supported services, including access to Google Connect.
Engineers believe that at approximately 11:37AM issues began, and were able to determine that a DNS server within the SA network was running at high CPU load. Steps were taken to failover to a secondary DNS server and add additional hardware resources to the original that had been experiencing high CPU load. At present, even though resource utilization of these network resources appears to be stable, colleagues are continuing to receive reports of general network slowness.
Posted Jan 15, 2020 - 15:03 PST
This incident affected: Atlassian Suite (Atlassian JIRA Cloud, Atlassian Confluence Cloud, Atlassian Developer Confluence Cloud APIs), Box (Box Mobile, Box Website, Box Sync), Zoom (Zoom Group Messaging, Zoom Meetings, Zoom Telephone Services), Slack (Slack Messaging, Slack Posts/Files, Slack Login/SSO, Slack Apps/Integrations, Slack Workspace/Org Administration), Google Connect Suite (Google Apps Gmail, Google Apps Hangouts, Google Apps Calendar, Google Apps Docs, Google Apps Drive), Disabled Students Program (DSP Services, DSP Website), PNC (Point N Click Application, Point N Click Database, Point N Click Broker, Point N Click iOS Web Services, Point N Click Open Communicator), Office of the Registrar (Registrar Student Service, DeltaSys Web Services), Admissions (Admissions Staff Site, Inside Admissions, Applicant Authentication, Online SIR, Admissions Website), Authentication Services (On-Premise Authentication Services, Cloud Authentication Services), BLUE (SREG, STAR, CRES, IADM.Broker, ADMB.Broker, UADM.Online, ADMS.Online, CTAB.Online, INDX.Online, STAB.Online), Graduate Division (UCSB Online Graduate Application, Graduate Application Review, GradNET, Graduate Financial Competitions, Graduate Student Fee Remission, GradDiv Website, GradPost Website, Graduate Student Association Website), Academic Initiatives (Academic Initiatives Website), Alcohol and Drug Program (ADP Website, CASE), Counseling and Psychological Services (CAPS Programs Reservation Application, CAPS Website), Student Health Service (SaySomething, Student Health Website), Career Services (Career Services Website, Handshake), Children's Center (Childrens Center Website, NoHo Software Care), Campus Learning Assistance Services (CLAS Website), Early Academic Outreach Program (EAOP Website), Economics Department (Economics Website), Office of the Vice Chancellor of Student Affairs (Emergency Resources Website, Getman/Villa Awards Website, Giving Website, Professional Development Website, Red Folder Website, SABOG Website, SFAC Website), Educational Outreach Program (EOP Advising Application, EOP Website, Dream Scholars Website), Basic Needs Resources (Basic Needs Resources Website), Student Conduct (Judicial Affairs Website, Advocate), MultiCultural Center (MCC Website), International Students and Scholars (OISS Website, International Student and Scholar Management (ISSM), International Scholar Dossier (ISD)), Orientation Programs and Parent Services (Orientation Website, Gaucho Parents Website), Student Engagement & Leadership (OSL Website, Campus Groups), Women, Gender, and Sexual Equity (WGSE Website, Osniums WS), Resource Center for Sexual and Gender Diversity (RCSGD Website), Recreation (Recreation Website, Fusion), Veteran and Military Services (Veterans Support Website), Health and Wellness (Wellness Website), Arts and Lectures (Arts and Lectures Website, Arts and Lectures Ticketing Site, Tessitura/RAMP), Jira/Confluence (Jira Core Viewing content, Jira Core Administration, Jira Core Create and edit, Jira Core Authentication and User Management, Jira Core Mobile, Jira Core Search, Atlassian Developer Jira Cloud APIs, Confluence Administration, Confluence Authentication and User Management, Confluence Comments, Confluence Create and Edit, Confluence Android App, Confluence iOS App, Confluence Search, Confluence View Content), GOLD, eGrades, AAA, AppStatus, ProSAM, HIPAA website, ServicePro, and Email Services (SA Email Server Queue).