Posted by Brian Tharp, program manager, UCOP. UCOP moved its mainframe work load over four weekends between September 5 and October 11, 2015, with 70% of the applications moving on October 11. The movements were practically flawless, due to the impeccable teamwork of more than 200 UC colleagues.
Customers were pleased with project execution. Nathan Affleck, system operations manager for UCD Accounting & Financial Services, said, “Thank you very much for the extra attention over the last couple weeks. Our preparations paid off over here–we have had zero problems with our production feeds and only a couple reported issues from campus users.”
The reason for the move
The UCOP data center lease is set to expire mid-2016, offering UCOP an opportunity to save costs by terminating the lease, as well as create a more modern, agile way to deliver technology – primarily via cloud or campus service providers. Moving our mainframe services is just one of the numerous technical projects required for us to vacate this aging facility.
I was hired as program manager in October 2014, and the program was fully funded in early February 2015. We had just fourteen months to execute.
UCOP’s mainframe service supported nine campus payrolls with $13B in annual payments; the UCB, UCSF, and UCSC campus mainframe-based applications; as well as a number of UC services, including AYSO (open enrollment) and the UC Retirement System. A total of 62 mainframe-based applications, involving 139 interfaces and subsidiary systems sustained by ten unique UC and UCOP business units had to be moved.
How fast did it happen?
The mainframe deadline had to be pushed to October 11 because of the need for a freeze on changes affecting AYSO, the system UC employees use in November to enroll in benefits. That left Ray Vierregger, UCOP mainframe manager, and all supporting UC business units just 176 business days to purchase, plan, test, and move the mainframe. Testing alone involved over 2,000 tasks.
In addition, the network security model at the new mainframe site needed to be upgraded. Over 5,500 network firewall rules were added to support the mainframe at its new home and campus partners had to update all their corresponding rules as well.
During the project design phase, team members identified an opportunity to collocate mainframe production hardware at UCSD and mainframe disaster recovery hardware at UCLA. As a result, a robust partnership has developed among the UCOP, UCSD and UCLA mainframe teams, opening up opportunities for future collaboration, potential cost savings, and additional operational benefits over time. The three teams meet weekly and are planning future technology assimilations. UCLA is already planning to leverage the new UCOP/UCSD Disaster Recovery platform early 2016.
The mainframe project schedule was a “mission impossible” from the day the project was funded. None of the UC campuses requested this change and all had their own internal business drivers and needs to meet. But we listened to what each UC team member or customer needed from us to help accomplish our goal together. The project plan was obsolete after the first test period, as few plans this complex survive execution, but every team member understood the criticality of the goal and adapted schedules to test findings. I would like to acknowledge the extraordinary work of all our UC team members. This project demonstrates what UC can accomplish when we all work together.