By John Ruzicka. Last July, UC’s largest-ever business transformation initiative concluded, with all UC locations now live in a single payroll system called UCPath. This was a multi-year project involving teams at the UC Office of the President (UCOP) and all of the locations. It included creation of the UCPath Center in Riverside, which manages UCPath day-to-day operations.
UC did not host the UCPath software itself. Instead, UCOP Information Technology Services (ITS) and the UCPath Center engaged the services of a managed service provider. The provider physically hosted and ran the software. The team coordinated with the provider on things such as pushing out new updates, creating additional environments when needed, and backing up the entire system in case of disaster.
By 2020, it became apparent that the managed service provider for UCPath, while offering some benefits, was not meeting UC’s needs. Processes that needed to run overnight were not being completed in time and we weren’t receiving an acceptable level of customer service. This cost time and money and, most importantly, caused pain for UCPath customers on the development side and sometimes even for end-users (i.e., UC employees).
Additionally, the service provider notified the team that they would have to move to the provider’s new platform. In some ways, this would have been an easier route, but it would not have resolved the ongoing support issues. After consultation with the UCPath Governance Committee, the team decided to move UCPath to Amazon Web Services (AWS), with the assistance of a partner and with a plan to ultimately manage UCPath ourselves.
Moving to AWS
UCPath is a complex and critical software system and there were two choices, both of which had a very short timeline: By December 2020, we had to be either on the provider’s new platform or completely off it. Moving UCPath to AWS in such a short period of time required significant coordination, communication, and teamwork between UCOP and all of the UC locations.
Our architecture consisted of clustered environments, which ran both an operating system and the database software necessary for UCPath. AWS offered a cloud-hosted relational database service (RDS). Hundreds of scripts had to be refactored, with some AWS services configured to mimic an operating system. At the same time, we had to upgrade our database from major version 12 to version 19. This meant periods of freezes—times when no changes could be made to the system. Freezes had to be carefully coordinated around schedules so as not to affect payroll.
We also moved from an old-school service-oriented architecture design to a modern, Java-based enterprise service bus. This allowed for much faster data exchange and greatly reduced runtime of various processes. It also involved a lot of transitional work.
To meet the tight deadline and lighten the testing effort at each location, the team decided to divide the work between locations and the central functional team, rather than having each of the 18 locations perform the same testing. As a result, locations were able to help each other overcome problems and issues both with and often without UCOP’s help. By the end of testing, all teams had successfully worked together to close out all defects and issues!
The timeline was tight, and there were moments of doubt, but we used a system approach that had been used at other sites and applied sound architectural and design principles. The team knew UCPath inside and out, having worked with it for years, but AWS was newer to us. Working with the provider, our team not only met the deadline, but also managed to make major improvements in UCPath.
Benefits from this project include:
- Improved reliability and better performance for all locations
- Quicker turnaround for upgrades and system improvements
- Cost savings of about $2 million per year with an additional $2 million in savings in FY23, once we move to a fully self-managed cloud
- Significantly reduced payroll processing, such as reducing the general ledger process by 10 hours
- Greatly improved maintenance and operations support, including enhanced security
- Ability to provision a new environment in three to five days vs. 30+ days
- Ability to refresh environments in eight hours instead of 48
- Ability to quickly terminate and, if necessary, easily resurrect underutilized environments
- A transparent infrastructure that lets us more quickly resolve issues
- High availability and instant disaster recovery
In January 2022, we will start work on a six-month transition from the service provider so that UC will be completely responsible for ongoing AWS hosting support of UCPath.
The UCPath to AWS project expanded the cooperation, communication, and coordination between UCOP and all of the locations we serve. Those relationships and lessons learned will continue to help us with future initiatives, such as the Reporting Instance (RI) and the UCPath Upgrade projects. Reporting Instance will allow locations to pull real-time reports at any time, and the UCPath Upgrade project will continue to improve UCPath and make it faster and easier to use.
A big “thank you!” to everyone involved in making this project a success. If you have any questions about the project, please contact Director of Infrastructure Kari Robertson.