By Charles McIntyre. Early in 2017, UC Santa Cruz went live with its first enterprise IT service in the cloud, the Identity Management (IDM) system. This was followed in rapid succession by other migrations to the Amazon Web Services (AWS) cloud: PeopleSoft Campus Solutions in October 2017, Banner Advancement 9 in May 2018, and the Data Warehouse and the Operational Data Store (ODS) for UCPath in June 2018. Banner Finance, the last system, will migrate to AWS by the end of the year.
When PeopleSoft Campus Solutions went live in AWS in October 2017, UCSC was the third university in the United States to do so.
These migrations were successful because trust and collaborative models had been intentionally cultivated between the systems and security teams, database architects, application experts, architects, developers, and business analysts – and, of course, because of excellent project management.
This could not have happened just five years ago. At that time, for various reasons, there was a deep divide between application and system engineers. There also were system outages, delays in building systems, and little innovation or creativity.
A History of team and trust building
This is the story of how UCSC turned that culture around and made it possible to open the door to massive operational change.
In 2012, the enterprise application group recognized it needed to increase quality of service and delivery speed. It also realized that collaboration and improved performance would not be achievable unless better relationships among the systems, networking, and security teams were built.
To that end, the managers of the application infrastructure teams and data center teams spent years nurturing relationships, building trust, and encouraging collaboration between the security, network, application, database, and systems groups.
Trainings in healthy conflict management were scheduled and well-attended. Informal technical trainings across teams shared knowledge and built mutual respect. Postmortems were done in a blameless fashion. Relationships were improved and honored. Strategy and disaster recovery “meetings” were held at local breweries. All-day offsite workshops were designed and delivered to further increase collaboration, innovation and engagement (see feature photo of the ITS Enterprise Operations group in 2015).
By 2015, the level of trust had reached an all-time high. Incidents were rare. When they did happen, engineers immediately provided expertise and support to resolve the issues together. An ethic of operations excellence was promoted within the teams, which also contributed to the reduction in fire-fighting.
Hardware refresh strategy
It was also time to start looking at replacing the Oracle servers that provided compute services for the campus’ most critical enterprise IT services. The servers were reaching end of life and vendor support quality was declining. We needed to explore options, even though any new environment would be quite a leap, technically and organizationally.
A team of senior engineers, architects, and managers determined that realistic options were migration to:
- The on-premise virtual infrastructure,
- A vendor-managed infrastructure from Ellucian, Oracle and/or Dell, and
- Infrastructure as a Service vendors: Amazon Web Services, Google Cloud, and/or Azure.
At the time, six out of the seven senior engineers were convinced we would migrate to the on-premise virtual infrastructure.
After all, decades of infrastructure customization and optimization made migrating to vendor-managed services cost-prohibitive: Vendor-managed infrastructure generally involves no more than three to four non-production environments, while we had an average of nine per enterprise service.
While the on-premise virtual infrastructure was a viable candidate, it lacked a native and robust disaster recovery environment. When we began researching Amazon Web Services, the team was skeptical of migrating complex legacy ERPs like PeopleSoft, Banner, and Business Objects, but gave it a fair review.
AWS provided a seasoned and helpful solutions architect to work with our team, and our senior PeopleSoft engineer was able to build a working proof of concept (POC) in AWS within two weeks. Considering the risks and benefits of the viable options, the strategy team recommended that we migrate our enterprise systems to AWS.
Subsequently, in August 2016, the ITS senior management team concurred that AWS was to be the future environment for UCSC’s enterprise applications. Due to the high degrees of trust and positive organizational culture, there was widespread confidence in the ability of the engineering teams to work together to pull off such a massive migration and change.
Migration to AWS
The ITS senior management team decided to scope this project as an infrastructure migration, with minimal changes to the various systems’ architecture. The cloud strategy would start with a small bespoke web application, then the IDM service, which was a mix of PHP, LDAP and Shibboleth technologies. If those systems could be migrated, it would be determined whether the larger vendor-based enterprise systems would go to AWS as well.
The application group brought in an accomplished project manager with experience in AWS migrations who worked collaboratively with the knowledgeable and adept infrastructure manager. Together they were highly detail-oriented, focused, and able to drive multiple complex and critical projects at the same time. The project management team was given deep and wide support to complete the infrastructure migration before support expired on the Oracle servers.
The project management team focused on the overall strategy to ‘lift and shift’ then optimize the architecture. The primary goal of the project was to migrate the enterprise systems to AWS, only after that work was complete would the team focus on cloud optimizations. This allowed the engineering team to establish a comprehensive understanding of infrastructure in the cloud before re-architecting enterprise systems to recognize cloud efficiencies.
Success, opportunities, and risks
The enterprise migrations to AWS have been very successful. The teams working on the projects are excited to be learning and involved with cutting-edge technology, like cloud design patterns, Git and Chef. The success of the migrations has built a sense of pride.
Scoping the project as purely an infrastructure migration had advantages and disadvantages. This approach bypassed the vexing and political discussions around changing how developers or the business utilize the environment.
A disadvantage is that initial runtime costs can be considerably more than a future cloud optimized architecture. While the project provided the business with real value in improved disaster-recovery capability, as well as an end to perennial time-consuming hardware-refresh projects, it was not scoped to realize the benefits of native cloud architecture.
Another significant risk of this approach is the technical and operational aftermath. Architectural compromises were accepted with the common understanding that they would be dealt with during a post-migration optimization phase.
And then there is organizational debt from any long and intensive project. There have been many months of long hours and strict deadlines. Traditional roles have become more ambiguous. We’ve been focusing on moving mountains for so long. Now we need to reinspire motivation and engagement to take on the next steps of delivering efficiency and business value.
We are looking at next steps. First, we are prioritizing the optimization of our enterprise architecture in AWS. Our biggest priorities are to reduce runtime costs and enable faster developer cycle time. We intend to do that by investing in developer pipeline technology and re-architecting systems in more native AWS design patterns. The development teams and business partners will need to engage together more than ever to realize the true potential value of cloud infrastructure, agile development, and a DevOps approach.
A little further into the future, a more comprehensive organizational strategy will need to be developed to cultivate staff with the curiosity to jump into leading-edge technology, and to strengthen cultural norms that will enable potential DevOps, Agile, Lean and digital transformations. Based on our success to date moving to the cloud, we have the confidence that we can fulfill that promise.
Charles McIntyre is manager, Infrastructure & Operations, in Application & Project Management, Information Technology Services, UC Santa Cruz.