Many things have changed during my 12 years of delivering content migration projects with Vamosa and T-Systems. From the early days of migrating unstructured websites into simple content management systems through to the complex enterprise cloud migrations that we see today. There has been a constant evolution of our tool-set and processes to adapt to rapidly growing volumes, increasing complexity and higher customer expectations.
With the introduction of SaaS models running on cloud platforms the challenges are even greater. This became apparent on a project we have just completed for the biggest networking company in the world – which was also one of the largest content migration projects that we have delivered. More than 6 million content objects, almost 10TB of data, were migrated in a project running for just over 12 months. The scale and complexity of this migration tested the limits of both the outgoing source Jive system and the new IBM Connections Cloud target.
The flexibility of the Vamosa Migration Architect software means that it can easily be scaled up to take advantage of multiple servers and this allows us to process and load data at the maximum speed the target system can handle. In this case we actually had to throttle the load at some points to allow the target cloud environment to catch up. In a shared cloud platform anything which could potentially impact the performance of the service for end users or other customers is a major concern, so being able to control the migration speed dynamically becomes very important.
This was also one of our first cloud-to-cloud migrations which did highlight some other challenges which need to be considered when working with cloud platforms. The environments are subject to regular maintenance patches which can impact the migration process. Advanced warning is only given for larger updates or potential down-time with maintenance patching expected to be transparent to end users. Despite this there were instances on this project where changes to both the source and target environments mid-migration caused issues. Our migration process uses the source and target APIs and occasionally these can be changed without warning causing either the content capture or load to fail. Unfortunately there’s no easy way to avoid this situation. Even coding defensively and building very flexible processes can still result in a situation where the migration must stop until a change is made to support the modified API.
The roadmap for Vamosa Migration Architect takes a slightly different architectural approach to dealing with scenarios where there is either an issue with the availability of source/target servers, or potentially an API change which causes a migration process to fail. Rather than the migration process running sequentially and stopping completely whenever there is an issue, the plan in the future is to move to an event source driven architecture where unpredictability becomes less of a problem and dependencies between components are reduced. The major benefit from a migration delivery point of view will be that processes can run independently and in parallel, so if a situation arises where a process is blocked it will simply wait until the issue is resolved and it’s dependencies met while other, un-blocked processes can continue as normal.
So, we continue to evolve and look forward to even bigger and more complex projects with software that anticipates change and accepts that goalposts will never really stay still. The new architecture is an exciting development and is worthy of a blog or two on its own but I’ll leave that to Adrian who is actually building it and can explain in much more detail than I can!