Data Migration is a very important topic where one could write a whole book but in this blog post, I will highlight key considerations of a data migration project.
Data Management Policy and Guidelines
With ever expanding data, comes greater need to establish a Data Management policy. Doing this immediately will deliver dividends during a Data Migration initiative.
If data isn’t properly maintained in your existing system, then you either migrate bad data into a new system or you will have to include a data cleansing exercise as an interim task during the data migration process. Therefore, it is prudent to establish a data management policy at the onset.
Data management policy guidelines should include the following:
- Type of data to be stored.
- Objects
- Field Types
- Transactions
- Communications
- Client Data
- Change Logs
- Data quality metrics
- % Data Complete
- Record age
- # Records with old data in fields. (Old sales reps on accounts, Products no longer supported)
- Periodic data cleansing and removal schedule,
- Data security and access control,
- Data Breach Response Plan,
- Data storage locations,
- Data volume reporting,
- Data retention policy
Data Migration and ETL – What is it?
ETL stands for Extraction, Transformation and Loading. ETL is an old acronym but what may not be common knowledge is the number of cycles and the time it may take to perform the ETL exercise. When you extract data from your existing system; the data validation phase in the new system may re-trigger additional ETL cycles. The number of cycles can be unexpected especially if the original data has not been cleaned up regularly.
Each phase of ETL can go through many cycles with the “Transformation” cycle being the most dominant. During this cycle, meetings may take place with business users to determine what data should be kept and discarded, discussion of data validation rules, duplicates identification rules, with the end goal of having the right data available for users at go-live.
Therefore, it’s best to have a dedicated and experienced Data specialist who is familiar with the project data migration tools and the business data requirements. By having a Data specialist on your project, there is a high probability that things will be done correctly.
Major Considerations during Data Migration
- Start Early: Unexpected surprises may loom up as you begin to dive deeper into the data migration process. Build plenty of buffers into the project plan.
- Backup: Back up your existing data in a neutral format and location. Because the data will be transformed and deduplicated during the migration, there may be a risk that certain edge cases and logic errors may not be identified until much later.
- Data Migration Tool/Application: After an initial exercise to determine the volume of data for each object that needs to be migrated, selecting the right Data Migration or ETL tool and understanding its strengths and weaknesses is important to ensure the tool could meet and/or exceed expectations.
- Data Specialist: Having a dedicated and experienced resource is an important consideration, someone who could entirely focus on the data migration initiative as business users will be busy in conducting day-to-day business and may not be able to devote focused effort on the data migration initiative.
- Planning: Plan and strategize if migration is going to be a big bang approach or trickled data migration, what objects would you start with, how would you track and report the progress of data migration, how would you execute the plan and finally do post migration data validations.
Conclusion
The reasons for data migration could be more than one; it could be the limitations of current systems to meet and exceed users’ expanding needs due to increasing data volume year-over-year, sluggish application performance, addition of new applications, and lack of automated data integrity controls, and improved security to avoid data breaches. However, in all cases, there is no doubt that before embarking on a complex data migration implementation journey, it should be thoroughly planned.
In addition to transforming your legacy technologies with the state of the art Cloud based NetSuite ERP SuiteSuccess methodology, collaborating with a partner like Plative can help make your data migration initiatives go more smoothly and ensure a seamless go-live.
By Aleem Khan
Senior Project Manager at Plative
