This article is relevant if you’re looking to migrate files from local storage, the NetSuite File Cabinet, or from one cloud storage repository to another. We expect the requirements to retain associated joins with all associated NetSuite records.
Tips for Implementing a NetSuite Data Migration Strategy
File migration to cloud storage allows organizations to simplify and scale their workflows, free up local storage, and increase opportunities for collaboration. Many of our clients want to move files out of their file cabinet or local network drives, or from one cloud storage platform to another. Organizations prefer cloud-based options for a range of reasons, like coming live on NetSuite from an on-premises solution or simply moving to a more cost-effective and versatile object storage environment.
When developing a NetSuite data migration strategy, much of the process comes down to choosing the proper tools. This article will cover how to perform a NetSuite data migration, including how to migrate files from local storage, the NetSuite File Cabinet, or from one cloud storage repository to another.
Multi-Repository Migration Use Cases
A proper data migration requires special care and consideration to ensure all files transfer appropriately and without error. Fortunately, we have experience in conducting multiple NetSuite data migration processes. Below are some of the file migration projects we performed for clients in conjunction with our eXtendFiles solution, demonstrating four different methods for NetSuite data migration.
1. OneDrive to S3
Migration from directory based to non-object storage
Our client required migration from OneDrive to AWS S3 as they wanted to migrate from a directory-based option to non-object storage. The client was already running eXtendFiles for about two (2) years. The client wanted to switch to S3 to increase file availability and gain more control on accessibility.
To do this, the client had to authenticate using OAuth and provide S3 IAM user access on the required S3 bucket where files needed to land. We had to ensure the eXtendFiles records in NetSuite were updated with the S3 links upon migration. Once we did, we moved all the files over to S3, replaced all links and then deprecated the OneDrive storage.
2. Box.com to S3
10.8 million files / 16.3 TB of data = Annual Savings of over $110,000 per year!
While the Box.com integration for NetSuite has its advantages, user costs can escalate quickly. For clients leveraging existing Box.com for NetSuite integrations, we had to ensure files are migrated to S3 while retaining all the versions of the Box.com file. We also ensured that each file was linked to the correct NetSuite record.
One of our larger-scale projects had more than 10.8 million files, totaling 16.3 Terabytes of data. This migration replaced the extensive history of linked files and took almost a month of migration processing time to reconnect all the records and rebuild. However, by the end, the Box file migration yielded annual savings for our client that exceeded a stunning $110,000 per year.
3. Box.com to OneDrive
Directory based migration, consolidated operations in Office 365
This same client wanted to move their employees’ personal files from Box.com to OneDrive while retaining the same folder structure as created in the initial platform. Migrating to OneDrive allowed them to consolidate their operations in Office 365 and support collaboration. Users were granted access using a simple OAuth 2.0 screen (similar to the authentication process we have in eXtendFIles for adding a cloud storage repository).
Once authenticated, we were able to migrate everyone’s existing structure they had originally built in Box into the OneDrive environment. The client is now running S3 for most of their high velocity, externally facing files while using OneDrive for employee collaboration.
4. Migrate From Google Drive to S3
Image repository / CDN requirement – migrate and upgrade
In this endeavor, the client was running Google Drive, which was also acting as a CDN (content delivery network) for their item images. Unfortunately, Google Drive isn’t a great foundation for CDN-like functionality, and the links had expired over time. Therefore, the client wanted to upgrade their storage repository to AWS S3.
We performed a Google Drive to S3 file migration while upholding all the related record information so we could “swap” the links upon landing in their new destination. The client is now running S3 and no longer has to worry about link expiration issues.
Data Migration Audit Process
During the migration process, we were very cautious to ensure no files were broken or omitted. Several saved searches allowed us to reconcile and confirm all files had successfully been replaced.
These examples show media images on the item record from Google Drive to S3. We provided the client with a side-by-side comparison of their GDrive and S3 image links for review after the migration.
We even added a Slack notification to notify us of the migration progress on a Slack channel. This integration enabled us to receive periodic status updates about the file migration process and determine how many files had been updated back into NetSuite with original links.
The examples shown are media images on the item record from Google Drive to S3. We provided client with side by side comparison of their GDrive and S3 images links for review after the migration.
We even added a slack notification to notify us of the migration progress on a slack channel. This enabled us to receive periodic status updates about the file migration process with how many eXtend Files had been updated back into NetSuite with original links.
Migrating File Cabinet to Cloud Repository
With some of the new NetSuite tier thresholds recently introduced as well as the common file cabinet challenges, we’ve had several clients reach-out to us seeking to migrate their file cabinet usage to eXtendFiles. For each migration, it was imperative we retained all of the associated records and rebuild the connections upon the eXtendFiles creation. We needed to uphold the link to the file cabinet file as well to the newly created eXtendFiles record before deleting the file cabinet files. This dual retention prior to deletion allows for auditing and UAT to ensure client has not lost any data prior to purge. Once the client verified the parity of files, we were clear to purge the File Cabinet of the original files.
Private / Local Files Repository to S3 (Using Cloudberry)
When files are on a network drive or a local Windows/Mac computer, we have used MSP360 (formally known as Cloudberry) to move those local files to the storage cloud. We also use Node.js scripts to index the content of directories prior to migration and put them into CSV files for auditing considerations.
An excellent alternative to network transfer is the Amazon Snowball, with exciting features like an LED screen that acts as a two-way Shipping Label, which changes once it arrives at your facility and needs to be sent back!
Ongoing Synchronization
File Cabinet to Cloud Repository (AWS S3/One Drive/SharePoint)
In this situation, the client wanted to use the native Expense Log available in the NetSuite Mobile App. The users would log their expense reports and upload receipt images. These postings would then wait until the expense report is processed (fully paid). Once it is, we would move the File Cabinet files into the repository, create eXtendFiles records, then link eXtendFiles to expense reports, which ensured all native aspects are sound. After, we migrate and purge files contextually when it was appropriate. This feature can be configured using the bundle settings and support.
eXtendFiles Cloud Storage Approach
eXtendFiles lets you upload your files into the cloud storage platform of your choice and lets you store all the files’ metadata onto a custom record. This enables our clients to do more with it using native saved searches, workflows and SuiteScripts.
We wrote scripts that would first index all the contents of the source repository and then transmit it to another repository using multiple technologies involving Node.js, AWS EC2, shell scripts, RESTlets and native CSV uploads.
Integrity Checks
We wanted to ensure our clients were confident about file content not being corrupted over the network, so we did the following:
- We added hash integrity checks for both downloading the file and after uploading the file.
- Developed the script also ensured all the multiple versions of files are downloaded and uploaded to the target repository with hash checks on each version of the file.
- Took a backup of file content metadata from the source repository into the target repository to audit the contents for post-mortem analysis.
- Created CSV files for eXtendFiles uploads to enable much faster updates of links back into NetSuite records and avoid the execution of SuiteScripts and workflows.
- Made the number of rows per file configurable to allow for customization and avoid NetSuite limitations on CSV import file sizes.
NetSuite Resource Utilization
We also wanted to stay conscious of NetSuite resource utilization and avoid any disruptions to workflows, so we:
- Performed the CSV uploads back into NetSuite during non-business hours to avoid performance issues.
- Helped clients that have a SuiteCloud Plus license by taking advantage of multiple CSV processing queues for higher throughput when updating eXtendFiles back into NetSuite.
- Added the ability to configure the concurrent calls while downloading and uploading the files to fully exhaust the available concurrency limits of the source and target cloud repository considering the applicable rate limits.
- For migrations to AWS S3, used EC2 instances within the same region to ensure minimal data transfer cost and higher performance.
We performed the CSV uploads back into NetSuite during non-business hours in an attempt to avoid performance issues. We even took it a step further for clients that have a SuiteCloud Plus license. This enabled us to take advantage of multiple CSV processing queues which gave us a higher throughput for updating eXtendFiles back into NetSuite.
We further added ability to configure the concurrent calls while downloading and uploading the files so, that we could fully exhaust the available concurrency limits of the source and target cloud repository considering the applicable rate limits. This is important from being able to meet the deadlines of the project. When performing migrations to AWS S3, we use EC2 instances within the same region to ensure data transfer cost is minimal and we get higher performance.
NetSuite Data Migration Best Practices
Several approaches that can be taken to ensure your file migration can be properly audited. In some cases, it can be as basic as going into your cloud storage interface to review the count of files before and after migrations. Other times, you may want to have the ability to audit the exact content and perform comparisons file-by-file when needed.
Typically, when we migrate the files related to NetSuite records we also provide a saved search that shows both the original and updated file links so clients have a simple side by side comparison of the migration results. We also usually provide the migrated content file name and paths in a CSV file which you can use as reference for future audit needs.
Take Charge of Your NetSuite Data Migration Strategy
eXtendTech provides professional NetSuite data migration services to simplify your file migration process, offering tailored services to your specific business needs. Whether you need to migrate files from one source to another or need some support following NetSuite data migration best practices, we can provide the support you need for success. Learn more about all of the features our eXtendFiles app offers for NetSuite file migrations, or contact us today to learn more about how we can support you in your migration.
Subscribe to Our Blog
Receive updates whenever a blog goes live!
