AWS database migration services from IT Svit
Your company surely runs various infrastructure and databases to support different aspects and facets of your IT operations. Quite often the dire need to cut CAPEX and OPEX forces the business to migrate all their operations and systems to the cloud. As Amazon Web Services is the most popular cloud computing provider worldwide, it is the most common destination for database migrations. The process has some pitfalls and underwater problems, however.
First of all, AWS is a mammoth of a company, but it is a very agile and ferocious beast. All AWS products and services are actively developed, new features are introduced on a regular basis and keeping abreast with AWS is quite hard if you are not dealing with it on a daily basis. Thus said, designing and building the most performant and cost-efficient AWS cloud infrastructure for your databases and systems is nearly impossible if your IT department doesn’t have previous — and quite recent — experience with such tasks.
Thus said, there are three approaches to AWS database migration:
- Obtaining the required expertise in-house. This is a standard process of hiring the talents you need to do the job required. The main benefit is that you will get the needed specialists, who will be able to accomplish the project and deliver much more value over time. The drawbacks include the possibility of a long hiring time, the risk of recruiting sub-par database administrators (especially if you don’t have the expertise required to evaluate their skills preemptively), the need to pay them once the project is over or the expenses related to firing them.
- Delegating the task to AWS customer support. These IT engineers have an in-depth understanding of AWS products and services, so they can design and implement the required infrastructure for your databases, apps, files and workloads. However, the downside here is that AWS support will prefer AWS services over open-source alternatives unless they are explicitly told not to do so. The issue here is that unless you know exactly what needs to be done, AWS support will be sure their approach is better, but if you knew it — there would not be any need in ordering any assistance from AWS, isn’t it?
- Hiring a third-party IT services provider. A company like IT Svit, that is a Managed Services Provider, must be well-versed with all kinds of cloud-based tools and services, both vendor-specific and open-source. Thus said, this DevOps team should be able to compose the best mix of modules and tools to enable a smooth and error-proof AWS database migration for your project.
The cloud database migration itself can be done in two ways. You can update the database to the oldest version supported by the cloud platform (and later update it to the latest version in the cloud), make a database dump, upload it to the cloud and restore it to the new system. The other approach
Thus said, IT SVIT has ample experience with both kinds of a database migration: dump/restore and migrate/update. However, the effort required to do the latter is much more notable, so selecting the most suitable database for your project and migrating your data to AWS in the most cost-efficient manner is an essential part of a cloud migration process.
Use cases of working with Amazon database migration service
There are multiple scenarios of AWS database migration that can be applied based on the project needs. The most common roadmap of Google Cloud to Amazon database migration is described below.
List of AWS databases:
- Amazon Aurora
- Amazon RDS
- Amazon Redshift
- Amazon DynamoDB
- Amazon ElastiCache
- Amazon Neptune
- Amazon Timestream
- Amazon Quantum Ledger Database (QLDB)
- Amazon DocumentDB (with MongoDB compatibility)
- AWS Database Migration Service (DMS)
List of GCP databases:
- Cloud Spanner
- Cloud SQL
- Cloud Bigtable
- Cloud Firestore
- Firebase Realtime Database
- Cloud Memorystore
The most commonly used databases:
- MySQL
- PostgreSQL
- Redis
- Amazon Aurora
- Amazon Redshift
- MongoDB
MySQL is by far the most popular database out there and IT Svit covers a wide range of needs for this database migration:
- Migrate MySQL RDBMS from AWS EC2 to AWS RDS
- Migrate MySQL RDBMS from AWS EC2 to on-premise
- Migrate MySQL RDBMS from AWS EC2 to Google Compute Engine
- Migrate MySQL RDBMS from AWS EC2 to Google CloudSQL
- Migrate MySQL RDBMS from AWS RDS to AWS EC2
- Migrate MySQL RDBMS from AWS RDS to on-premise
- Migrate MySQL RDBMS from AWS RDS to Google Compute Engine
- Migrate MySQL RDBMS from AWS RDS to Google CloudSQL
- Migrate MySQL RDBMS from on-premise to AWS EC2
- Migrate MySQL RDBMS from on-premise to AWS RDS
- Migrate MySQL RDBMS from on-premise to Google Compute Engine
- Migrate MySQL RDBMS from on-premise to Google CloudSQL
- Migrate MySQL RDBMS from AWS RDS to AWS RDS in another AWS region
- Migrate MySQL RDBMS from AWS RDS, AWS EC2, on-premise, Google Compute Engine, Google CloudSQL to Percona XtraDB Cluster
The most typical database migration scenario is as follows:
- Stop the incoming traffic
- Make a dump of old DB, copy everything we need and can copy
- Upload the dump to the new infrastructure and restore it to the new DB
- Verify that everything works as intended, copy the data that was not copied automatically
- Switch the app to the new DB and shut down the old DB
However, there are varying scenarios when this roadmap can be updated. Specialized AWS DMS — Amazon Web Services Database Migration Service — can help with that.
- MySQL can be easily moved from one Amazon RDS instance to another through simple dump/restore. Just replicate both databases as master-slave, then promote the new DB to master and shut down the old DB instance.
- MySQL RDS can also be quite simply moved to another AWS Availability Zone or region, by using the same replica-slave-to-master approach.
- Self-hosted MySQL DB can be moved to the cloud following the same scenario.
- PostgreSQL can usually be moved between RDS instances without issues through dump/restore.
- MariaDB can be moved to another instance. You need to create a dump of the old DB with the data for activating the master replica and restore it to the new DB instance. Then the new and the old DB work as master-slave, then as master-master, then the old instance is shut down and the app is switched to the new DB. This is easily done using gtid.
- MongoDB can be moved to another instance even without replicas if the versions don’t differ critically, through simple dump/restore operation.
If there are significant differences between DB versions or you have to move to another DB title, the process changes a lot.
- Create a mirror of the existing DB and try to move it to the new DB
- Use dif to check the size of the errors and inconsistencies in the DB
- Remove the faulty or the deprecated data
- Create a dump of the data that can be moved without errors, restore it to the new DB
- Manually set up the rest of the data
- Switch the app to the new DB, shut down the old DB instance
There were also border cases when we had to build custom solutions. For example, we once had to migrate PostgreSQL from bare metal servers to Amazon RDS. The task was largely done through standard pg_dump/restore procedure, but we faced certain issues with the owner role rights, which had to be reconfigured from scratch.
In another case, we had to migrate MongoDB from bare metal servers to Atlas. They are not fully compatible, so we had to move one collection at a time and build the indexes afterward. When the same had to be done with Oracle DB, we had to write a big base of scripts to enable automated replication.
Here is an example of a script for moving one AWS RDS PostgreSQL to another AWS instance:
##postgres logical replica Create 2 instances (master,replica) RDS PostgreSQL V 10.4 For the master instance, attach a custom parameter group and modify 2 parameters (rds.logical_replication=1 , wal_sender_timeout=0). This is required for replication to succeed. Note: rds.logical_replication is a static parameter that requires a reboot of the DB instance for the parameter to take effect. Create "test1" table on both instances and insert some rows on the master CREATE TABLE public.test1 ( c1 numeric(28,6) NOT NULL, c2 character varying(3) COLLATE pg_catalog."default", CONSTRAINT test2_pkey PRIMARY KEY (c1) ); Create Publication on the master as following CREATE PUBLICATION pub FOR TABLE test1; Create SUBSCRIPTION on the replica as following : CREATE SUBSCRIPTION sub CONNECTION 'host=master.XXXXXXXx.eu-west-1.rds.amazonaws.com dbname=NAME user=USER password=PASSWORD' PUBLICATION pub ; NOTICE: created replication slot "sub" on publisher CREATE SUBSCRIPTION
Need a database migration service to AWS? Ask IT Svit!
Thus said, IT Svit has ample experience with various kinds of Amazon database migration services. We have ready scenarios for the majority of cases and our expertise allows building bespoke database migration scenarios in no time. We can help you accomplish the required tasks and improve the overall efficiency of your IT operations, by providing a wide range of DevOps services for startups.
Should you require reliable database migration services to and from AWS — IT Svit is ready to help!
