Migrating Email Hosts and Registrars with Minimal Downtime

Email is the focal point of our professional online identities. Whether you like it or not, that is the most reliable and consistent way for people to get in touch with you, and it’s not going away anytime soon (sorry, Slack). That’s why it’s imperative for your email provider to be reliable, resilient, and especially in today’s world, trust-worthy. 

Recently I realized that my email host is not great. I registered my domain and set up email accounts there long before I was thinking about privacy concerns or shopping for competitors. They were ok at what they did, but they specialized in assisted hosting services and email was clearly an afterthought, let alone any privacy or scalability concerns. 

I knew I needed to migrate email providers. And I knew that this is going to be a tricky and intricate operation. What I was looking for was:

  • minimal downtime,
  • end up on a provider that I trust long term and,
  • migrate all of my current emails and folders. 

It also happens that my domain was registered through the same email provider host and I wanted to transfer that over to my everyday registrar, Namecheap. Might as well knock everything out at once and be in good shape for the next while so that I can sleep well at night without worry. 

Decisions, decisions, decisions

First thing’s first is you have to find the new registrar and email provider you want to migrate to. This involved some heavy research on my end, because you don’t want to have to do this migration more than once. Everyone has their own priorities in what they’re looking for; for me it was privacy, management, and support. That One Privacy Site has a great article on what to look for in email providers, along with a Simple Email Comparison chart. I spent quite a bit of time looking through these options, and ironically ended up picking a host that wasn’t even on the list, Migadu. As for a registrar, I default to Namecheap as I already have quite a few domains registered there and like their service. 

Inform Your Users

I am a big believer in transparency, and especially with today’s internet climate, more transparency is better. I sent a letter to all of the active users on the domain stating that 1) I plan to migrate our email provider and why, 2) to which provider and why and any differences in services they should expect, 3) estimated timeline, and 4) I need their email passwords. 

That last point is a point of contention. In order to back up and migrate my users’ email accounts to a new host, I need to ask them to give up their privacy temporarily. Since I am my users’ everyday ‘IT guy’, they trust me and didn’t worry about this point. As a good measure, I linked them to the page where they can change their passwords before they share it with me, so if their password is one they use a lot, they can change it to something they’re more comfortable sharing. 

That being said, even with the proper transparency, be sure to listen to your users and address their concerns. If you did the proper research ahead of time, they should trust you with this transition, but it is up to you to keep them at ease and make sure that they are ok with your plan moving forward. 

Start Back Up & Migration (Emails)

I wanted to not only migrate my current emails, but back them all up as well, in case something went horribly wrong. Here’s the thing, there were a lot of emails on our server. I wanted this back up and migration to run continuously on a stable connection. AWS to the rescue! I simply spun up a t2.micro EC2 instance running Ubuntu and attached a 100GB Block Store to it. This was going to be my ‘command center’ for all things IMAP (and SMTP) during this transition. 

Behind every software success story is a powerful open source tool behind it. This story is no different. 

As for the tools necessary, I searched far and wide for reliable, stable, and open source solutions capable of iterative processes. I came across imap-backup for backing up emails and imapsync for migrating. Once you have these 2 tools installed, follow their respective set up processes and plug in your user’s credentials. 

imap-backup

For imap-backup, you want your config.json file to look something like this: 

{
  "accounts": [
    {
      "username": "username@domain.com",
      "password": "",
      "local_path": "/ext_vol_data/username_domain",
      "folders": [

      ],
      "server": "imap.server.com"
    }
  ],
  "debug": true
}

Of course, be sure to fill in the values properly. You can leave the folders array empty to move over all of the folders. The local_path here is currently pointing to an EBS store that I mounted onto the EC2 instance. And I have debug set to true so that I can monitor the progress of the script through its output. Then, simply create 1 account object per user on your email domain. 

Since this is simply backing up our accounts, you can start running this immediately! This script works iteratively so you only need to do a ‘full’ backup once, and every time after that it will only download the difference. 

imapsync

Once imapsync was installed properly, I found this Github Gist to help sync multiple accounts at once. For multiple accounts, simply duplicate the line in the accounts.list file with the proper credentials. Make sure you also properly set the CONFIGURATION variables in the imapsync.sh file. This script is now ready to run, but it won’t work until you have both hosts set up and ready to receive requests. More on this to come. 

Start Back Up & Migration (Hosting)

The domain I was migrating also happened to have 2 different sites hosted on it on 2 subdomains that I wanted to bring along with me. Be sure you’re prepared for this by picking a new host and making sure that your environments, databases, and source code are all backed up and can be spun up in a new hosting environment. 

Take any action you need to try and spin it up before the actual migration. This is easier to do than an email migration since you can have the same site set up on multiple servers, but you can only have 1 email host at a time! Ideally, your sites should be migrated and simply waiting for the DNS change to take effect. 

Configure New Email Host

Because I wasn’t only migrating email hosts, but also registrars, I couldn’t pull the trigger on the domain transfer until I was sure that things looked right on the new email host’s side. I got in touch with Migadu’s support and they worked with me to ‘enable’ my domain name even though it wasn’t transferred yet. They also gave me a 1 month free trial ‘Basic’ account so I can see the full extent of the service. 

Using this, I was able to make sure that the new email host was up and running. 

I created email accounts for all our users on Migadu with template passwords that I could provide them for their first login. I also set up things like alias email addresses (so one can use emails like name+service@domain.com) for everyone. 

Regex for ‘+’ email aliases: ^username\+(.+)@domain.com

Since I know had the original email host up and the new email host up, I was able to use imapsync to start transferring all the data to the new email host. With the proper configuration, the previously mentioned imapsync.sh script made the transfer smooth and seamless, migrating over all emails within their respective folders, and incrementally. 

Admin Stuff

If you also plan on transferring your domain to a new registrar, be sure that the ‘webmaster’ email for that domain is an email you have definite access to and that is not tied to the domain that you are migrating. 

Pulling the Trigger

Ok, now everything is in place and backed up. What’s left is actually pulling the trigger and starting the migration process. Pick a Friday or Saturday night, a relatively quiet night for emails in order to minimize the ‘blast radius’ in case anything does go wrong. Again, for transparency purposes, be sure to inform your users when you plan on doing this, so that if they see anything fishy, they know that you’re be behind the scenes working on it. 

Before officially starting the transfer process, be sure to run the imapsync and imap-backup commands one last time to ensure the mailboxes on each host and backups are as up to date as can be. 

Now, begin by transferring the domain to your new registrar. You can do this by going to the ‘transfer domain’ page at your new registrar. You will need to ‘unlock’ your domain from your previous registrar and make a note of the domain code for the transfer. 

Once that is triggered, be on the look out for any emails coming through to the webmaster email of the domain from your previous registrar. A lot of times they will confirm it with you before enabling the transfer to go through. 

Now you should start configuring any MX and DNS settings for your domain on the new registrar. The MX records are the most important to not have any downtime for your email accounts. Once those are complete, be sure to set up any other DNS records you may need for the sites you had hosted on the domain and point the nameservers over to the new web host. 

Time to make sure everything is working properly! 

In your mail client of choice, update your email settings to point to the new email host with updated credentials. The first thing you should notice is that your Inbox and all other folders should be present just as you expected, thanks imapsync! Now compose an email from any other email address of yours to the email on the domain you just transferred and make sure it comes through – IMAP is working properly. And compose an email from your newly transferred domain to your other email account – SMTP is working properly. 

Inform your users that the migration has taken place and provide them with the new host’s settings (IMAP and SMTP servers, ports, etc) and their new passwords. Of course, you’ll have to do this through means other than their email 🙂 Be sure to point them towards the settings page they need in order to update their passwords to something personal and unique. 

Once you’re sure your email is working properly, be sure to also test any hosted sites you migrated and all of their functionality as they now live in a new environment. 

It’s All About the Customer

Follow up with your users to make sure that everything is working as expected for them. This is also a good opportunity to talk to them and see how they felt about the whole process: was it smooth? Painful? What could have been better? Hopefully you won’t need to do this again, but it’s good to get this type of feedback in general. 

Clean Up

So now you should be sure that all of your emails and sites are migrated properly and working as expected. 

What’s left here is cleanup, which is just as important as set up! Do not cross this off your list until you finish properly cleaning up any set up you may have done. I would still wait a solid few weeks or so before starting this to make sure that everything is in place before you can’t go back, but be sure to set yourself a reminder so you don’t forget. 

Our main cleanup consists of running rm -rf on any and all email backups in the EC2 and Block Store. Be sure to do this for all the settings and config files as well, as your user’s (as well as your own) passwords are present there! You can now tear down these resources confidently knowing that your information lives only on your new host’s servers. 

Aftermath

Phew! Give yourself a pat on the back for a job well done! 

Be sure you have a real, paid account with your new host to stay on their good side and not let a trial expire without realizing. 

That’s it! Congratulations on a completed migration performed well with minimal downtime!

Photo by Web Hosting on Unsplash

Share Your Stack – How to Architect the Perfect Travel Blog, WhereTheHellAreTomerAndMichelle.com

When Michelle and I started hatching together our plan for our trip, I knew I wanted to grow, both as a person, and as a developer. The very first step that I could do to start growing was to pick a new technology stack to host our travel blog. I had already worked with WordPress, published on Dev.to, and I knew I wanted to host something (i.e. not Medium).

Requirements

Besides the basic blog necessities, my requirements were:

  • Static – This would be a simple site and I wanted to keep it light weight – no need for server side lifting here.
  • Easily updatable – I wanted to be able to easily update it and remove any roadblocks from publishing new posts or updating the site. Once you go CI/CD, you never go back!
  • Posts in Markdown – Markdown is simple, and it’s now everywhere. This is also a good way to separate your content from your site if you ever want/need to migrate your posts to a diffrent stack.

For a long time I had been hearing about Jekyll and Github Pages hosting and the simplicity and integration between the 2 technologies. This seemed as good a time as any to pick it up and run with it!

I use the Lagrange Jekyll theme for our site as it seemed pretty simple and we liked the way it looked (how does anyone pick anything??). I went ahead and forked the project and then started putting our own touches on it.

Basic Setup

Jekyll makes it very easy to customize certain parts of the site in the settings.yml file. We made sure to update our social links, About page, contact, etc.

We also definitely wanted our own domain name and not a github.io subdomain. Who would’ve thought that WhereTheHellAreTomerAndMichelle.com was available?? With my favorite domain name registrar, NameCheap, and $13, we were the proud new owners! Github Pages makes it easy to host your site on a custom domain name as long as the repository is public. Using the Pages Settings for your repo, it will configure your custom domain name by creating a CNAME file in your repository. Github Pages will also automatically take care of an SSL certificate, how thoughtful! We were off to the races!

Travel-ize It

Since it is our travel blog after all, we knew we needed to put our itinerary and picture slideshow on it. For the interactive map/itinerary we chose TravellersPoint.com‘s embedded map. I have used them before for my Eat, Pray, Code travel and loved what they provided. To customize the embedded map even further, I upgraded to a ‘Budding Member’ and placed the embed code directly in the Jekyll page. Since this is an iframe, I also added some Javascript and proper styles to ensure that the map works well on mobile as well as desktop views.

Pictures were slightly trickier. The first part was figuring out where to store my photos. I wanted to be able to upload only select photos, not have to worry about storage limitations (at least for the forseeable future), and be able to share a public album. Eventually Google Photos won out because of their ability to share an album publicly. (I am still open to suggestions of other photo storing/sharing solutions out there!)

The next issue with Google Photos is that I wanted to create an automatically updating embedded slideshow on our homepage. The tricky part comes with Google Photos’ lack of an API. Luckily, I came across this page where, given a publicly accessible Google Photos Album, it will create an embedded slideshow widget. At the moment this does not automatically update, I simply have to update the slider code every so often when I publish new photos to the shared album.

Publishing Posts

One thing that I was really looking forward to with this new stack (and Jekyll in general) was publishing posts in Markdown. As a developer, I love the simplicity and ubiquitousness of it, while still giving the author control over any elements they may need. When I begin to create a new post, I simply pop open my Markdown editor of choice, MacDown, and start typing away! When I finish a post and run it by Michelle for final touches and editing, I can publish it using a simple git push command!

Comments

I’m sure we’ve all come across Disqus at one point or another in our WWW adventures, and that is what I originally set up for our site. But seeing as I am a self hosting connoisseur, when I heard of Commento.io, I immediately knew I wanted to try it out. Luckily, not long after that, the Cloudron team pushed out the Commento.io self hosted application on their platform. I set it up on my Cloudron instance, imported all the data from the Disqus instance, and replaced the comments code with the proper Commento embed script. Viola! You now have self hosted, secure, and privacy focused comments on our site.

Analytics

Google Analytics is probably the go-to for many sites and blogs, but I have long been using Matamo (formerly Piwik), also self hosted. Using the Matamo Analytics dashboard to set up a new site is a breeze. Then, simply replace the Google Analytics code provided by default by the Jekyll theme with Matamo’s code, and push it up!

“But what if you get hit by a bus?”

Michelle, out of her love and thoughtfullness, one day asked me, ‘how do I publish new posts if you die?’ She had a point. Not only is she not a developer and has never used Github before, but the site’s repository is also only under my username.

After some quick research, I came across Forestry.io – a Jekyll CMS. This seems to be a fork off of Jekyll Admin, but either way, Forestry is a fantastic tool to make everything easier for managing a Jekyll site in the way of hosting media, direct integration to Github and Github Pages, and of course, creating and editing posts. And yes, they do offer a generous free plan for a personal site! Getting this set up was a breeze, the only slightly tricky thing was getting the media folder set up properly, but even that was done without a hitch. And since they support multiple users per site, both Michelle and myself have access to update our Jekyll site – including writing and publishing posts – through their CMS platform.

Stack Birds Eye View

As most good things in life, this took time and is not done evolving. You can check out the Github repository to see for yourself how each of the items above grew and were implemented over time. Sure, I had a general idea of what I wanted on the blog to begin with, but a lot of these ideas took time to develop and implement. Be sure to check out the final product (for now)!

Simple PHP + JavaScript CAPTCHA

While I was developing Fuck It All, I came across an issue that I’ve never had to deal with before. Since I was accepting user input that would in turn be posted on my site, I needed a way to make sure SPAM bots wouldn’t be able to make posts. There is quite a simple solution to this and I’m sure you have all come across it before, CAPTCHA inputs.

CAPTCHA inputs are small verifications, usually found on forms, which are easy enough for all humans to solve, but something that an automated machine wouldn’t be able to compute. These are commonly just a few words, but with some sort of distortion applied to them. Such text is, up until now, unreadable for computers. However, most humans can distinguish the individual characters without a problem.

For me to set up a system like that (without outsourcing), was way beyond what I wanted to do/had time to do. I could have easily used a publicly available widget and integrated that into my site, but I wanted to keep it classy and in-house. My friend helped me implement a simple CAPTCHA mechanism that can easily be implemented into a site using only PHP to generate the CAPTCHA and JavaScript to validate it.

HTML entities are reserved characters in HTML. To read more about HTML entities, go here.

Now for the actual CAPTCHA. Using these HTML entities, we can show a simple addition problem with two numbers. Most humans should be able to pass this quick test, while spam bots won’t be able to recognize what your site is asking them due to the HTML entities.

In your form, add the following PHP code:

<php 
   $num1 = "&#" . rand(49,57) . ";";
   $num2 = "&#" . rand(49,57) . ";";
   echo "<label for='captcha'>* <span id='captchaval1'>$num1</span> + <span id='captchaval2'>$num2</span> = </label><input type='text' name='captcha' id='captcha' maxlength='2' required /> (just making sure you're not a robot)";
?>

This generates two random numbers between 1 and 9 and tells the user to solve the math problem. Now we need to check if the user is correct using the following JavaScript code:

<script type='text/javascript'>
function validate() {
   var passed;
   var val1 = document.getElementById('captchaval1').innerHTML;
   var val2 = document.getElementById('captchaval2').innerHTML;
   var captchaval = document.getElementById('captcha').value;
   var result = parseInt(val1) + parseInt(val2);
   if (result != captchaval) {
      passed = false;
   } else {
      passed = true;
   }

   return passed;
}
</script>

By returning the ‘passed’ value to the form, you can check whether the user got the simple math question right or not. From there, you can handle the form submission as you please.

I would also like to thank my good friend Daniel for bringing this to my attention and helping me out with this code. If you have a second, you should check out some of his stuff as well. Right along the same alley: http://omniimpact.com/

Thanks for reading! I hope you guys enjoyed my first code posting!