When you have to move data from one computer to another or from a client to a server, you need tools that can reliably move your files and verify that the move was successful. In this guide, we’re going to look at eight such tools, each of which solves a different sort of problem or gives you a different approach or interface to get the job done.
This category has a ton of software products. FTP software, for example, has been around for decades. Our approach was to look for products that worked (or had versions for) multiple platforms, as well as a deep feature set and a solid interface. Later, I’ll show you two tools I personally use regularly and rely upon, as well as two enterprise-class tools for when you need to move vast amounts of data on an ongoing basis.
Best multi-platform file sync tool
The raison d’etre of a good sync tool is keeping two directories in sync, making them identical. Most sync apps add to that, allowing for backups, test runs, etc. But the key is finding files that are supposed to be moved and moving them. Sync tools can often be used in a one-way sync, and when that happens, they also remove files from the destination directory if they’ve been removed from the source directory.
Our pick for the best multi-platform sync tool is GoodSync. There are many other excellent sync tools out there, but GoodSync distinguishes itself by running on MacOS, Linux, Windows, iOS, Android, and even some NAS boxes. GoodSync is sold in 5-license sets, so you can mix and match platforms with a great deal of flexibility.
Pros:
- It runs on a wide range of platforms
- Single license for all platforms
- Free (more limited) version
Cons:
- Multi-device support is not free
- Users report licensing snafus with lifetime license
Honorable mention: rsync
No discussion of file sync utilities is complete without mentioning rsync. This is a command-line utility available in most Linux implementations. It was created initially to solve the problem of syncing large file structures between Australia and other continents over the low-bandwidth intermittently reliable connections of the 1990s.
The utility distinguishes itself by keeping bandwidth usage to a bare minimum and carefully understanding the differences between two directories, only syncing the bytes (not the files) necessary to bring the two into parity. I am personally grateful to rsync for getting me out of jams on numerous occasions.
Best volume cloning tool
There are times when the best way to move data is to make an image of an entire volume. That can be for backups, quick restores, migration to virtual machines, or — as I often use it — to quickly be able to reconstitute test machines when evaluating a series of products against each other.
Many volume cloning tools exist, but most of them are specific to a single platform. If you think about what cloning is all about, it includes an awareness of volume format down to the sector level — and that differs from OS to OS. True Image distinguishes itself by being able to create images for a wide range of PC-based file systems including NTFS, Ext2/Ext3/Ext4, ReiserFS, Linux SWAP, HFS+/HFSX, FAT16/32/exFAT as well as Mac-based formats including APFS, HFS+, Core Storage, FAT32, and NTFS (including Boot Camp).
Pros:
- Clear interface
- Includes anti-malware support
- Real-time imaging option
Cons:
- No native Linux version
- Some users report difficulty creating bootable media
Best cross-platform FTP tool
FileZilla is not my favorite FTP tool. I don’t actually have one favorite. I tend to keep three or four loaded on my development machine and switch between them depending on which interface suits my needs on any given day. While all FTP programs move files, each has its own set of features that prove useful on occasion.
FileZilla, on the other hand, is free and it’s multi-platform. It supports the basic FTP protocol (which you really shouldn’t be using) as well as FTP over TLS (FTPS) and SFTP. FileZilla works on Windows, Mac, and Linux. There’s a Pro version available for twenty bucks that adds connections to popular file-sharing services. But we’re recommending the base FileZilla. It works, it does its job, it’s free.
Pros:
- Free
- No SCP support
Cons:
- Interface feels dated
- Support for cloud services add-on upsell
Best cross-platform cloud-mount-as-drive software
FTP will do fine for moving files between volumes, but its interface is usually a two-pane affair where you see one directory on one side and one directory on the other. Another way to move files is to create a mountable drive, where you use your machine’s native Finder or File Explorer interface to move and place files. Sometimes, applications can even access files on the remote volume as if they were local.
That’s what NetDrive does. We particularly like that NetDrive not only mounts remote FTP volumes, but it will allow you to mount most cloud services as drives as well. NetDrive is available for Windows, Linux, Mac, and Android.
Pros:
- Supports mount-on-boot
- Supports locking files
Cons
- No support for client-side encryption
A couple of my favorite problem solvers
I include in this category two tools I often rely on to solve two very different challenges: bulk image manipulation and WordPress database transfer. They’re not related, except that they are data migration tools — and that both have saved me a ton of time and gotten me out of some very tight challenges.
Best bulk image converter
Here’s a task I was facing last week. I had about 20 images in a variety of formats. I needed to scale them to be all 1024 pixels wide (I didn’t care about height, but the aspect ratio had to be maintained). Once scaled, I needed a 4-pixel white border on each. Then I needed to save them to JPEG, with a naming convention that began “fig-” and an index number ranging from 00 to 20.
Sure, I could have done that in Photoshop, but it would have taken an hour or more of very tedious mousing. Instead, I did it in XnConvert in less than five minutes. All I did was select the files to operate on, tell it a few transformations to perform and specify the output format. That’s XnConvert. That’s what it does, and it does it very, very well — for free. What more could you want?
Best WordPress database migration tool
If you develop WordPress sites, sooner or later, you’re going to need to move a WordPress database from one site to another and one domain to another. My example is that I develop on my local machine (which use domains like mydomain.local), but when I push the site into production, I usually push it to a site with the domain mydomain.com.
Simply exporting a WordPress database on the local machine and importing it on the production machine won’t work because a lot of the internal data will have the wrong domain. This is where WP Migrate DB Pro comes in. It helps you move databases back and forth and modifies the records so that the database is appropriate for the machine it’s running on. It does more like allows you to make that database move relatively regularly, syncing changes, so you don’t lose any data.
The bottom line is this: if you don’t develop WordPress sites, you won’t care. If you do develop WordPress sites, you’ll need this, even if you don’t know you need it. My butt: saved numerous times. It’s a commercial tool with a non-cheap price tag, but the company’s name is Delicious Brains, so what more can I say?
Enterprise-level tools
We’re ending our list with two outstanding open-source enterprise-level tools from the Apache Foundation: NIFi and Airflow. Both of these can manage small projects but can scale to vast projects.
Best dataflow automation tool
Apache NiFi has a pretty storied pedigree: It originated at the US National Security Administration (the NSA) and was open-sourced back in 2014 as a technology transfer. NiFi is slick. It can take just about any data, process it in some way, and spit out new data. For example, if you want to convert a JSON file so that it’s imported into MySQL, you can.
More to the point, NiFi generalizes that process. It has an enormous library of data processors (tools for parsing and moving data in a wide range of formats and from a wide range of systems). You can then string these processors together with logic and data-relevant operations. You can split streams, allowing various threads to run concurrently. You can also back-limit processing, which means you can slow down until another processing flow catches up.
This is a big, wonderful, powerful project, and since it’s open-source, it’s also free (although there is commercial support for it through HortonWorks).
Best workflow management system
AirFlow was built at Airbnb to help the company manage its very complex workflows. It has, subsequently, been released as open-source and is managed by the Apache Foundation.
Unlike every other tool on this list, AirFlow is entirely controlled through programming. You need to code in Python for every operation managed by AirFlow. What AirFlow does, though, is provide the ability to manage tasks and dependencies. The product comes with a wide range of integrations, as well as tools for managing and tracking the processing of any given flow. Flows can be triggered by external events or on a schedule (replacing the much more cumbersome crontab mechanism).
So there you go. You’re not required to choose the tools we present as best since there are many great tools in this category. That’s why we included alternatives, especially if you want something specific for your favorite platform.
FAQ
How critical is encryption?
If you’re moving files across your own internal LAN, encryption is not necessarily mission-critical. But if you’re moving data across the open Internet, encryption is pretty much mandatory. Keep in mind that encryption doesn’t always have to live inside the file transfer application. For example, I’ve been opening SSH tunnels and doing FTP transfers through those tunnels for years. It’s not the most convenient approach, and I prefer tools with nice UIs, but you always make sure your data is protected no matter how you do it.
When should I use sync?
That’s a good question because they kind of do the same thing, kind of. Sync tools are designed to keep two directories up to date, in sync. When you’re dealing with entire directories, and especially if you want them to match, sync is your friend.
When should I use FTP?
FTP is the tried-and-true go-to choice for file movement. FTP (or, really, SFTP) will do fine if you’re moving a few files around. Modern UIs usually present the source and destination side-by-side in panes, and I like being able to see both sets of files and dragging (or clicking arrows) to move them back and forth.
When should I use a cloud mounter?
Cloud mounting is a different approach but can solve the same problem. If you use a cloud mounter, you’re using your OS’s native file management environment and don’t have to launch a separate application or manage separate windows. That means that if you’re already drilled down a bunch of levels to a directory and have that open in a window, you don’t need to find it again in the FTP program. Depending on what I’m working on, I tend to alternate back and forth between FTP and cloud mounting.
What are your favorite tools? Let us know in the comments below.
You can follow my day-to-day project updates on social media. Be sure to follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.