MongoDB replica set over ssh tunnels


I'm trying to archive something like this
enter image description here

The MongoDB instance on is currently a standalone, and On-Site periodically pulls a backup via mongodump over a ssh tunnel.

I now want to create two instances, one On-Site and one in a second datacenter, and make the three instances become a replica set, so that the backup is almost instant. This way I would have an On-Site backup and one in the second datacenter, which would always be up to date.

If the main MongoDB instance in fails, then that's it, no data can be served, this will then need to get fixed manually, as the On-Site instance and the one in the secondary datacenter should not take over. Nor are those two meant to be queried. They are there just as a backup.

My problem is that when I add –replSet to the main database, and then a rs.initialize(), that this instance is then somehow not recognized as bound to localhost, but to, which is the docker interface, which I need to bind to so that the containers can connect to MongoDB. If I then try to add a member via rs.add("localhost:2001"), which is the instance on, I get an error about not being able to mix localhost and non-localhost databases. The main instance is recognized as instead as localhost.

"Either all host names in a replica set configuration must be
localhost references, or none must be; found 1 out of 2"

I then used in order to issue a rs.initiate({_id:"rs0", members: [{"_id":1, "host":""}]}) to force the initialization of the replica set on to be bound to localhost. But when I then try to add a member via rs.add("localhost:2001") I still am not able to do it. It's no longer the hostname error, but another one, which I can't quite remember, as I got tired of trying and had to rollback everything. It was something like connection rejected. When I do a mongo --port 2001 I get disconnected after a warning about something related to "isMaster".

Is it possible to create this kind of setup? All I'm trying to do is to avoid using TLS on MongoDB and binding the backups to globally accessible interfaces (the On-Site instance would be behind a firewall, so would not be publicly accessible)

Best Answer

The following setup is possible, without using TLS.

Datacenter 1 is the primary datacenter. All clients connect only to this one. Datacenter 2 is only for realtime backups via replication. On-Site is also only for realtime backups via replication.

This setup is NOT meant for failover, only for disaster recovery.

Datacenter 1:


mongod --bind_ip, --replSet test --port 2000 --rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger --wiredTigerDirectoryForIndexes --fork is for SSH access, is for access from Docker containers.

SSH Tunnels (this server is responsible for creating the tunnels to Datacenter 2):

autossh -f -N -L 2001:localhost:2001

autossh -f -N -R 2000:localhost:2000

Datacenter 2:


mongod --bind_ip --port 2001 --replSet test --rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger --wiredTigerDirectoryForIndexes --fork



mongod --bind_ip --port 2002 --replSet test --rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger --wiredTigerDirectoryForIndexes --fork

SSH Tunnels (On-Site is responsible for creating the tunnels to Datacenter 1, no persistent connection exists between Datacenter 2 and On-Site)

autossh -f -N -R 2002:localhost:2002

autossh -f -N -L 2000:localhost:2000

The server in Datacenter 1 contains all the data which is to be replicated, Datacenter 2 and On-Site are empty databases.

On the machine in Datacenter 1 I enter mongo --port 2000, then I issue a

      _id: "test",
      version: 1,
      members: [
         { _id: 0, host : "localhost:2000", priority: 1, votes: 1 },
         { _id: 1, host : "localhost:2001", priority: 0, votes: 0, hidden: true },
         { _id: 2, host : "localhost:2002", priority: 0, votes: 0, hidden: true }

I guess priority: 0, votes: 0, hidden: true on the secondaries is optional here, it does what I want, just have the two secondaries be a part as silent, invisible data collectors for backup purposes. I'm not sure if changing this would have any side-effects as it could be possible that then a connection between both secondaries should be available, which it is not, in this case (they can't see each other).

I could use a slaveDelay on one of the hidden members in order to make it lag for let's say half a day, but since I will be making plenty of backups from both hidden databases via mongodump, it's not really neccessary.

In order to be able to connect to the secondaries via mongodb tools I needed to issue a rs.slaveOk()

Nothing had to be changed in the clients, they connect to

I tested a reboot of the machine in Datacenter 2 and during the reboot I inserted new items in the DB, after about 30 seconds the rebooted machine synced to Datacenter 1 as expected.

Datacenter 1 has a fast server with lots of RAM and CPU and only a little bit of SSD storage, Datacenter 2 has a little bit of RAM but lots of HDD, so this seems to be a good solution.