Replicating a repository to preserve the bandwidth on remote sites¶
When WAPT is used on bandwidth limited remote sites, it makes sense to have a local device that replicates the main WAPT repository to reduce the network bandwidth consumed when deploying updates on your remote devices.
With remote repositories, WAPT remains a solution with a low operating cost because you don’t have to implement high bandwidth fiber links to take advantage of WAPT.
It works as follows:
- a small form factor and no maintenance appliance with the role of secondary repository is deployed on the local network of each remote site;
- the remote repository replicates the packages from the main repository and other repositories;
- the WAPT clients connect in priority with the repository that is the closest to them, the local repository;
To easily implement the replication of your main repository on your distant sites, Tranquil IT sells inexpensive appliances that are pre-configured according to your storage needs (16/30/60Go).
The appliances are available in our online shop.
Syncthing is a multi-OS open source peer to peer synchronization utility.
It allows to synchronize folders on several machines while guaranteeing the security, the authenticity and the integrity of the files.
The official Syncthing documentation is available online.
Implementing the replication¶
The following documentation is applicable to Linux Debian and CentOS/ RedHat based WAPT Servers and remote repositories.
Setting up the remote WAPT repository¶
echo "deb http://wapt.tranquil.it/debian/ ./ " > /etc/apt/sources.list.d/wapt.list apt-get update apt-get upgrade apt-get install tis-waptrepo
CentOS/ RedHat Linux¶
The remote WAPT repository is set up, we now must implement the Syncthing replication.
Configuring the Syncthing service¶
a2enmod ssl a2ensite default-ssl.conf
- modify the Apache configuration files to define the correct roots of the VirtualHosts:
- change the value of DocumentRoot in each configuration file:
- DocumentRoot /var/www/html + DocumentRoot /var/www
- finally, restart the Apache web service to apply the new configuration:
It is advisable to specify valid SSL certificates in the Apache configuration of remote repositories.
- empty the content of the folders
/var/www/wapt-host; Syncthing will fill again these folders with data from the main repository.
rm -rf /var/www/wapt/* rm -rf /var/www/wapt-host/*
Installing Syncthing on main and remote WAPT repositories¶
This procedure is to be applied on the main repository and on the remote repositories.
## Debian apt-get update apt-get install sudo curl apt-transport-https curl -s https://syncthing.net/release-key.txt | apt-key add - echo "deb https://apt.syncthing.net/ syncthing stable" | tee /etc/apt/sources.list.d/syncthing.list apt-get update apt-get install syncthing ## CentOS 7 wget https://github.com/mlazarov/syncthing-centos/releases/download/v0.14.7/syncthing-0.14.7-0.el7.centos.x86_64.rpm --no-check-certificate yum install syncthing-0.14.7-0.el7.centos.x86_64.rpm
Operations to follow:
- add the Syncthing service to systemd;
- change the listening port to 0.0.0.0;
- create an administrator account and enter a strong password;
- activate the HTTPS protocole for the web access;
create the definition file for the waptsync service by editing
[Unit] Description=WAPT respository sync with syncthing Documentation=http://docs.syncthing.net/ After=network.target ;Wants=syncthing-inotify@.service [Service] User=wapt ExecStart=/usr/bin/syncthing -logflags=0 -home=/opt/wapt/.config/syncthing/ -no-restart Restart=on-failure SuccessExitStatus=3 4 RestartForceExitStatus=3 4 [Install] WantedBy=multi-user.target
create the tree structure required for the waptsync service to start:
mkdir /opt/wapt/.config/ mkdir /opt/wapt/.config/syncthing/
- change the owner of the files:
chown -R wapt:www-data /opt/wapt/.config/
- activate the waptsync service and start it.
The configuration files will appear in the
systemctl enable waptsync systemctl start waptsync systemctl stop waptsync
- change the listening port in the
<gui enabled="true" tls="true" debugging="false"> <address>0.0.0.0:8384</address> <apikey>4jvEiL24UbFddsdsAQxqsfixNaLt</apikey> <theme>default</theme> </gui>
- start the waptsync service:
systemctl start waptsync
Configuring Syncthing’s web service¶
Syncthing’s web interface is now accessible by visiting http://<server_ip>:8384.
Operations to follow:
- change the host name of the remote repository;
- add a GUI authorized user;
- add the password for the GUI authorized user;
- check the box Use HTTPS for the GUI;
- click on Save;
- connect with SSH on the WAPT Server and restart the Syncthing service:
systemctl restart waptsync
Syncthing’s web interface is now only accessible with HTTPS on https://<server_ip>:8384.
in the list of shared folders, remove the default folder:;
configure the replication:
Those actions must be run on WAPT Server.
In the list of shared folders (Shares):
- add a shared folder with ;
- fill in the path to the directory to be shared,
- in the scroll-down menu ;
- in the scroll-down menu ;
- redo the same operation for wapt-host:
add the remote repository to WAPT Server’s Syncthing:
Once Syncthing has been installed on the main and remote repositories, recover the remote repository’s ID ().
This unique identifier appears like
On the main repository, in the list of remote devices (Other Devices):
- add the remote repository with ;
- fill in the ID of the remote repository;
- tick the checkbox for the shared folder wapt and wapt-host;
On the remote repository, follow these steps:
- the remote device receives a Syncthing notification that it has been approved and added to the main device’s replication schedule;
- the client receives a popup form asking to accept the synchronized
The replication is now operational.
secure the replication:
By default, the following parameters are active in Syncthing:
|Activate the NAT||Use a UPnP port mapping for incoming synchronization connections.|
|Local discovery||Syncthing will then broadcast to announce itself to other Syncthings.|
|Global discovery||Syncthing registers on an external cloud service and can use this cloud based service to search other Syncthing devices.|
|Possible relay||The use of relays allows to use external servers to relay the communications. The relay is activated by default but it will be used only if two devices can not communicate directly between themselves.|
This operating mode simplifies the global setup but it is not the most advisable method in relation to security.
- uncheck all boxes in network configuration;
- define the listening port (by default port 22000);
- replace default by tcp://0.0.0.0:22000;
Then go on the web interface of the remote repositories, click on Change and fill in the IP address of the remote repository:
- replace dynamic by tcp://<remote_repo_ip_address>:22000;
This configuration is useful for limiting inbound connections to the Syncthing service.
Configuring the WAPT agents¶
WAPT clients on remote sites must now be configured to look for updates from their closest available repositories.
Two solutions exist:
- use automatic detection via DNS SRV fields;
- change manually or via a WAPT package the parameter repo_url
in the WAPT agent’s
Configuration example of the WAPT agent - filled in local repository:
[global] waptupdate_task_period=120 waptserver=https://srvwapt.mydomain.lan repo_url=https://localrepo.domain.lan/wapt/ use_hostpackages=1
Example of a WAPT package designed to remotely change the repo_url
# -*- coding: utf-8 -*- from setuphelpers import * uninstallkey =  def install(): print('Modifier la configuration agent pour le site de Colmar') inifile_writestring(WAPT.config_filename,'global','repo_url', 'https://wapt.city02.mydomain.lan/wapt/')