Replicating a repository to preserve the bandwidth on remote sites

When WAPT is used on bandwidth limited remote sites, it makes sense to have a local device that replicates the main WAPT repository to reduce the network bandwidth consumed when deploying updates on your remote devices.

With remote repositories, WAPT remains a solution with a low operating cost because you don’t have to implement high bandwidth fiber links to take advantage of WAPT.

Replicating WAPT repositories

Replicating WAPT repositories

It works as follows:

  • a small form factor and no maintenance appliance with the role of secondary repository is deployed on the local network of each remote site;
  • the remote repository replicates the packages from the main repository and other repositories;
  • the WAPT clients connect in priority with the repository that is the closest to them, the local repository;

Hint

To easily implement the replication of your main repository on your distant sites, Tranquil IT sells inexpensive appliances that are pre-configured according to your storage needs (16/30/60Go).

The appliances are available in our online shop.

Introducing Syncthing

Syncthing is a multi-OS open source peer to peer synchronization utility.

It allows to synchronize folders on several machines while guaranteeing the security, the authenticity and the integrity of the files.

The official Syncthing documentation is available online.

Implementing the replication

Hint

The following documentation is applicable to Linux Debian and CentOS/ RedHat based WAPT Servers and remote repositories.

Setting up the remote WAPT repository

Debian Linux

echo  "deb  http://wapt.tranquil.it/debian/  ./  "  > /etc/apt/sources.list.d/wapt.list
apt-get update
apt-get upgrade
apt-get install tis-waptrepo

CentOS/ RedHat Linux

The remote WAPT repository is set up, we now must implement the Syncthing replication.

Configuring the Syncthing service

a2enmod ssl
a2ensite default-ssl.conf
  • modify the Apache configuration files to define the correct roots of the VirtualHosts:
/etc/apache2/sites-available/default-ssl.conf
/etc/apache2/sites-available/000-default.conf
  • change the value of DocumentRoot in each configuration file:
- DocumentRoot /var/www/html
+ DocumentRoot /var/www
  • finally, restart the Apache web service to apply the new configuration:
/etc/init.d/apache2 restart

Note

It is advisable to specify valid SSL certificates in the Apache configuration of remote repositories.

  • empty the content of the folders /var/www/wapt and /var/www/wapt-host; Syncthing will fill again these folders with data from the main repository.
rm -rf /var/www/wapt/*
rm -rf /var/www/wapt-host/*

Installing Syncthing on main and remote WAPT repositories

Note

This procedure is to be applied on the main repository and on the remote repositories.

## Debian
apt-get update
apt-get install sudo curl apt-transport-https
curl -s https://syncthing.net/release-key.txt | apt-key add -
echo "deb https://apt.syncthing.net/ syncthing stable" | tee /etc/apt/sources.list.d/syncthing.list
apt-get update
apt-get install syncthing

## CentOS 7
wget https://github.com/mlazarov/syncthing-centos/releases/download/v0.14.7/syncthing-0.14.7-0.el7.centos.x86_64.rpm --no-check-certificate
yum install syncthing-0.14.7-0.el7.centos.x86_64.rpm

Configuring Syncthing

Operations to follow:

  • add the Syncthing service to systemd;
  • change the listening port to 0.0.0.0;
  • create an administrator account and enter a strong password;
  • activate the HTTPS protocole for the web access;
  • create the definition file for the waptsync service by editing /etc/systemd/system/waptsync.service:

    [Unit]
    Description=WAPT respository sync with syncthing
    Documentation=http://docs.syncthing.net/
    After=network.target
    ;Wants=syncthing-inotify@.service
    
    [Service]
    User=wapt
    ExecStart=/usr/bin/syncthing -logflags=0 -home=/opt/wapt/.config/syncthing/ -no-restart
    Restart=on-failure
    SuccessExitStatus=3 4
    RestartForceExitStatus=3 4
    
    [Install]
    WantedBy=multi-user.target
    
  • create the tree structure required for the waptsync service to start:

mkdir /opt/wapt/.config/
mkdir /opt/wapt/.config/syncthing/
  • change the owner of the files:
chown -R wapt:www-data /opt/wapt/.config/
  • activate the waptsync service and start it. The configuration files will appear in the /opt/wapt/.config/syncthing/ folder:
systemctl enable waptsync
systemctl start waptsync
systemctl stop waptsync
  • change the listening port in the /opt/wapt/.config/syncthing/config.xml file:
<gui enabled="true" tls="true" debugging="false">
    <address>0.0.0.0:8384</address>
    <apikey>4jvEiL24UbFddsdsAQxqsfixNaLt</apikey>
    <theme>default</theme>
</gui>
  • start the waptsync service:
systemctl start waptsync
Configuring Syncthing’s web service

Syncthing’s web interface is now accessible by visiting http://<server_ip>:8384.

Operations to follow:

  • change the host name of the remote repository;
  • add a GUI authorized user;
  • add the password for the GUI authorized user;
  • check the box Use HTTPS for the GUI;
  • click on Save;
  • connect with SSH on the WAPT Server and restart the Syncthing service:
systemctl restart waptsync

Syncthing’s web interface is now only accessible with HTTPS on https://<server_ip>:8384.

  • in the list of shared folders, remove the default folder: Modify ‣ Remove;

  • configure the replication:

    Note

    Those actions must be run on WAPT Server.

    In the list of shared folders (Shares):

    • add a shared folder with Add a Share;
    • fill in the path to the directory to be shared, ex: /var/www/wapt/;
    • in the scroll-down menu Directory Type ‣ Only Send;
    • in the scroll-down menu File Recovery Order ‣ Older First;
    • redo the same operation for wapt-host: /var/www/wapt-host/;
    • add the remote repository to WAPT Server’s Syncthing:

      Once Syncthing has been installed on the main and remote repositories, recover the remote repository’s ID (Actions ‣ Show My ID).

      This unique identifier appears like

      DSINDDC-23ORDNM-PAK6FCL-ZJAKNCH-61GWXAT-77PC3JM-RZ4PPYP-K1QERAV
      

      On the main repository, in the list of remote devices (Other Devices):

      • add the remote repository with Add a Device;
      • fill in the ID of the remote repository;
      • tick the checkbox for the shared folder wapt and wapt-host;

      On the remote repository, follow these steps:

      • the remote device receives a Syncthing notification that it has been approved and added to the main device’s replication schedule;
      • the client receives a popup form asking to accept the synchronized shares wapt and wapt-host;

    The replication is now operational.

  • secure the replication:

By default, the following parameters are active in Syncthing:

Options Definition
Activate the NAT Use a UPnP port mapping for incoming synchronization connections.
Local discovery Syncthing will then broadcast to announce itself to other Syncthings.
Global discovery Syncthing registers on an external cloud service and can use this cloud based service to search other Syncthing devices.
Possible relay The use of relays allows to use external servers to relay the communications. The relay is activated by default but it will be used only if two devices can not communicate directly between themselves.

This operating mode simplifies the global setup but it is not the most advisable method in relation to security.

  • uncheck all boxes in network configuration;
  • define the listening port (by default port 22000);
  • replace default by tcp://0.0.0.0:22000;

Then go on the web interface of the remote repositories, click on Change and fill in the IP address of the remote repository:

  • replace dynamic by tcp://<remote_repo_ip_address>:22000;

This configuration is useful for limiting inbound connections to the Syncthing service.

Configuring the WAPT agents

WAPT clients on remote sites must now be configured to look for updates from their closest available repositories.

Two solutions exist:

  • use automatic detection via DNS SRV fields;
  • change manually or via a WAPT package the parameter repo_url in the WAPT agent’s wapt-get.ini file;

Configuration example of the WAPT agent - filled in local repository:

[global]
waptupdate_task_period=120
waptserver=https://srvwapt.mydomain.lan
repo_url=https://localrepo.domain.lan/wapt/
use_hostpackages=1

Example of a WAPT package designed to remotely change the repo_url in wapt-get.ini:

# -*- coding: utf-8 -*-
from setuphelpers import *

uninstallkey = []

def install():
  print('Modifier la configuration agent pour le site de Colmar')
  inifile_writestring(WAPT.config_filename,'global','repo_url',
                    'https://wapt.city02.mydomain.lan/wapt/')