My trading bot just ceased this morning from its loyal 24/7 service. It’s running on an Amazon EC2 server with Ubuntu 16.04 and I’m sure this time I’m not having an unpaid-bill issue any more. After some time digging I think I finally figure out the cause of this unexpected strike – asynchronism.

Asynchronism, or in simple terms, timing discrepancy, usually means a tiny bit of difference between local time on your computer/server and the global NTP time. It can be as undetectable as several milliseconds but in some applications like trading, such discrepancies are reckoned intolerable and any request sent from such computers/servers are ruthlessly rejected. Computers are just machines and they cannot be accurate in time forever. That’s why we need (time) synchronization. In fact, EC2 does have such regular synchronization built in, but it seems it only happens once after a rather long period, like days. In order to adjust the synchronization period length to avoid similar issues in the future, I’ll need the Amazon Time Sync Service.

First we install the chrony package for synchronization, and open its configuration.

sudo apt install chrony
sudo vi /etc/chrony/chrony.conf

Append in the opened chrony.conf file a line as follows.

server 169.254.169.123 prefer iburst

Restart chrony service.

sudo /etc/init.d/chrony restart

[ ok ] Restarting chrony (via systemctl): chrony.service.

Make sure that chrony is successfully synchronizing time from 169.254.169.123

chronyc sources -v

210 Number of sources = 7

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 169.254.169.123               3   6    17    12    +15us[  +57us] +/-  320us
^- tbag.heanet.ie                1   6    17    13  -3488us[-3446us] +/- 1779us
^- ec2-12-34-231-12.eu-west-     2   6    17    13   +893us[ +935us] +/- 7710us
^? 2a05:d018:c43:e312:ce77:6     0   6     0   10y     +0ns[   +0ns] +/-    0ns
^? 2a05:d018:d34:9000:d8c6:5     0   6     0   10y     +0ns[   +0ns] +/-    0ns
^? tshirt.heanet.ie              0   6     0   10y     +0ns[   +0ns] +/-    0ns
^? bray.walcz.net                0   6     0   10y     +0ns[   +0ns] +/-    0ns

where ^* denotes the preferred time source.

Finally, check the synchronization report.

chronyc tracking

Reference ID    : 169.254.169.123 (169.254.169.123)
Stratum         : 4
Ref time (UTC)  : Thu Jul 12 16:41:57 2018
System time     : 0.000000011 seconds slow of NTP time
Last offset     : +0.000041659 seconds
RMS offset      : 0.000041659 seconds
Frequency       : 10.141 ppm slow
Residual freq   : +7.557 ppm
Skew            : 2.329 ppm
Root delay      : 0.000544 seconds
Root dispersion : 0.000631 seconds
Update interval : 2.0 seconds
Leap status     : Normal    

As a conclusion, the server is now synchronizing time to the assigned source every 2 seconds and we shall never encounter similar issues.