Seychelles-based cryptocurrency Exchange and derivatives Trading Platform BitMEX experienced two unrelated service outages on January 9th, according to a blog post by the exchange published today, January 11th.
The exchange wrote that luckily, “there was no data loss during these events and an additional safety mechanism to prevent a similar situation from impacting the trading engine has already been deployed.”
Additionally, “the root causes have also been identified and we are currently working on permanent fixes to prevent a recurrence.”
Here’s what happened.
Outage #1 Affected WebSocket API
The post said that at approximately 02:44 UTC, the exchange’s WebSocket API “saw a degradation in performance for a minute where 7% of commands sent by clients failed.” After the minute passed, “connections continued undergoing a 1% failure rate of commands until servers recovered at 02:47:00 UTC.”
The “rolling restart” of the servers that ended the outage may have also caused a temporary delay in response times for some of the exchange’s market data REST points.
Two More Outages Affected BitMEX’ Trading Engine
The second pair of outages lasted for 30 seconds apiece and took place starting almost exactly three hours later at 05:48 UTC and 06:10 UTC, respectively.
The exchange’s post explains that during these 30-second periods, requests to the trading engine were “load-shed” by the trading engine, which was busy at the time. Load-shedding is when action is taken to reduce the load on something; the term usually refers to the interruption of electricity to prevent an overload on the generating plant.
According to BitMEX’ post, the trading engine’s preoccupation was not because of a surplus of traffic. Instead, the engine was overloaded “due to data replay complications during a regularly scheduled market data distribution component restart.”
The WebSocket API was also affected once again as a side-effect of the overload.
The outages come just a couple of weeks after BitMEX experienced another temporary outage for approximately one minute on December 27.
On 27 December at 18:38 UTC, BitMEX experienced a minor outage for approximately 1 minute whereby the trading engine was unavailable. We apologise for the inconvenience. For more info: https://t.co/zCcCKebrHx
— BitMEX (@BitMEXdotcom) December 27, 2018
Because most cryptocurrency exchanges are available round the clock, outages are a normal part of the game--as long as they don’t happen too frequently.
BitMEX CEO and co-founder Arthur Hayes recently said on an episode of Laura Shin’s Unchained that he believes that the 24-hour trading that’s available on BitMEX and throughout the cryptosphere is having an effect on other trading industries: “I think 24/7 trading of all different types of assets is something that’s going to be the future, and that will bleed into other markets we’re all familiar with, you know, FX, fixed income, and equities.”
BitMEX CEO Arthur Hayes.
At the time of writing, BitMEX’ 24-hour trading volume was $1,327,276,079.
