EcomDev Logo

Building Cluster Application

In pure PHP for Magento 2

Once upon a time...

A customer had issues with Magento 2 webshop...

About customer

  • Fashion retailer
  • Magento CE 2.2
  • Heavy use of newsletter campaigns
  • Sells internationally
  • A lot of last pieces

Customer Issues

  • Random Deadlocks During Checkout
  • Stock updates from ERP were "loosing" qty of not yet exported orders
  • Customer had to perform complete stock re-count every month
  • It was easy to abuse system by creating pending orders and depleting stock for last in stock items

The source of all evil is...

...quirks of CatalogInventory implementation

Random Deadlocks During Checkout

SELECT ... FOR UPDATE on every sale

Price & Stock re-indexation on product changing stock status

Inconsistentency in Stock Data

Magento 2.2 has only one qty field - that is the single truth for the whole process

Magento MSI has multiple fields, but has a race condition that allows to bypass qty check

CatalogInventory Behaviour Stock Item Product 1 Stock Item Product 2 Visitor Places Order Locks all ordered items on read Validates Stock Item Quantities are deducted from Stock Items Reindexes Price & Stock Index If item becomes out of stock
MSI Behaviour Stock Source Product 1 Stock Source Product 2 Visitor Places Order Receive available qty + aggregate all current reservations Validates Qty Creates Negative Reservations

MSI clears all history of mutations


public function execute(): void
{
	$connection = $this->resource->getConnection();
	$reservationTable = $this->resource->getTableName('inventory_reservation');

	$select = $connection->select()
		->from(
			$reservationTable,
			['GROUP_CONCAT(' . ReservationInterface::RESERVATION_ID . ')']
		)
		->group([ReservationInterface::STOCK_ID, ReservationInterface::SKU])
		->having('SUM(' . ReservationInterface::QUANTITY . ') = 0');
		$connection->query('SET group_concat_max_len = ' . $this->groupConcatMaxLen);
	$groupedReservationIds = implode(',', $connection->fetchCol($select));

	$condition = [
		ReservationInterface::RESERVATION_ID . ' IN (?)' 
			=> explode(',', $groupedReservationIds)
		];
	$connection->delete($reservationTable, $condition);
}

So I had 3 different options to fix customer's issue

Completely disable stock processing

Ping ERP on every stock calculation

Build an async PHP application cluster with EventSourcing

Clustered ES Application Timeline Epoch 1 Events Ordered SKU1 UpdateItem SKU2 Adjustment SKU1 Returned SKU3 Shipped SKU2 Cancelled SKU4 .... Epoch 2 Events Ordered SKU1 UpdateItem SKU2 Adjustment SKU1 Returned SKU3 Shipped SKU2 Cancelled SKU4 .... Epoch 3 Events Ordered SKU1 UpdateItem SKU2 Adjustment SKU1 Returned SKU3 Shipped SKU2 Cancelled SKU4 ....

Database Design

Epoch


+------------+------------------+------+-----+---------+----------------+
| Field      | Type             | Null | Key | Default | Extra          |
+------------+------------------+------+-----+---------+----------------+
| epoch_id   | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| started_at | datetime         | NO   | MUL | NULL    |                |
| ended_at   | datetime         | YES  |     | NULL    |                |
+------------+------------------+------+-----+---------+----------------+

Epoch Events


+--------------+------------------+------+-----+---------+----------------+
| Field        | Type             | Null | Key | Default | Extra          |
+--------------+------------------+------+-----+---------+----------------+
| event_id     | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| occured_at   | datetime         | NO   | MUL | NULL    |                |
| event_type   | varchar(255)     | NO   |     | NULL    |                |
| aggregate_id | varchar(255)     | NO   | MUL | NULL    |                |
| event_data   | text             | NO   |     | NULL    |                |
+--------------+------------------+------+-----+---------+----------------+

Aggregate


+------------------+---------------------+------+-----+---------+-------+
| Field            | Type                | Null | Key | Default | Extra |
+------------------+---------------------+------+-----+---------+-------+
| aggregate_id     | varchar(255)        | NO   | PRI | NULL    |       |
| last_event_id    | int(10) unsigned    | NO   |     | NULL    |       |
| last_event_epoch | int(10) unsigned    | NO   | MUL | NULL    |       |
| qty_reserved     | int(11)             | NO   |     | 0       |       |
| qty_on_hand      | int(11)             | NO   |     | 0       |       |
| is_in_stock      | tinyint(3) unsigned | YES  |     | 0       |       |
+------------------+---------------------+------+-----+---------+-------+
Async-Stock-Service Commands Events H T TP API A sync PHP S e r v er Command Processor A sync PHP S e r v er Dist r ibu t es c ommands based on c onsis t e n t hashing algo r ithm Single C ommand S t ock S e r vi c e W o r ker Node M age n t o F r o n t end Node E v e n ts & A g g r eg a t es Single E v e n t S t ock S e r vi c e DB Node

Choosing Responsible Worker

  • Hash SKU of an item
  • Spread hash by the number of workers
  • Every worker connects to each other
  • Each worker uses named MySQL lock to prevent race condition

Results

  • Zero deadlocks related to stock in the server log
  • Customer didn't complain about stock inconsistency
  • Single write worker for each stock
  • Supports 3000 concurrent stock deductions per worker

Built With

ReactPHP HTTP Server

https://github.com/reactphp/http

ReactPHP MySQL connector

https://github.com/friends-of-reactphp/mysql

Thank You

Q&A