Magento 2.0

Indexation techniques for high performance

by Ivan Chepurnyi

What is a Magento Index?

Magento Index

• Represented as a separate table
• Optimized for retrieval, filtration and/or sorting.
• Just a snapshot of data at some point in time.

When to create it?

Data Filtration

• Product Visibility
• Product Status
• Category Relation

Data Sorting

• Category Position
• Bestsellers
• Stock Quanttities

Complex Logic Cache

• Stock Availability
• Product Price
• Anchor Category Relation

When do NOT to use them?

For Data Retrieval

It does not make sense to flatten EAV structure for retrieval

When Accurate Data is Critical

Using price or stock index in shopping cart is a bad idea

High Availability Indexation Challanges

Table locks

Index generation can create a lock on table you don't expect

UDPATE some_index_table index 
    INNER JOIN catalog_product_entity_varchar product_data ON ... 
    	SET index.some_value = product_data.value
           

Solution

• Avoid cross table updates on live tables
• Create snapshot of data for your index

MySQL Memory

MySQL memory consuption is increasing with amount of records you need to update.

These are known MySQL serial killers:

INSERT INTO .. SELECT ... FROM

UPDATE ... JOIN ... SET ...

Solution

Range your updates into smaller data-sets

Index Downtime

During re-indexation there is a step when index table is locked and no data is available in it.

Solution

• Create two identical index tables
• Re-index only one at a time
• Switch one to use on the frontend via configuration model

Did anyone created an indexer in Magento 1.x?

It was a nightmare comparing to Magento 2.0...

It is now incredibly simple...

Only two configuration files required, and no event matching magic:

indexer.xml and mview.xml

mview.xml

Allows you to subscribe to changes in any table

Magento\Framework\Mview\ActionInterface

<view id="mview_id" class="MviewClass" group="indexer">
    <subscriptions>
        <table name="table_to_check" 
                  entity_column="primary_key" />
    </subscriptions>
</view>

Why is it called MView?

indexer.xml

Just a regular Magento indexer definition

Magento\Framework\Indexer\ActionInterface

<indexer id="indexer_code" view_id="mview_code" 
	class="YourIndexerClass">
    <title translate="true">Your Title</title>
    <description translate="true">Your Description</description>
</indexer>

So how to implement HA index in Magento 2.0?

I really like re-usable components

So I come up with such indexer structure

Indexer

• Generic interface for usage in MView and Indexer Action
• Has knowledge about the index structure
• Creates index conditions for processor

Sample Interface

interface IndexerInterface
{
    public function executeFull();

    public function executeByIds(array $ids);
}

List Generator

• Creates fundation for our indexer
• Consists mostly from entity identifiers

Sample Interface

use Magento\Framework\Db\Ddl\Table;

interface ListGeneratorInterface
{
    /**
     * @return Table
     */
    public function generate(ConditionInterface $condition);
}

Index Condition

• Decouples creation and applying of condition

Sample Interface

use Magento\Framework\DB\Select;

interface ConditionInterface
{
    public function apply(
        Select $select, 
        $alias, 
        array $columnMap = []
    );
}

Data Generator

• Creates final subset of index data
• Data is based on List Generator data

Sample Interface

interface DataGeneratorInterface
{
    /**
     * @return Table
     */
    public function generate(Table $listTable);
}

Index Processor

• Recieves list of index conditions from indexer
• Decides how to populate an index table
• Toggles index table via metadata instance

Sample Interface

interface ProcessorInterface
{
    public function process($conditions, $live = false);
}

Index Metadata

• Controls which table to use for indexer and which for frontend
• Should be used in your entity repository to filter data by it
• Provides usefull attribute information

Sample Interface

interface MetadataInterface
{
    public function getTableToIndex($live = false);

    public function getIndexTable();

    public function toggleTable();

    public function getAttributesInfo($entityType, $attributeCodes);
}

Real Index Example

Created in 3 hours during the workshop on the first #DevParadise day

http://bit.ly/m2-boilerplate

Want to learn more about these techniques?

Catch me at coffee break or during after party!

Thank You

Q&A

@IvanChepurnyi

ivan@ecomdev.org