Strategy Tester Review. Part 4 (Analysis of statistics and problems of backtest evaluation)
In previous reviews, you got acquainted with the principles of testing trading systems in MT4. The result of any testing is MT4 statistics (backtest or backtest), which are downloaded in several formats. It has two forms: numerical values and an equity curve graph. We will not go into the principles of mathematical and statistical analysis, since this question is individual. The purpose of this review is to tell you about what this or that backtest parameter means, that is, to provide a base with which you, at your discretion (according to your criteria), can evaluate the effectiveness of the advisor or indicator.
Decoding MT4 statistics
At the very bottom of the tester there are several tabs: “Settings” (window for direct data entry before testing), “Graph”, “Results”, “Report” and “Journal”.
- Schedule… Here is a graph of the change in the amount of the deposit (the so-called equity, deposit curve). If at the end of testing it is falling, then the EA is working at a loss. On the upward equity, it is important that there are no sharp drawdowns and jumps, which means that the EA works stably at any site.
This is the deposit curve of a grid advisor using Martingale, as evidenced by frequent small drawdowns.
A few tips:
- If no open position appears on the chart, there is an error. The error code is indicated in the “Journal”, the code number designation is on the MQL website (MT4 developer) in the “Documentation” tab (Reference).
- If the EA works in profit, but equity does not have a stable ascending line, try running it on separate sections, excluding segments with sharp jumps. Perhaps the EA works optimally only during certain hours or session.
- results… In this tab you can see which, when and how the advisor opened deals. It is easier to upload these MT4 statistics to Excel, so that you can then use filters and grouping to sort transactions by direction, date or profit. In the same tab, the intermediate balance is indicated after each transaction and the prices at which positions were opened and closed (including for pending orders).
- Report… Let’s dwell on this tab in more detail:
- The number of bars. The testing period, expressed in candles.
- Modeled ticks. If we assume that a candlestick is a sequence of 4 prices (High / Low, Close / Open), then ticks are price movements within the candle. The higher the value of the parameter, the more accurate the results, but also the longer the run of the advisor over the site.
- Modeling quality. MT4 statistics – 90%, the tester never shows more. If the value is less than 90%, there are problems in the quotes added to MT4 (gaps, failures, etc.). According to traders (from the forums), the accuracy of 90% is far from the real market and the value of this indicator should be at least 98%. Since MT4 does not allow you to rise to such a figure, you can use the Tickstory Lite program.
- Mismatch errors. They appear when simulating ticks at different time intervals. The reason is the difference between MT4 and broker quotes data. Green bar – no errors, gray areas – no quotes. The rest of the colors are intermediate options that can be ignored during rough testing. If there are gray areas, reload quotes and restart testing.
The rest of the key parameters have already been discussed in this review, we will make just a few clarifications:
- The minimum number of open positions in MT4 statistics is over 150. You will find the rationale for this figure in this review.
- Drawdown types: absolute – this is the difference between the initial deposit and its minimum value. Maximum – the difference between the maximum and minimum deposit amount.
- Expected value. It is calculated according to the formula “net profit / number of positions”. Indicated in monetary terms, but can be converted into points. If this parameter is less than 10 points, then the advisor closes profitable positions too early.
If you still have questions after reading, write in the comments. We’ll take a look at each backtest field.
Backtest (backtest) – the main MT4 statistics, showing the effectiveness of a trading advisor. Therefore, it is often faked by those who develop robots for sale or are going to raise money for trust.
How to spot fake MT4 statistics:
- HTML and HTM. The HTML format is more common, therefore, when faking a backtest, the file is saved in this format. When saving statistics from MT4, the HTM format is automatically set. And although this does not really matter, and you can manually change the format when saving, HTML is a signal that you should take a closer look at the statistics.
- Change formatting. Margins have been changed, there are gaps, shifts, etc. – the backtest was re-saved through different editors and possibly edited. Therefore, immediately ask not for backtest screens, but for the file unloaded from MT4.
- Extra characters. Run any Expert Advisor, unload statistics and compare two backtests for extra commas, slashes, etc. They catch the eye quickly.
- Identical marking (tickets) of different positions. Inconsistency of the sequence of numbers of deals and their opening times. Upload backtest to Excel and sort transactions by ticket numbers.
- Correspondence of the size of the spread to the volume of the transaction. Also checked in Excel.
After the Expert Advisor has been tested, but its results are unsatisfactory, the optimization stage begins. Read about it in the next part.