Measuring Consistency in Text-based Financial Forecasting Models

by   Linyi Yang, et al.

Financial forecasting has been an important and active area of machine learning research, as even the most modest advantage in predictive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the opportunity to leverage textual data, such as earnings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models – their invariance under meaning-preserving alternations in input – is a crucial property for building user trust. Despite this, current financial forecasting methods do not consider consistency. To address this problem, we propose FinTrust, an evaluation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP models for financial forecasting is poor. Our analysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suitable for robustly predicting market information. All resources are available on GitHub.


page 1

page 2

page 3

page 4


NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting

Financial forecasting has been an important and active area of machine l...

Financial Market Trend Forecasting and Performance Analysis Using LSTM

The financial market trend forecasting method is emerging as a hot topic...

Financial data analysis application via multi-strategy text processing

Maintaining financial system stability is critical to economic developme...

Text analysis in financial disclosures

Financial disclosure analysis and Knowledge extraction is an important f...

Investor Behavior Modeling by Analyzing Financial Advisor Notes: A Machine Learning Perspective

Modeling investor behavior is crucial to identifying behavioral coaching...

How is Machine Learning Useful for Macroeconomic Forecasting?

We move beyond "Is Machine Learning Useful for Macroeconomic Forecasting...

Please sign up or login with your details

Forgot password? Click here to reset