While evaluating research proposals at a recent PSSP meeting, I realized that the level of knowledge about current econometric methodology both among researcher AND among those who evaluate -- is ABYSMALLY low.
Now that I am VC PIDE, an adequate level of understanding of econometric methodology will automatically be expected from researchers. I would like to provide some training along these lines on a weekly basis.
The FIRST place to start is the Davidson-Hendry-Srba-Yeo (DHSY) paper on the estimation the consumption function in England. This is my dividing line -- if you understand this paper, you can call yourself an econometrician, otherwise not. This is BY NO MEANS the state of the art -- it was written forty years ago, in 1970's. BUT it does provide the minimal level of understanding required to do sensible econometrics. Unfortunately, I have rarely seen applied work which conforms to this minimal level of understanding.
I have created a website where I plan to do a detailed explanation of all the issues discussed in the paper. The truth is that the paper is highly compact -- a lot of deep ideas are condensed into a very brief text, so it requires a HASHIA -- explanations written on the margin. I and my students have finished the first section, which is available on the website DHSY
I would like to explain the Hendry methodology in the context of a critique of real paper which are submitted to our journals. I am attaching a few extracts from a recent submission which was rejected.
The paper explores the relation between Economic Growth (EG) and FDI. After doing the usual integration cointegration analysis, author finds that EG and FDI are both stationary, and they are also co-integrated, so have a long term relationshipl. Then he runs a Vector Error Correction model following the Engle-Granger two step procedure -- first step is to save the errors from the cointegration equation, the second step is to run current EG and FDI on lagged errors. The results are displayed in the extract from the paper, which is available from the links below. The SECOND (and final equation) estimated in the paper is a regression Log (EG) on Log (FDI ) and some other variables -- you are asked to PROVIDE a CRITIQUE of this SECOND equation on the basis of your understanding of the first section of DHSY
Please write your critique on THIS BLOG -- ALL are welcome to write there own critiques -- I will give a prize to the best critique -- I will also provide additional hints and information as we go along.
This is ad-hoc model. Construction of econometric model must be based on some economic theory. Theory is missing in this model.
ReplyDelete Proposed model does not based on any existing model or at least author did not refer.
Econometric model based on some strong assumptions, for example normality of residuals, linearity of parameters. Author has ignored these assumptions.
Specification or functional form of the model is also critical issue in this paper.
Compatibility of assumption of the model with properties of data is questionable in this paper.
There is outlier in FDI variable which may affect the results.
Potential endogeniety problem because FDI is not a true exogenous variable
Muhammad Javid
Research Economist
These are all GOOD critical comments
ReplyDelete1: AD-HOC model, not based on theory
2: Author does not refer to existing/previous models
3: Regression models are based on strong assumptions
4: Functional Form -- whether it should be log log, linear, or other
5: Is the model compatible with the data?
6: There is an outlier in FDI -- actually, the data shows a regime change -- for a small period of time there was a large inflow of FDI -- this is actually very useful in detecting causal effects -- as we will discuss later -- insha Allah
7: Causal direction of relationship is not clear -- for example, it could be the case that high GNP growth pulls in FDI. Or other types of causal sequences are also possible
However, in relation to SECTION 1, the main issue of interest is CRITICISM 2: The author does not compare his result to any previous results in the literature -- So our second question is WHY DOES THIS MATTER? What difference does it make whether the author considers other models and compares and evaluates his model to these others?
NOTE that I have uploaded the GDP and FDI data from WDI on Pakistan onto the DHSY website so that those who want to replicate the results can do so – though I do not have the export and literacy variable – if someone can get these too, it would be useful.
Dealing with second question "WHY DOES THIS MATTER?" i-e to compare with other models. Since we GDI and GDP are observationally equaivalent and true model is hardly known, therefore, there are many candidate models for this data. Which of these candidate models is the best depend on our objective of modeling- are we modeling to forecast or modeling to study causal behavior. A single model cant be suitable for all the objectives. So comparing it with other models is important as there is no need to highlight that the model which is selected not only explains the behavior but also able to explain the behvior of other competing models. THis is a concept of Encompassing which is sometimes same as nesting but other times it is different than encompassing. So a model which explains FDI and GDP behavior and is superior to other models will be considered as a useful model.
DeleteZahid has pointed in the right direction. However, deeper understanding results from making your ideas CONCRETE and specific. Furthermore, in the first section, the concept of encompassing has been hinted at but not introduced. The criticism being made here is simpler than the more difficult idea of encompassing. I would like students to consider the following two articles, also about growth, and THEN EXPLAIN how taking these two articles into account would affect the analysis being done in this paper --
ReplyDeleteI Just Ran Two Million Regressions by Sali-i-Martin
http://www.ecostat.unical.it/aiello/didattica/economia_Crescita/CRESCITA/CRESCITA_Sala-i-Martin-AER-1997.pdf
We Ran One Regression by DF Hendry and HM Krolzig
http://www.nuff.ox.ac.uk/economics/papers/2004/w17/OneReg.pdf
Above is the expert and well articulated critique. I have tried it as
ReplyDelete Missing variable bias can be observed in FDI-Growth model. As missing relevant variables can make irreverent variables more significant as results are shown by the author. Similarly, putting extra variables in the model cannot affect the significance of relevant variables. So General to specific modelling can be one of the option. A general unrestricted model of potential variables can be selected and then it can be transformed and reduced in size by performing different restrictions tests. By doing so economically and statistically viable model can be chosen.
However, in case of growth model, formulating general model is a rigorous exercise. As relevant variables impacting growth can exceed than the number of observations. So encompassing methodology can be considered best to handle this issue.
Moreover, author has neglected all other models illustrating the same kind of relationship as one can’t estimate a single model (even the selected model is based on some theory) depicting some relationship without comparing it with other existing models.
The best suited or most popular models which include variables of author’s interest can be chosen for encompassing. Parameter or forecast encompassing can be sought as per nested or non-nested models. Encompassed model can be selected and estimated to know the existence of actual relationship b/w variables.
Mamona writes that:
ReplyDeleteMoreover, author has neglected all other models illustrating the same kind of relationship as one can’t estimate a single model (even the selected model is based on some theory) depicting some relationship without comparing it with other existing models.
The best suited or most popular models which include variables of author’s interest can be chosen for encompassing.
This is true, and exactly what I am trying to explain HOWEVER, the point is much better understood with the help of a SPECIFIC model -- READ the two papers listed in my previous comment, and then explain this point by comparing the models discussed.
By going through the suggested papers I came to know following facts:
ReplyDeleteInstead of lubricious work of running many regressions to find out significant variables, it is good to consider G to S methodology. To employ general to specific modelling for the selection of appropriate model from the bunch of models, algorithm(making comparison) may be considered as best option. By doing so a specific regression model can be chosen automatically (as 'We Ran One Regression' suggests) through comparing different models on the basis of set of statistical tests. An automatic selection program PcGive/PcGets is available in OxMetrix where one click can help to choose suitable model and save a lot of time.
P.S: I wonder if this procedure can help to select model based on both statistical and THEORETICAL grounds.
Please pick out MODELS estimated in the papers referenced --
ReplyDeleteWHAT are the variables that are chosen to explain growth? Are the variables considered by this author present in the models that are analyzed by Hendry-Krolzig and by Sala-i-Martin? Are any model comparisons made by these two authors?
SEPARATELY -- go through the Literature Review in the original paper under discussion -- PICK out models that are SUGGESTED by the literature review
If you investigate what brings taste in a Biryani, you can cover all important things in one writeup and result will be only one paper. If you want to write many papers from the same problem, you can write in this way:
ReplyDeletea. effect of basmatti rice on biryani
b. effect of ordinary rice on biryani
c. effect of iodized salt on birayni
d. effect of dalda cooking oil on biryani
e. effect of tullo cooking oil on biryani
e. what makes a good biryani: comparative analysis of tullo and dalda cooking oil
f. effect of black peper on biryani
g. effect of red chili on biryani
h. appropriate time for cooking a biryani
In similar way, you can choose hundreds of titles for which you can write separate papers. And for each paper, you can have very good justification like 'no one earlier has has tested effect of red chili on biryani'
However, you can easily see this kind of analysis is strictly conditional on 'all other things remain same alias ceterus peribus'. If you are analyzing the effect of red chili on biryani, you have to ensure all other things remain same but for some reason amount of salt changed in you recipe; would you be able to conclude something reasonable about your research problem?
Think of Economic growth in similar way: No doubt the growth is much more complex than the recipe of a good biryani. Do you think you can analyze the effect of single variable e.g. FDI in isolation without taking any care of ceterus peribus or controlling the effect of other variables?
Therefore a reasonable analytical methodology must take care of control variables even if they are not directly related to your problem. While analyzing impact of FDI on economic growth, you can never ignore Human Capital on the basis that "I do not intend to analyze the effect of Human capital, my task is to analyze the effect of FDI on growth"
Similar recommendations for research were given by DHSY in their seminal paper, however, these recommendations were 'kept under red tape' for the wider interest of academic community'. This is because if DHSY is followed, dozens of questions would be solved by a single paper and the CV of the academics will get shorter. In addition to that, the business of academic journals will go in deficite
Atiq has made a delicious comment -- note that you cannot discuss the question of how much salt to put into Biryani without taking into account everything else that is going to be put in -- One has to look at the big picture.
ReplyDeleteComing back to the regression equation, the question is: What is the final model recommended for growth in the two papers? How does it impact on the analysis in the paper under discussion?
If we analyze this project in the light of DHSY, the following points could be observed
ReplyDeletea. First requirement DHSY recommended for a new model is that: "we consider it an essential (if minimal) requirement that any new model should be related to existing model" . In the paper under discussion, the author is first estimating model without log transform and shows that it is sufficiently good by testing the co integration etc. After that, author go for estimating another model with log transform without relating it to the previous model estimated. DHSY see this kind of practice as a reason for proliferation of models.
Second recommendation of DHSY is: "to avoid directionless “research” and uninterpretable measurements, a theoretical framework is also essential". The authors of this paper do not provide any theoretical frameworks for any of their model. This leads to adhoc model construction.
Third recommendation by DHSY is "to be empirically acceptable, an econometric model obviously must account for the properties of the data " I do not see any thing to investigate match between the assumption of the model and empirical properties of data, in both kind of models i.e. with and without log transform. For example, the assumption of an econometric model is that residuals should not have autocorrelation but there is no test to insure that this assumption is valid for the data under consideration.
DHSY assume that the econometric modeling as an attempt to match the data with the hypothetical properties of assumed data generating process. This means that you should search for a new modeification of model untill you get a model such that all of the assumptions of the model are valid. I do not see any attempt of this kind in the extract of paper provided.
Therefore the paper ignores almost all of the important recommendations made by DHSY
Now I analyze how the paper under consideration could be related to the papers ‘I ran 4 million regressions’ by Sala-i-Martin and ‘I ran one regression’ by Hendry and Doornik.
ReplyDeleteThough the two papers differ a lot with respect to their methodology, they are assumed to solve one problem. The two papers find 67 determinants of growth from existing literature and try to find how they can choose most appropriate determinants of growth these 67 variables. On the other hand, the paper under consideration gives absolutely no consideration to these variables, instead, takes few variables by his/her own choice and runs regression. Therefore this papers has no resemblance with that of Sala-i-Martin and Hendry. On contrary, this paper chooses arbitrarily few variables to include in his model, therefore introduces a selection bias
Last, the author of this paper ignores variables found significant by both Hendry and Martin, this implies that paper is suspected for serious bias caused by missing variables. This kind of regressions lead to spurious significance of irrelevant variables.
ReplyDeleteSala-I-Martin in his paper "I Just Ran Two Million Regressions" have a total of 62 variables. According to him a typical growth regression in the regression has at least seven right-hand-side variables.
ReplyDeleteHe estimated the model as:
γ=α_j+β_yj y+β_zj z+β_xj x+ε
This model combines some variables which appear in all regressions (y), the variable of interest (z) and the trio (xj) taken from the pool of remaining variables.
The final model contain the variables:
Dependent Variable= Growth
Independent Variables:
y= log (GDP per capita 1960),
life expectancy in 1960,
primary school enrolment rate in 1960.
"22 variables out of 59 appear to be significant "
z= Fraction of the population that follows the Confucius Religion.
x= Regional variables, political variables, Religious variables, Market distortions and market performance, types of investment, Primary sector production, openness, types of economic organization and former Spanish colonies
Hendry and Hans-Martin Krolzig in his paper " We Ran One Regression" showed that only one regression is needed, the GENERAL UNRESTRICTED MODEL by adopting General-to Simple approach. The Model selected :
Dependent variable: growth
Independent Variables:
Number of years open economy, Equipment Investment, Fraction of the population that follows the Confucius Religion.
Most of the variables were from Barro and Jong Wha LEE (1993) data set, considered by Sala-I-Martin in his paper.
Now that the other models are on the table, we can illustrate in a very clear way, what is wrong with the analysis of this model. NOTE THAT NEARLY ALL PAPERS that I see with regression models, and many M Phil and Ph.D. are written in the same way and therefore subject to the same criticism. I am putting this critique on the WEBSITE. Please read and digest and try to comprehend. THEN I will put another paper up for similar treatment, and ask readers to critique it, just to ensure that the message is understood. Please look at the link below for the critique:
ReplyDeletehttps://sites.google.com/site/hendrymethod/1-introduction/fdi-growth
1) OLS assumptions like normality, and homoscedastic variacne of error term are not checked.
ReplyDelete2) Proper tool for detecting outliers are not used.
3) Robust standard errors are given but it is not showed that when the data having outliers especially in the direction of explanatory variable OLS produces very poor estimates of the regression coefficients. The best way is to compare your OLS results with different robust procedures (LTS, LMS, LAD, M-estimation, Redescending M-estimation), author missed this point.
Muhammad Ali
Assistant Professor of Statistics
Higher Education Department, KP, Pakistan.
Testing the assumptions of models is indeed a KEY component of the Hendry Methodology, and Muhammad Ali has pointed out important defects in the paper. Our goal here is to UNDERSTAND the DHSY Paper, and also the Hendry Methodology as it is demonstrated by example in the DHSY Paper. Therefore, critique should be made as an interpretation and explanation what is written in DHSY -- For example, the following quote from the last paragraph in section:
ReplyDeleteThat the general model is not obtained by every investigator seems to depend on the operation of (self-imposed) constraints limiting the range of specifications, estimators, diagnostic tests, etc., which are employed. Such arbitrary and unnecessary constraints can play a large role in determining the final equations selected a
The context is that we are trying to find out why everyone gets different models for the consumption function. So one of the factors mentioned among the possible reasons is "diagnostic tests" -- this is not the focus of the current section, but it is mentioned here. Later on it will be emphasized. The main focus of the current section is the process of encompassing -- comparing the model given to the others available in the literature.
AFTER we have gone through the DHSY paper, I will explain the deficiencies in the Hendry methodology, and what must be done to overcome these -- this is not at all well known.
For the moment, I would just like to REPLICATE the critique on ANOTHER sample paper, which studies the relationship between Intellectual Property Rights and Growth: Relevant excerpt of econometric analysis is given on the link below, and also the full paper is attached:
https://sites.google.com/site/hendrymethod/1-introduction/fdi-growth
Sir, while first phase of discussion is going to close, people are interested to know the best discussant and his prize
DeleteMost relevant comments were made by yourself. I am counting you among the co-teachers, so not eligible for prizes. Most other comments were correct criticisms, but did not relate to the issue of encompassing, which is the key idea of section 1. However, people get a second chance on the second article IPR and growth. This time it should be easier as the pattern and explanation has already been given. All that is needed is to replicate things which have already been said.
DeleteWe are not closing the first section as yet -- Not until after receiving valid encompassing based criticisms of the second article. Also, there are few more things that remain to be clarified in the context of section 1 of DHSY. I think I will set up a quiz after this part is done, and then we can give prizes to the highest score on the quiz
In this study it is not clear that whether a new model is proposed or an existing model is used for Pakistan. If they are using an existing model then why they have not compared their results with existing model and if they are proposing a new model then they have not stated that why their model is better than the existing models. They are using many segments of theories and combining these segments in a single model. The question is “whether they have taken all of the existing theories to form a General model or not?” After estimation of the model they have not tested for underlying assumptions. If the underlying assumptions are not valid then our data does not support the model.
DeleteThe selection of variables are adhoc and no firm grounds provided. if the author selected the model on the basis of any theory which I think not the case here, he/she should describe how all of these variables form a GENERAL Model and Encompasses the previous models. the author did not provide any justification about the functional form he/she used in his/her model and also he/she ignored to tests the underlying assumptions of the model.
ReplyDeleteIPR do not directly affect GDP. IPR affect R &D, Innovation and hence Production and Investment. IPR work through the Law enforcement institutions of the country.
ReplyDeleteIPR, FDI, EFW, TRADEOP, SYR15 determine the GDI. More over IPR, EFW, TRDEOP measure almost the same instruments. Similarly SYR15 and POPgrowth are related variables. There are chances of strong multicollinearity among these variables.
First of all the purpose of a study must be clear.
Following DHSY a General model including all the variables, guided by the theory, empirical evidence and data should be included in the model.
Then using General to Specific methodology the insignificant variables should be dropped. Only the statistically significant variables should be retained in the final model.
These steps are totally ignored in this study. The results of the present study are also not compared with the earlier studies.
In DHSY Author compare three models that have the same data set on consumption and income to find the common elements, and re-estimated a general model that encompass the three model in a single model. They followed G to S methodology and tested the underlying assumption to validate the General model.
ReplyDeleteIn the article of FDI and Growth model, author suggested two models that have some issues as compare to DHSY methodology. The following are the critique points:
i- The model selection is on ad-hoc bases.
ii- The underlying assumptions have not tested.
iii- In second model author skip the step to test the long run relationship between Human capital, FDI and Growth that may be caused the spurious regression as author tested in first model, which is against the validity of the model.
iv- Author use literacy rate as a proxy of human capital that may not be support all the characteristic of Human capital, because literacy rate is one of the ingredients of human capital, So on the base of that proxy variable author conclude the validity of model.
Waqar Khan comments:
ReplyDelete1. Ad-hoc selection of variables === CORRECT and RELEVANT
what does this mean? It means that anyone can come up with any other model -- I can take out many of the variables and introduce any other variable that I like -- as discussed in Million Regressions, there are 63 variables of interest for explaining growth. SO Ad-hoc variable selection is almost sure to lead to wrong model.
2: NO THEORETICAL Basis == VALID comment. If a model comes from theory then the variables are theoretically justified. For example, in consumer theory, income is a determinant of consumption.
3. For selection of AD-HOC variables, the encompassing approach is recommended since it picks the best variable combinations out of a huge set. NOTE that if we have 60 variables than we have 2^60 possible models this is 1E18 or 1 followed by eighteen zeros which is 1 QUINTILLION models.
all three are good comments related to the DHSY methodology. Note that this is meant to teach you how to DO econometrics, since current methods in use by most authors are just simply wrong. I will get to other comments later.