- HOME
- Reports
- Simulation of infection spread and control
- Deviation between the basic scenario and actuals
Deviation between the basic scenario and actuals
/
-
Graduate School of Economics, Faculty of Economics, The University of Tokyo
Prerequisites
■Periodic validation of past analyses is desirable for effective use of model analysis in policy
■Verification of past analyses can improve the quality of the current analyses
■The assumption that verification will be conducted in the future gives discipline to the current analyses
■Accountability associated with analyses used as policy inputs can be fulfilled
■The accumulation of verifications can be reference materials for future crises
■There may be implications with reference to crises other than the current pandemic
Analysis
■Self-validating analysis by the Fujii-Nakata team
■(August 17) Past outlook verification,https://covid19outputjapan.github.io/JP/files/FujiiNakata_PastProjections_20210817.pdf
■(August 20) Impact of the Olympics on infections: Review,https://covid19outputjapan.github.io/JP/files/FujiiNakata_OlympicsReview_Slides_20210820.pdf
■(October 25) Factors behind the infection decrease in Tokyo: Quantitative analysis,https://covid19outputjapan.github.io/JP/files/FujiiNakata_SharpDecline_Slides_20211025.pdf
■A record of "conditional" predictive accuracy, https://covid19outputjapan.github.io/JP/nationwide.html
※Examples of Spaghetti Charts

Caldara et al. (2021):Monetary Policy and Economic Performance since the Financial Crisis
・https://www.federalreserve.gov/econres/feds/files/2020065pap.pdf
※Examples of Spaghetti Charts

Duarte et al. (2021): Strengthening the FOMC’s Framework in View of the Effective Lower Bound and Some Considerations Related to Time-Inconsistent Strategies
https://www.federalreserve.gov/econres/feds/files/2020067pap.pdf
See also Nakata (2020): Raising the Inflation Target: Lessons from Japan
https://www.federalreserve.gov/econres/notes/feds-notes/raising-the-inflation-target-lessons-from-japan-20200108.htm
Analysis
■The periodic "outlook" by Fujii-Nakata team
■Updated weekly from January 21, 2021 to August 31, 2021
■A projection for the number of seriously ill patients has been updated from May 25, 2021
■Bi-weekly in September, bi-monthly from October
■September 14, September 28, October 12, November 2, and December 21 (tentative)
■Disseminated to the public and people in the policy field throughout 2021
■https://covid19outputjapan.github.io/JP/media.html
■Subcommittee, MHLW Advisory Board, Prime Minister's Office, Council of Ministers, Office for COVID-19 and Other Emerging Infectious Disease Control of Cabinet Secretariat, Olympics Expert Round-table conference
■It is not always productive to evaluate the value of scenario analyses solely on the basis of deviations from realized values. However, since the Fujii-Nakata team has been aiming to "conduct scenario analyses in a relatively realistic context," it is important to record the deviation between the presented scenarios and the realized values
Transition of Method for Setting Up the Base Scenario
■From January 21, 2021 to July 27, 2021 (Regime A)
■Assume the fluctuation of the number of newly infected people during the state of emergency declaration
■After the declaration was lifted, the contact rate parameter (before adjusting for economic activity) was an average over the previous 17 weeks
■Adjustments were made based on overall judgment as needed
■Until mid-June, the role of overall judgment was limited in the base scenario
■Until mid-June, a more realistic scenario is presented as the risk scenario
■Example: The "lockdown fatigue scenario" around the time of the lifting of the second emergency declaration
■Without an overall judgment, the "average over the previous 17 weeks" greatly underestimates the increase in infections after the lifting
■With this approach, the basic reproduction number in steady state is 3 or lower
■Unable to grasp the behavioral change in people (in the broad sense) due to the lifting
Transition of Method for Setting Up the Base Scenario
■From August 3, 2021 (Regime B)
■Assume the fluctuation of the number of newly infected people during the state of emergency declaration
■Analyze "If the number of newly infected people is like this, the number of seriously ill patients will be like that."
■After the state of emergency declaration is lifted, the contact rate parameter (after adjusting for economic activity) is given "a slight surge at the time of lifting and then a gradual decrease over time to the basic reproduction number X"
■The value of X is determined comprehensively by referring to the relationship between past contact rate parameters (adjusted) and human flow
■"4 to 5” until September 14
■4 on September 28
■3.75 from October 12
Results

Results (Regime A, until July-end)

■The number of newly infected people (2 weeks): RMSE = 280, MAE = 129, Bias = -105
■Reference: RMSE for infection prediction based on simple human flow and temperature was around 600 to 700 (from November 2020 to September 2021)
■“(September 27) The relationship between COVID-19 infections and human flow":https://covid19outputjapan.github.io/JP/files/FujiiNakata_Mobility_Slides_20210927.pdf
Results (Regime A & B, until mid-December)

■The number of newly infected people for 2 weeks (0.5 months): RMSE = 873, MAE = 398, Bias = 157
■RMSE for infection prediction based on simple human flow and temperature was around 600 to 700 (from November 2020 to September 2021)
Comments
■Bias went from negative to positive as we shifted from Regime A to Regime B
■Since the goal of the transition was to move Bias from negative to near zero, the result was as intended
■However, the objective was overshot
■Note that the fact that the degree of divergence indicators until mid-December is higher than that of by the end of July reflects not only a change in the way the scenarios were set up, but also two other factors below
■The number of newly infected people remained high in August and September
■The frequency of updates in August and September was higher than the one subsequently updated by the Fujii-Nakata team (from October)
Comments
■After the fact, it can be said that we should have moved from Regime A to B a little earlier and in stage
■"Even preliminarily," it can be said to some extent
■In general, it is undesirable to make major changes to analytical methods in the course of ongoing real-time policy analysis
■However, given (1) the "crisis" situation at the end of July and (2) the limited analytical resources, it is difficult to say that this was preliminarily and necessarily a failure.
■With regard to (2), a major constraint was that the Fujii-Nakata team devoted almost all of its analytical resources to the analysis and dissemination in relation to the Olympics from mid-May to the end of June
■Resource constraints in this period also led to a relative lack of risk scenarios at the same time
■Compared to the second half of March and the first half of April, when the "lockdown fatigue scenario" provided a more realistic scenario
Conclusions
■It is important to verify past analyses, but hard to supply
■Since verification of past analyses is unlikely to lead to research papers, there is little incentive for researchers to verify their analyses or those of others
■It is little wonder if there is psychological barrier to face the imperfections of one's past analysis
■In communities where a culture of validation is not well established, validation by others may be perceived as a personal criticism and not constructive
■Verification is to improve the quality of policies based on analyses of the present and for the future
■"The danger past and God forgotten”
■Concrete steps that can be taken to ensure an adequate supply of verification
■Public requests analysts for self-validation
■Foster a culture where analysts appreciate the time and effort used for the verification process, regardless of the results of the self-verification
■Analysts publish codes and other information for others to easily verify
■Foster a culture where analysts recognize and appreciate validation conducted by others (regardless of the results of the validation)
Number of Newly Infected People (by the end of July)
2 Weeks

1 Month

Number of Seriously Ill Patients (by the end of July)
2 Weeks

1 Month

Number of Newly Infected People (by the end of December)
2 Weeks

1 Month

Number of Seriously Ill Patients (by the end of December)
2 Weeks

1 Month

■Analysis update and Zoom briefing on Tuesdays: https://Covid19OutputJapan.github.io/JP/
■Reference materials: https://covid19outputjapan.github.io/JP/resources.html
■Zoom briefing video: https://covid19outputjapan.github.io/JP/recording.html
■Economic Seminar Series
■https://note.com/keisemi/n/n9d8f9c9b72af
■https://note.com/keisemi/n/n7f38099d0fa2
■https://note.com/keisemi/n/nd1a6da98f00e
■https://note.com/keisemi/n/n430f8178c663
■Papers available at: https://link.springer.com/en-article/10.1007%2Fs42973-021-00098-4
■Twitter: https://twitter.com/NakataTaisuke
■Questions, requests for analysis, etc.
■taisuke.nakata@e.u-tokyo.ac.jp