Prerequisites
■Periodic validation of past analyses is desirable for effective use of model analysis in policy
■Verification of past analyses can improve the quality of the current analyses
■The assumption that verification will be conducted in the future gives discipline to the current analyses
■Accountability associated with analyses used as policy inputs can be fulfilled
■The accumulation of verifications can be reference materials for future crises
■There may be implications with reference to crises other than the current pandemic
※Examples of Spaghetti Charts
※Examples of Spaghetti Charts
Analysis
■The periodic "outlook" by Fujii-Nakata team
■Updated weekly from January 21, 2021 to August 31, 2021
■A projection for the number of seriously ill patients has been updated from May 25, 2021
■Bi-weekly in September, bi-monthly from October
■September 14, September 28, October 12, November 2, and December 21 (tentative)
■Disseminated to the public and people in the policy field throughout 2021
■https://covid19outputjapan.github.io/JP/media.html
■Subcommittee, MHLW Advisory Board, Prime Minister's Office, Council of Ministers, Office for COVID-19 and Other Emerging Infectious Disease Control of Cabinet Secretariat, Olympics Expert Round-table conference
■It is not always productive to evaluate the value of scenario analyses solely on the basis of deviations from realized values. However, since the Fujii-Nakata team has been aiming to "conduct scenario analyses in a relatively realistic context," it is important to record the deviation between the presented scenarios and the realized values
Transition of Method for Setting Up the Base Scenario
■From January 21, 2021 to July 27, 2021 (Regime A)
■Assume the fluctuation of the number of newly infected people during the state of emergency declaration
■After the declaration was lifted, the contact rate parameter (before adjusting for economic activity) was an average over the previous 17 weeks
■Adjustments were made based on overall judgment as needed
■Until mid-June, the role of overall judgment was limited in the base scenario
■Until mid-June, a more realistic scenario is presented as the risk scenario
■Example: The "lockdown fatigue scenario" around the time of the lifting of the second emergency declaration
■Without an overall judgment, the "average over the previous 17 weeks" greatly underestimates the increase in infections after the lifting
■With this approach, the basic reproduction number in steady state is 3 or lower
■Unable to grasp the behavioral change in people (in the broad sense) due to the lifting
Transition of Method for Setting Up the Base Scenario
■From August 3, 2021 (Regime B)
■Assume the fluctuation of the number of newly infected people during the state of emergency declaration
■Analyze "If the number of newly infected people is like this, the number of seriously ill patients will be like that."
■After the state of emergency declaration is lifted, the contact rate parameter (after adjusting for economic activity) is given "a slight surge at the time of lifting and then a gradual decrease over time to the basic reproduction number X"
■The value of X is determined comprehensively by referring to the relationship between past contact rate parameters (adjusted) and human flow
■"4 to 5” until September 14
■4 on September 28
■3.75 from October 12
Results (Regime A, until July-end)
Results (Regime A & B, until mid-December)
■The number of newly infected people for 2 weeks (0.5 months): RMSE = 873, MAE = 398, Bias = 157
■RMSE for infection prediction based on simple human flow and temperature was around 600 to 700 (from November 2020 to September 2021)
Comments
■Bias went from negative to positive as we shifted from Regime A to Regime B
■Since the goal of the transition was to move Bias from negative to near zero, the result was as intended
■However, the objective was overshot
■Note that the fact that the degree of divergence indicators until mid-December is higher than that of by the end of July reflects not only a change in the way the scenarios were set up, but also two other factors below
■The number of newly infected people remained high in August and September
■The frequency of updates in August and September was higher than the one subsequently updated by the Fujii-Nakata team (from October)
Comments
■After the fact, it can be said that we should have moved from Regime A to B a little earlier and in stage
■"Even preliminarily," it can be said to some extent
■In general, it is undesirable to make major changes to analytical methods in the course of ongoing real-time policy analysis
■However, given (1) the "crisis" situation at the end of July and (2) the limited analytical resources, it is difficult to say that this was preliminarily and necessarily a failure.
■With regard to (2), a major constraint was that the Fujii-Nakata team devoted almost all of its analytical resources to the analysis and dissemination in relation to the Olympics from mid-May to the end of June
■Resource constraints in this period also led to a relative lack of risk scenarios at the same time
■Compared to the second half of March and the first half of April, when the "lockdown fatigue scenario" provided a more realistic scenario
Conclusions
■It is important to verify past analyses, but hard to supply
■Since verification of past analyses is unlikely to lead to research papers, there is little incentive for researchers to verify their analyses or those of others
■It is little wonder if there is psychological barrier to face the imperfections of one's past analysis
■In communities where a culture of validation is not well established, validation by others may be perceived as a personal criticism and not constructive
■Verification is to improve the quality of policies based on analyses of the present and for the future
■"The danger past and God forgotten”
■Concrete steps that can be taken to ensure an adequate supply of verification
■Public requests analysts for self-validation
■Foster a culture where analysts appreciate the time and effort used for the verification process, regardless of the results of the self-verification
■Analysts publish codes and other information for others to easily verify
■Foster a culture where analysts recognize and appreciate validation conducted by others (regardless of the results of the validation)
