The budget fiasco

From the Inquiry's findings into the Treasury budget leak:

33. The Inquiry considers the senior leadership did not actively consider or promote a view of the Treasury’s appropriate obligations in relation to the production of Budget information. The organisation has faced ever increasing demands for greater volume and more complex Budget products. This resulted in:
a. Managers and teams feeling they had no option but to deliver whatever was requested of them, irrespective of the impact on resourcing and potential organisational risk; and
b. Critical decisions being made for expediency’s sake, in the absence of consideration of the wider organisation and security risk.

34. The Inquiry considers some of the above findings may be indicative of wider issues within the Treasury and invites the current Secretary of the Treasury to consider these matters further. 

The potential for snippets to be indexed from the clone site was known in 2018, but not addressed for 2019.

72. When the cloned Treasury website was subsequently created, approximately 2 weeks before Budget Day 2018, the clone was configured to use the shared index with the live website, thus breaking the “vault” as per the stipulated configuration. W&P team members responsible for producing Budget documents worked to develop those documents on the cloned website, and once finalised documents were set to “published” state on the cloned website. Because the cloned website and live website used a shared index, document headline and snippet information on the clone website where documents were set to “published” were able to be accessed by search users.

73. From interviews with relevant Treasury staff it is apparent that the ability to view Budget Sensitive document headline and snippet information in response to specifically worded search activity followed by an Error 404 script was known by individuals involved in determining to deploy the clone as the BDS solution for the Treasury website in 2018. There is no evidence to indicate the risk associated with visibility of document headlines and snippet information was formally escalated outside of this group. While it was contemplated not linking the index and cloning the index as well as the site, the testing method employed by CASS staff indicated it would take approximately three days to re-index the clone site prior to Budget Day which did not meet business requirements and consequently this option was disregarded. On the day of the incident the platform vendor was able to deploy an alternative method that re-indexed the clone site within one hour. The CASS IT team had not previously sought any advice from the vendor regarding methodology for re-index.

I think I've heard similar grumbles elsewhere in the public sector: business requirements that don't quite get why something that's just-released won't be immediately indexed. 

86. Upon observing the configuration which drove both site searches to the shared index, the vendor recommended changing the configuration to remove the link to the shared index. The CASS IT team raised concerns regarding the time to rebuild the index. The vendor was able to show the IT team a standard command line which took just one hour to re-index the site (not three days as experienced by the CASS team in testing their own method). 

And the whole thing goes then to governance:

91. In the view of the Inquirer, this flawed technical solution coupled with a lack of good practice was able to occur as a consequence of failures in the application and appropriateness of Treasury wider security, risk, control and governance settings. In the Inquirer’s view, those issues created an environment in which similar incidents are possible until such time as the Treasury improves the application of its systems, processes and governance of similar activities. The adequacy of action taken by the Treasury post the incident was outside the scope of the Inquiry and has not been assessed.

The report suggests that the problem would have been fixed if someone had escalated the clone design up to the IT Security Manager.  

113. The Inquiry did not find evidence of effective oversight in relation to the teams involved in the  incident and many of the people spoken to in relation to the incident raised concerns regarding the inability of managers to successfully escalate to senior managers matters such as the non-engagement by the wider organisation in the Treasury Website project or the descoping of the BDS from the same.

114. The Inquiry considers the organisational structure contributed to the incident particularly in relation to the operation of the Treasury/CASS IT team. As the senior manager of the Treasury and CASS IT function, the CIO was not a member of Kaiurungi at the time of the incident. The CIO reported to Kaiurungi on operational matters via monthly reporting on IT and Information Management however the Inquiry was told that corporate services was not a priority area of focus for Kaiurungi.

Kaiurungi is chaired by the Chief Operating Officer - Fiona Ross at the time. The CIO also reports to the COO. 

115. The Inquiry heard from a number of interviewees of the challenge within the organisation of gaining senior level engagement or commitment on matters associated with organisational functioning or performance, particularly where it related to corporate services. Furthermore the Inquiry observed an apparent organisational divide between “the business” and corporate services. In the Inquiry’s view, a disregard for the role of corporate services coupled with a lack of prioritisation of delivering organisational objectives contributed to the incident in a number of ways as evidenced by:
a. The inability of the Treasury Website Project to gain engagement from “the business” in the design, content and management of the new Treasury Website;
b. The lack of consideration of the impact on the W&P team of the continual increase in demand for the production of Budget documents in the final 6 weeks of the Budget preparation and subsequent impact on organisational risk profile;
c. The non-involvement of W&P in the BOG;
d. The failure to develop end-to-end process or governance oversight of the Budget process;
e. The failure to undertake effective close-out or other review procedures to inform organisational performance
f. The non-inclusion of the CIO on Kaiurungi.

116. The vulnerability in this area was further exacerbated by a reported organisational belief that work on core business operations is less valued or important than policy work or other core economic or fiscal functions of the Treasury and therefore not prioritised.

The CIO reports to the COO and the COO is on Kaiurungi.

Things start looking bleak as we read on.

121. Interviewees reported struggling to gain regular, consistent engagement with the Steering Group of the Treasury Website Project (subsequently renamed the Treasury Website Migration Project). Documentary evidence supports the assertion that meetings were held infrequently, not well attended and ultimately reduced to the COO and Project Manager in composition.

122. The Inquiry considers that the Treasury Website Project Steering Group (subsequently renamed Treasury Website Migration Project Steering Group) did not provide effective governance oversight of the project and failed to alert the wider organisation to the significant risks associated with the project.

Lack of Post-Implementation/Close-out Review

123. Based on the information reviewed by the Inquiry, had the Treasury undertaken a robust post implementation review of the TWMP and a review of the BDS solution in 2018 it may have highlighted the risk created by the visibility of document headline and snippet information. This may have led the Treasury to consider how it could improve the BDS solution for 2019 and thus averted the incident.

124. The lack of post project review in relation to the TWMP and BDS solution for 2018 is consistent with findings from a number of independent reports commissioned by the Treasury and reviewed by the Inquiry in relation to the Budget Process and other risk matters. Furthermore the Inquiry observes that the Treasury has not implemented recommendations contained in a number of independent reviews commissioned by the Treasury. This is consistent with the Treasury’s lack of prioritisation of working on core business operations and in implementing systems and governance to pursue the same.

Increasing Pressure on Treasury Staff working on the Budget

125. Following discussions with relevant Treasury staff, the Inquiry considers the Treasury’s senior leadership did not adequately consider the impact on staff of ever increasing production expectations, milestone slippage outside of the Treasury’s control or the impact on the risk profile for the Treasury itself of its unquestioning approach.

126. The Inquiry considers the lack of senior leadership consideration of the demands on the organisation contributed to an environment whereby:
a. managers and teams felt they had no option but to deliver whatever was requested of them, irrespective of the impact on resourcing and potential organisational risk; and
b. in which critical decisions were made for expediency’s sake, in the absence of consideration of the wider organisation and security risk.

The report goes on to note rather a few initiatives underway at Treasury to strengthen processes.

Overall it looks like a governance issue. The IT guys figured nobody up the chain wanted to hear about risks, and they didn't know that there was a solution to the up-the-chain problem of folks who wanted stuff kept strictly confidential but immediately (or almost immediately) indexed for search when released.

I wonder whether the IT team would have punted the thing up the chain had they known that they could de-risk it at the cost of only an hour's delay on indexing rather than a likely-to-be-shot-down 3-day wait. And I wonder whether, had they mentioned it earlier, someone might have told them to check if there were alternatives like that.

The Inquiry points to failure at senior leadership levels. They don't name names as such, but you'd think that would have to be the Chief Information Officer, and the Chief Operations Officer to whom he reported, and if you wanted to go up to the top, well, Makhlouf. They'd know more about what all happened than I do.

I just keep remembering that Treasury had Fiona Ross fronting a workshop and sun and moon feelings a few weeks before the budget fiasco. Danyl's reporting on that was withering.

Fiona Ross is a thought leader in the public service; an articulate and engaging public speaker. She stands at the front of the room: a seminar space on the third floor of Treasury. The 30 people in the audience fall silent. She begins. “We all know we live in a DEVUCA world.”

Everyone nods thoughtfully. Except me. I raise my hand. “We live in a what?”

Ross looks at me and blinks. “DEVUCA.”

I try to imitate the sound, unsuccessfully. Someone at my table explains the acronym in a low voice. “It stands for diverse, ambiguous, volatile, uncertain …”

“I think complexity is in there,” another person suggests. There is some disagreement. No one is quite sure exactly what kind of diverse complex ambiguous world we occupy.

“Google it,” Ross advises.

“I will. How do you spell …?”

“And in a DEVUCA world we all need to be more empathic and inclusive. That’s why Heartwork is so exciting.”

Maybe the whole card game thing really only took 15 minutes of her time. But you read this, and then you look at the IT mess that came under Ops at Treasury, and it's hard not to wonder if they maybe needed to be a bit more focused on core business than on DEVUCA worlds and card games.

Very glad that Treasury's now under new management.