UK.gov still drowning in legacy tech because no one's boarding Blighty's £700m data centre Ark
Little love for Crown Hosting from Whitehall depts
Analysis Only in IT is “legacy” a pejorative term, where it is used to condemn ageing systems and forgotten workarounds.
In the UK government, as with banks, increasingly difficult-to-maintain mission-critical systems are a huge problem. Not least because of the dwindling number of folk who remember how the damn things work.
One solution to Whitehall’s myriad string and sticky-tape systems was the creation of the Crown Hosting programme in 2015. That was intended to house departments’ legacy systems in facilities run by a joint venture of which the government owned 25 per cent and small data centre biz Ark the remainder.
By “lifting and shifting” all the legacy systems and housing them in one data centre, it was hoped replaceable systems could be identified, contained and run at a much lower cost – with the eventual plan to ditch them entirely.
But since it was announced, that programme has gone eerily quiet.
The Register whipped out the Freedom of Information Act to ask for a list of public sector bodies that have signed up to the arrangement. However, we were told the Cabinet Office could not disclose the customer list for this commercial arrangement because “authorities could be targeted by individuals or groups willing to use malicious or other hostile ways to gain unauthorised access to information (sensitive or otherwise) stored at colocation sites.”
One can’t help wonder why the UK government bothered to announce the deal at all: it was supposedly meant to contain all of government’s data centre estate via Ark's two data centres in Farnborough and Corsham in a deal worth up to £700m.
But there could be another reason why the Cabinet Office veiled the project under a cloak of invisibility (beyond that being its usual modus operandi). According to numerous sources, uptake has so far been extremely low.
One source revealed that the Department for Work and Pensions had intended to shift 250 of its systems to the data centre, but is now migrating just five. That was part of the department’s mega £340m hosting services refresh to tackle its ageing infrastructure. In this instance, the department had hired hundreds of contractors to help it virtualise the current platforms onto the new kit.
Reg readers are welcome to ponder the plausibly of a government project of that size being ahead of schedule. Nevertheless, it’s possible some of the systems have been refreshed.
One source told us: “My guess is that it will be the Customer Information System (CIS) that is moving as IBM were already commissioning a replacement [CIS] at Corsham & Farnborough, currently hosted on ancient Sun E25k frames, which have caused serious outages [of] Critical National Infrastructure due to hardware failures.
“Therefore it would it make a lot of sense to put these together in the same data centres. Moving CNI systems to Ark should be a lot more secure."
One contact said part of the problem with Ark for government use was the fact that some of the departments’ legacy kit won’t fit in its racks, while in other cases the hardware is partly or fully owned by a system integrator – making it difficult to shunt their kit somewhere else.
Another said the problem is that the government knows little about its old systems – citing the Home Office’s 1995 Casework Information Database (CID) as an example – and has been patched many times in haste and changed only when legislation requires.
"There probably isn’t anyone who really knows how it works. Worse, CID isn’t just ‘CID’ – it’s a system that interacts and exchanges data with dozens of other systems, some inside the immigration department, some in the rest of the Home Office and some across government. So when you move ‘CID’ you are moving a living thing.
“Many of these systems don’t even have true disaster recovery – the ideal option would be to move the disaster recovery (DR) to the Crown Hosting site and fail over to it, so avoiding lots of hassle. But in the 1990s and 2000s government tended not to build real DR (in the sense of active / active or even active / near active).”
As a sidenote, the CID system was to be replaced by an Immigration Case Work (ICW) system in December 2008 by IBM that was intended to support applications for visas and immigration. However, the department was forced to write off £347m in 2013. The National Audit Office noted in 2014 that the CID system is plagued by problems such as freezing, a lack of interface with other systems, and a lack of controls.
According to one of our sources, the Home Office is still working on its Ark transition, building a couple of environments there for production services, but have yet to move anything. “I’ve done a lot of hosting moves in my time and, unless the folks doing it have also done a lot, they will massively underestimate how hard it is, especially if the hosting and the apps people are separate companies,” a contact told us.
For him, a failure to migrate the legacy kit comes down to "the cost being too big, with a relatively long-term payback, while managing an awful lot of risk, particularly the risk that it just won’t work because you don’t understand how the system works.”
No doubt many poor souls tasked with working out legacy replacements would love nothing more than to pull the plug, throw it in a skip and install something else. But unfortunately when it comes to mission-critical legacy gear, that particular Gordian Knot can’t be cut. ®
In the UK government, as with banks, increasingly difficult-to-maintain mission-critical systems are a huge problem. Not least because of the dwindling number of folk who remember how the damn things work.
One solution to Whitehall’s myriad string and sticky-tape systems was the creation of the Crown Hosting programme in 2015. That was intended to house departments’ legacy systems in facilities run by a joint venture of which the government owned 25 per cent and small data centre biz Ark the remainder.
By “lifting and shifting” all the legacy systems and housing them in one data centre, it was hoped replaceable systems could be identified, contained and run at a much lower cost – with the eventual plan to ditch them entirely.
But since it was announced, that programme has gone eerily quiet.
The Register whipped out the Freedom of Information Act to ask for a list of public sector bodies that have signed up to the arrangement. However, we were told the Cabinet Office could not disclose the customer list for this commercial arrangement because “authorities could be targeted by individuals or groups willing to use malicious or other hostile ways to gain unauthorised access to information (sensitive or otherwise) stored at colocation sites.”
One can’t help wonder why the UK government bothered to announce the deal at all: it was supposedly meant to contain all of government’s data centre estate via Ark's two data centres in Farnborough and Corsham in a deal worth up to £700m.
But there could be another reason why the Cabinet Office veiled the project under a cloak of invisibility (beyond that being its usual modus operandi). According to numerous sources, uptake has so far been extremely low.
One source revealed that the Department for Work and Pensions had intended to shift 250 of its systems to the data centre, but is now migrating just five. That was part of the department’s mega £340m hosting services refresh to tackle its ageing infrastructure. In this instance, the department had hired hundreds of contractors to help it virtualise the current platforms onto the new kit.
Ignore that burning, everything’s fine
At the end of October, several contractors got in touch with The Register to report there had been hundreds of layoffs and hundreds of millions in overspend at the DWP. But that has been vigorously denied by the department. Commenting on the refresh programme, a spokeswoman said of the eight-year SSBA refresh programme: “It is ahead of schedule and has already delivered three large-scale, secured and resilient platforms.”Reg readers are welcome to ponder the plausibly of a government project of that size being ahead of schedule. Nevertheless, it’s possible some of the systems have been refreshed.
One source told us: “My guess is that it will be the Customer Information System (CIS) that is moving as IBM were already commissioning a replacement [CIS] at Corsham & Farnborough, currently hosted on ancient Sun E25k frames, which have caused serious outages [of] Critical National Infrastructure due to hardware failures.
“Therefore it would it make a lot of sense to put these together in the same data centres. Moving CNI systems to Ark should be a lot more secure."
One contact said part of the problem with Ark for government use was the fact that some of the departments’ legacy kit won’t fit in its racks, while in other cases the hardware is partly or fully owned by a system integrator – making it difficult to shunt their kit somewhere else.
Another said the problem is that the government knows little about its old systems – citing the Home Office’s 1995 Casework Information Database (CID) as an example – and has been patched many times in haste and changed only when legislation requires.
Systems from the 1990s
“It uses old versions of everything at every layer of its architecture," we're told."There probably isn’t anyone who really knows how it works. Worse, CID isn’t just ‘CID’ – it’s a system that interacts and exchanges data with dozens of other systems, some inside the immigration department, some in the rest of the Home Office and some across government. So when you move ‘CID’ you are moving a living thing.
“Many of these systems don’t even have true disaster recovery – the ideal option would be to move the disaster recovery (DR) to the Crown Hosting site and fail over to it, so avoiding lots of hassle. But in the 1990s and 2000s government tended not to build real DR (in the sense of active / active or even active / near active).”
As a sidenote, the CID system was to be replaced by an Immigration Case Work (ICW) system in December 2008 by IBM that was intended to support applications for visas and immigration. However, the department was forced to write off £347m in 2013. The National Audit Office noted in 2014 that the CID system is plagued by problems such as freezing, a lack of interface with other systems, and a lack of controls.
According to one of our sources, the Home Office is still working on its Ark transition, building a couple of environments there for production services, but have yet to move anything. “I’ve done a lot of hosting moves in my time and, unless the folks doing it have also done a lot, they will massively underestimate how hard it is, especially if the hosting and the apps people are separate companies,” a contact told us.
For him, a failure to migrate the legacy kit comes down to "the cost being too big, with a relatively long-term payback, while managing an awful lot of risk, particularly the risk that it just won’t work because you don’t understand how the system works.”
No doubt many poor souls tasked with working out legacy replacements would love nothing more than to pull the plug, throw it in a skip and install something else. But unfortunately when it comes to mission-critical legacy gear, that particular Gordian Knot can’t be cut. ®
