The impossible task of modernizing government software

Not surprised. Government software infamously lags and is usually way behind the times. It just takes a moment in time to showcase how bad it has gotten.

“When the tide goes out, you see who has been swimming naked.”
Warren Buffet

COVID helped reveal government software’s true age

The COVID-19 pandemic has acted as a stress experiment that has revealed previously unknown or underappreciated weaknesses in governmental digitalization. It also revealed important social disparities. Uneven access to digital infrastructures is one such issue. The pandemic has shown that not everyone has equal access to digital tools needed to work in remote locations. This has resulted in inequities and hardships for those who do not have access to these tools.

The presence of analog elements in digitalization is another issue, as COVID-19 has shown that a number of highly digitalized processes yet lean on analog elements. For example, even though many people are employed from home, physical documents and signatures are often still required for many processes.  This slows down the digitalization process.

The instability and fragility of unchecked digitalization is another similar concern that has been exposed by the pandemic. Many algorithms used in digitalized inter-organizational processes are brittle due to overreliance on historic patterns, which can lead to errors and biases in decision-making. This can have serious consequences in areas such as healthcare, finance, and transportation.

The pandemic has breached fundamental expectations of privacy when organizational surveillance was extended into private and public spaces. This has raised concerns about the impact of digitalization on individual privacy and autonomy, and highlights the need for ethical guidelines and regulations to protect individuals from excessive surveillance.

The pandemic has revealed primary problems about the relation between technology and organizing. It has demonstrated the need for more equity in access to digital utilities, the need  for balancing digital and analog elements in organizational processes, and the need for regulations  and ethical guidelines to protect individual rights in a digitalized world.

Federal Information Technology Acquisition Reform Act (FITARA)

“A lot has changed in the world of technology and in the federal government’s use of technology ... and the need to streamline and strengthen how the government buys and manages technology is long overdue”  Gerry Connolly, Congressman.

In 2014, the Federal Information Technology Acquisition Reform Act (FITARA) was passed by Congress as a part of the National Defense Authorization Act. It was aimed at improving the management of information technology across the entire spectrum of the federal government, with the goal of reducing costs and improving the efficiency and security of IT systems. FITARA has several key provisions that impact the acquisition and management of IT systems in government, including:

1. Chief Information Officer (CIO) Authority
   FITARA requires agencies to elevate the position of the CIO to a more senior level and to give the CIO more authority over the agency's IT budget and personnel. This helps ensure that IT decisions are made by individuals with the technical expertise and decision-making authority to implement changes effectively.
2. IT Portfolio Management
   FITARA requires agencies to develop an IT portfolio management plan that includes a comprehensive view of all IT investments, including hardware, software, and services. This helps agencies better understand their IT spending and to make more informed decisions about future investments.
3. Data Center Consolidation
   FITARA requires agencies to consolidate their data centers to reduce costs and improve efficiency. This includes closing underutilized data centers and consolidating data center operations in other facilities.
4. IT Acquisition Reform
   FITARA includes provisions that aim to improve the acquisition of IT systems in the federal government. This includes requiring agencies to use commercial off-the-shelf (COTS) products where possible, to make it easier for agencies to access the latest technology at a lower cost.
5. IT Dashboard
   FITARA requires the creation of an IT dashboard to provide public access to information about federal IT investments. The dashboard provides information about agency IT budgets, workforce data, and details about IT systems and projects.
6. Enhancing IT workforce development
   FITARA requires federal agencies to develop and implement a plan for improving the skills and capabilities of their IT workforce. This is intended to ensure that federal agencies have the talent and expertise necessary to effectively manage their IT investments and operations.

FITARA is intended to modernize and improve the way that the federal government acquires and manages IT products and services, with the goal of increasing efficiency, effectiveness, and accountability in the use of taxpayer dollars. Since its passage, the Department of Defense (DoD), Department of Homeland Security (DHS), Department of Justice (DOJ), and Environmental Protection Agency (EPA) are among the agencies that have implemented FITARA-related initiatives, such as IT procurement streamlining, data center consolidation, IT governance improvement, IT infrastructure modernization, IT asset management improvement, and cybersecurity posture improvement.

Focusing squarely on the Environmental Protection Agency (EPA), they have launched several projects to modernize its legacy IT systems as a part of this initiative, including:

1. The Data Center Optimization Initiative (DCOI)
   In 2016, the EPA launched DCOI to reduce its data center footprint and improve the efficiency of its IT infrastructure. As part of this initiative, the agency consolidated its data centers and migrated many of its applications to cloud-based platforms.
2. The Office 365 Migration
   In 2017, the EPA began migrating its email and collaboration systems to Microsoft Office 365, a cloud-based productivity suite. This migration has enabled the agency to improve its communication and collaboration capabilities while reducing its reliance on on-premise hardware and software.
3. The IT Modernization Plan
   In 2018, the EPA released an IT Modernization Plan that outlined the agency's strategy for upgrading its legacy IT systems and applications. The plan focused on improving the agency's cybersecurity posture, increasing the efficiency and effectiveness of its IT operations, and enhancing its data analytics capabilities.
4. The Integrated Planning, Accounting, and Reporting (IPAR) System
   In 2020, the EPA launched the IPAR system, a cloud-based platform for managing financial and programmatic data related to the agency's environmental programs. The system replaces several legacy financial management systems and enables the agency to better track and report on its budget and program performance.

So we know it can be done, and FITARA was the law that finally moved the needle. But why is it so very difficult? And why does it take so long and cost so much? It feels like they’re building aircraft carriers rather than desktop software.

Problems unique to government software and IT

When it comes to challenges in our government, and information technology in general, which is riddled with challenges for any company, there is no shortage of targets for discussion. The challenges vary widely, but for most government agencies, they will include:

1. Legacy systems are forgotten and taken for granted
   Many federal agencies still rely on outdated and unsupported systems that were developed decades ago. These systems are expensive to maintain and can be a security risk, making it difficult for agencies to modernize their technology infrastructure. But so often they are not focused on and assumed to be just fine. There’s a common mistake in IT - to not upgrade or update the already working system. If it ain’t broke, don’t fix it. But this is shortsighted, as these legacy systems can become increasingly inefficient and problematic over time. And, with every update to software around the legacy beast, and as technology advances in general, it becomes more difficult to integrate legacy systems with newer technologies.
2. Budget committees are uneducated and ignorant to the real cost of software
   Federal IT projects are often subject to limited budgets and long procurement processes, making it difficult to acquire the latest technology and resources needed to support modern systems. But what makes it even more complex and disheartening for government IT leaders, is the review and approval process, often run by committee members that simply are less educated around the technology decisions they are approving or denying. This lack of understanding can lead to underfunded projects or the approval of costly systems that don't meet the organization's needs. For example, in 2013, the U.S. government spent $2.2 billion on a failed healthcare enrollment website that was riddled with technical issues and usability problems. This project was approved by budget committees that were not well-versed in the complexities of software development and did not fully understand the scope of the project. Another example is the Department of Defense's failed attempt to build a single, unified EHR system, which was plagued by delays and cost overruns, ultimately costing taxpayers billions of dollars. Budget committees and decision makers in government absolutely must be more educated and deeply vested on the true cost, risks, and impacts of technology decisions.
3. Fragmented IT management - too many layers, too many decision-makers
   The federal government has a large number of agencies, each with its own IT systems and infrastructure. This fragmentation can make it difficult for agencies to coordinate and share resources, leading to duplicated efforts and higher costs. The lack of centralized IT management also presents security risks, as it can be challenging to implement consistent security protocols across multiple systems. In 2015, the Office of Personnel Management (OPM) experienced a massive data breach that compromised sensitive information belonging to millions of federal employees. The breach was partly attributed to the agency's fragmented IT infrastructure and the lack of centralized management and oversight. The Government Accountability Office (GAO) has consistently identified IT management fragmentation as a key challenge for federal agencies, highlighting the need for a more unified approach to IT management and modernization.
4. Security and compliance are top priority, but hard to control from the top
   Federal agencies are subject to strict security and compliance regulations, making it challenging to implement new technologies and systems that meet these requirements. There have been instances where agencies have, while knowing and attesting to the importance of all forms of technological security, completely neglected their responsibility for security and compliance, leading to data breaches and incidents. In 2019, the Federal Emergency Management Agency (FEMA) exposed the personal data of millions of disaster victims due to a lack of proper security controls on its IT systems. Wired magazine reported, “In doing so, the agency violated the Privacy Act of 1974 and Department of Homeland Security policy, and exposed survivors to identity theft.” There were a handful of local non-profits and for-profit contractors that were brought in to help identify survivors and work with those experiences loss of home, property, and loved ones. When a FEMA representative sent the excel file to one of those contractors, it was suppose to be a list of the known individuals that would be eligible for assistance. Problem is, they accidentally included a a handful of extra columns of data on the spreadsheet. But no big deal, nothing critical, just their home addresses, their bank account information, wire/routing numbers, etc.... (Ya, this was a really bad oops!) Over 2 million Americans’ data was exposed. But the government doesn’t call it a data breach. It was an isolated incident, and nothing to see here.
   Brock Long, who was running FEMA, said, “Security is our top priority. We cannot be successful in our mission without ensuring that the personal information of disaster survivors is protected to the greatest extent possible.”
5. The best talent will usually shy away from government jobs
   The federal government faces challenges attracting and retaining the highly skilled IT professionals needed to support modern systems. The government also struggles to compete with the private sector in terms of salaries and benefits, making it difficult to attract and retain top talent. A 2019 report found that less than 7 percent of federal employees are under the age of 30, as compared to 23 percent of the entire U.S. labor workforce is under 30. The younger, hipper talent is going to startups or companies with vibes.
6. Hiring is not strategic or personal
   One government hiring manager put it this way when interviewed,  “Historically, hiring has not been very strategic. We’ve been on autopilot. ‘I lose one, I get one. I lose one, I get one.’ It’s reactionary hiring; it’s not strategic hiring.” This is not uncommon. The autopilot mindset is driven by the bureaucratic process. It’s a headless entity. There’s no CEO running around - just elected (short term) leaders with their (short term) agendas.  Many leaders see the issue, and know it’s important, but they don’t have the cultural tools to make a change. A survey conducted by Retensa found that 91% of respondents, consisting of 70 public sector leaders across 18 states, reported recruitment and retention are among their very most important organizational goals. So they know it’s important, but this is not connecting to how the hiring actually takes place, or what the culture can actually become.
7. Interoperability of too many systems over too long of time periods
   Federal and local agencies will quite often use different software applications, databases, and licensed commercial off the shelf software (COTS) systems to achieve it’s very specific needs. This makes the work, (of moving data, exporting data, important data, processing records, integrating with other systems, and viewing information), even more tedious and shadowy. Only a few folks around the office know how to operate the contrived series of concocted steps in order to get a widget to pop out the other side of the conveyor belt. It’s endless complexity. And that inevitably leads to random and out of control inefficiencies with duplicated efforts.

How constantly changing regulations digs the knife even further

Govtrack.us shows thousands of new laws and other legislation being added each year.

The Heritage Foundation found in their 2016 report Red Tape Rising that 20,642 new federal regulations were added under the Obama Administration. The trend appears to be accelerating.

So one year it’s this, the next year it’s that. Make up your mind! But alas, the law is the law, so let’s go implement it.

Government software must be designed and developed with the understanding that regulations are constantly changing. This means that the software must be flexible and adaptable enough to accommodate changes in regulations, without requiring significant redesign or redevelopment. This challenge has been solved many times in the past. The methods used include:

1. Compose a solution using a fully modular design
   Government software can be designed with a modular architecture, where individual components or modules can be easily added, removed, or modified as needed to accommodate changes in regulations. This allows for a more flexible and adaptable software system.
2. Use a long-term contracted provider to ensure compliance tracking
   Government software can be tested and monitored at intervals to ensure compliance is upheld. This is called compliance tracking. It is a collection of features that monitor and track changes in regulations and compare those changes to software and data in real time. These products can send notifications and alerts when new regulations come in and those new laws invoke immediate violations based on existing data and software. But this should be implemented and managed across multiple terms of leadership, (because leaders are reelected or ousted every few years). And, it should be done by a third-party contracted provider so it’s not ousted with the outgoing leader.
3. Deeper and more strategic integration with regulatory systems
   Government software can be integrated with regulatory systems, such as those maintained by central state or federal government agencies or industry associations.
4. Collaboration with regulatory agencies to ensure the data and software is headed in the right direction (product leadership being connected to the vine)
   In IT, its the role of product owners and product leaders to drive the software and establish its scope. Those product leaders tell programmers what to build. But often times, they are completely and hopelessly out of the loop. They are many times introverted, and stare at pixels on a screen, failing to go shake hands and talk to folks at conferences and such. Government software product leaders can should collaborate with regulatory agencies to stay closely and deeply informed about coming changes in regulations and to ensure that the software is on the right path, and continues to remain compliant and focused on the right things. Attend a few regulatory meetings or public forums...  Talk to regulators... just get out there and spin tires on pavement. “With who?” you may ask... With regulatory agencies like the Environmental Protection Agency (EPA), which manages data on air and water quality, hazardous waste management, and the environmental impact of chemicals and pesticides. Or like the Federal Communications Commission (FCC), which manages data on broadband internet availability and quality, radio and television broadcast stations, and telecommunications services. Or the Food and Drug Administration (FDA), which manages data on drug and medical device approvals, food safety inspections, and adverse events reporting. The Occupational Safety and Health Administration (OSHA), is another one. They protect our workforce, and watch over workplace injuries, illnesses, fatalities, etc., as well as information about hazardous materials and chemical exposures at all onsite work locations. All of these agencies collect and analyze data on industry trends, compliance with regulations, and enforcement activities in order to improve their regulatory oversight. You’d be shocked to discover that so many, (dare I say, most), other local entities rarely collaborate with these agencies. Though this is subjective, it’s often reality beyond our personal experiences. And yet, charts like this one prove the point that regulators and regulations are everywhere. The annual spend for regulatory compliance is [still] on the rise... as much as $60 billion dollars in 2016 alone... so crazy what we spend in our country to ensure we do the right things. And yet, software product owners in the government space are often playing catch-up to all that time and money and paperwork.  

Takeaways

While it’s a known fact and somewhat of a universal truth that government software is perpetually behind the private sector, hope is not lost. We think a few possible levers to pull include:

1. With new leaders, new subculture can invade the cubicles, breathing new life into stagnant office space. This can be done, but it takes a special leader and a great fit.
2. Product leaders need to expand their tent-pegs and get outside the walls of the IT department. Send them to conferences and book the flights. They need to go out to the marketplace and see / hear the fresh and latest trends. They also need to be connected to the government “vine” of upper level parent agencies and jurisdictions.
3. IT executives in government need to make it a priority to break-through the over-compartmentalization to make a dent in the current duplication and inefficiency that exists everywhere.
4. Build software in modules, each one unique and replaceable in waves. The monolithic application is a thing of the past, as it takes too much time to develop and deploy, and can be difficult to maintain and scale. By building software in modular, replaceable components, developers can more easily update and enhance the application over time, without disrupting the entire system. This approach also allows for more flexible and efficient deployment, as well as improved fault tolerance and resilience. With modern technologies like microservices and containerization, the monolithic application is no longer the only option, and developers can take advantage of a more flexible, scalable, and maintainable architecture.

In early June of 2022, our aging and obsolete United States unemployment IT systems suffered downtime. CNBC titled their news headline: Unemployment system plagued by delays, fraud and racial gaps during pandemic. It read, “The nation’s unemployment system suffered multiple failures during the Covid-19 pandemic, including delayed payments, elevated fraud and “substantial” disparities in receipt of benefits along racial and ethnic lines...Leaving [them] unaddressed will heighten the risk of the UI system not meeting fundamental program expectations of serving workers and the broader economy, and may undermine public confidence in the responsible stewardship of government funds.”