Lessons from Healthcare.gov

I’d like to reemphasize that the opinions in this article and on this blog are completely my own and that I have no access to privileged information. However, I’m a career technology executive and because of other past experiences people ask me about this subject all the time. I decided it was time to write some thoughts down.

A tale of two programs

It’s interesting to contrast the turbulent rollout of Healthcare.gov with that of another government technology program that has been very much in the news of late. This other program was everything Healthcare.gov was meant to be: wildly successful, able to handle massive scale and so brilliantly executed that it left the biggest technology companies in the world scratching their heads and wondering how it was done. I’m referring, of course, to Prism, the NSA’s controversial surveillance program.

To take useful lessons out of the comparison, it’s necessary to depoliticize both programs. Assume, for the sake of argument, that you either support their intent or that they do something totally different that you do support. The point is that one is a lesson in technology done right, the other in technology done…well, not so right.

What caused in the difference in outcomes between the two programs? I’m sure there are many other answers out there, but my own view is that there are three key points:

Don’t outsource mission-critical programs.

Most CTOs will tell you that outsourcing is great for things that are routine or non-critical, but when a company’s main product and reputation are at stake, it pays to do the work inside. Not only does this give the management team the highest level of visibility, but it aligns incentives with outcomes. Even the best contractors who bill time and materials want to run out the clock to maximize their revenue and those doing flat rate want to cut corners to keep profits high. And none of them build componentized systems with reusable parts that can help support future products and prevent the government from continuing to rebuild the same things over and over (for example a single registration process that would allow you to access all government sites in much the same way as you log into commercial sites using Facebook or Twitter.) And contractors don’t live with the results of their decisions afterwards. At least, not unless the debacle is as high profile as Healthcare.gov.

The NSA has hired a cadre of the smartest computer scientists, mathematicians and engineers in the world over the last decade or more. Friends who have been approached were intrigued by the opportunity to work with incredibly bright people on interesting and meaningful problems. The compensation might not match Twitter or Facebook, but there’s the advantage that you’re also serving your country. It’s a pretty intriguing proposition to someone who might otherwise have become a professor or a career engineer at Cisco, IBM or Intel.

Healthcare.gov was contracted. I make no comment about the quality of the contractors having not seen their work (though what has come out is dubious – no queuing or caching?), but the problems with all contractors remain.

Does this point mean that in order to do technology well that the government needs to grow and add tens of thousands of engineers to its payroll like Google? Not exactly. In fact, net of everything, it would probably lead to government getting smaller and more responsive over time as it made better use of technology. And even a relatively small outfit to handle or at least manage significant projects would be a win. Outsource things like IT support, not critical citizen-facing applications.

Follow proven methodologies.

No matter how good the implementation team was for Healthcare.gov, the task as it was defined was an impossible one. Until someone got in there and started building stuff there was no way to know all the complexities that would be discovered. Healthcare.gov needs to talk to systems run by Social Security, Medicare and Medicaid as well as those of dozens and dozens of private insurers. Each of those systems has its own eccentricities and complications. As any data architect will tell you, once you’ve seen one system integration, you’ve seen one system integration. They are all different. And lest you think the private insurance companies’ systems are any better than the government’s, think about how comically inefficient medical billing and claims management is. By some estimates, it eats as much as 10% of the entire amount of money spent on healthcare.

What this means is that trying to light up a system pulling all this stuff together on day one would be hard or impossible. Have some states done it? Yes, but the problem is smaller and there are many fewer systems to glue together. While a better architecture might have solved some problems, at the end of the day the best way to build consumer internet software that industry has yet discovered is to deploy small pieces of functionality, test them with a small group, gather feedback, learn from what you’ve done and fix the problems. Then do it again and again until you’ve built a big, robust piece of software. For example, in the case of Healthcare.gov, registration could have been tested months ago, long before anyone even started writing code to talk to insurers. But if everything has to go live on day one…

It didn’t with Prism. Admittedly, the actual time frames are confidential, but precursors have been leaking to the press for around 10 years. The product evolved through multiple names, iterations and implementations (TIA, Echelon, Prism) before becoming the globe-spanning behemoth it is today. Even now, according to the Snowden files, smaller pieces of functionality (like Muscular) seem to be developed as sub-projects. I’m not advocating that Healthcare.gov should have taken 10 years to develop – actually, the deadline might not have even changed – only that smaller pieces should have been available for comment and beta testing long before the expected public launch.

Empower the CTO.

I do not know exactly how much power the office of the CTO of the United States has. But, if projects like Healthcare.gov are to be successful, it should be on a par with a cabinet secretary if it is not actually a cabinet post. When successful companies are doing big technology projects, the CTO or CIO (who reports directly to the CEO) generally runs the project directly. If an accounting department needs new software or a sales team wants a new CRM system, the technology executive either makes the call or at least has a veto right and oversees the implementation. While domain expertise is completely essential, it is not sufficient. And it’s not enough to bury a few technical people in the mix somewhere or to create a new sub-bureaucracy of some existing agency. That will just result in people doing what they are told rather than contributing real expertise.

There are technical people at the very top of the Prism project, you may be sure of that.

Final thought.

None of this is meant to be armchair generalship. In fact, often you need to make mistakes in order to see the contrast and get better (just as you need to as part of the software development process.) And, as I began this article by saying, the issue has become so politicized that the conversation has been about consequences and trying to burn everything down rather than how to do things better. I’m not saying that I or anyone else would have made better decisions when Healthcare.gov was kicking off. Only that we can (and must) do better in future and make sure that the next Prism-level success is something everyone can agree is wonderful and so that government can get better through the use of technology. Forward.


2 responses to “Lessons from Healthcare.gov

  1. You make a very good point here, but I don’t think you should totally discount the amount of outsourcing that goes on in the NSA. There is a fair amount of outsourcing of IT and crypto work to big contractors like Booz Allen, Mitre, etc. I wonder if perhaps they’re able to keep the project management and/or strategy of projects in house while still outsourcing some of the tech work. If this is the case (I really don’t know), it may make a good argument for a middle ground between bringing all tech development in-house for projects like healthcare.gov and outsourcing them to the private sector entirely.

    • I totally agree that there’s a lot of outsourcing at NSA as well (Snowden was himself a contractor.) My point is similar to what you said: that the overall project management, strategy, architecture, etc., be handled in-house.