The Department of Veteran Affairs has been in the hot seat several times over the last year after reports of former service members dying while waiting to see a physician and employees tampering with scheduling records to cover the backlog and other mistakes. The VA has also been struggling with poor performance on annual cybersecurity audits, which revealed a significant number of weaknesses and vulnerabilities throughout the department's networks.
As part of the department's steps to remedy these issues, the VA is purchasing new patient scheduling and phone systems, looking to bolster network security while still meeting its mission and consolidating the department's 14 websites into a single nexus for all the services it offers. Stephen Warren, who took the helm as CIO of VA's Office of Information and Technology in October 2013, laid out the office's agenda for fiscal 2015 during a call with several members of the press, including Federal Times Senior Writer Aaron Boyd. Below are edited excerpts from that conversation.
Three topics: what we're doing to update our phone system, where are we on the scheduling system acquisition, as well as activities underway to meet our stewardship obligations with respect to veterans' identity.
Rolling Out New VoIP Phone System
We've had an ongoing program to replace our phone systems. As we sat down we started seeing the trend on replacement, we have approximately 1,300 systems. Twelve-hundred of them are at end-of-life or coming up on end-of-life. We realized that if we just continued doing a one-for-one replacement we were not moving the organization forward from a customer service standpoint.
So we went back and thought it through and realized that we need to change to a VoIP [Voice over Internet Protocol] solution. We needed to move to a Voice in the Cloud that would easily pick up the calls, move the calls, forward the calls, and get them to the appropriate call centers to the appropriate individuals to answer those questions.
We then looked further and tried to figure out what was the best way to do it, and we settled on a government-owned contractor-operated model to allow us to handle the shifts and changes that were coming our way. We actually call this program our Enterprise Voice Solution, or EVS.
So the effort is already started. We're in a pilot stage. We rolled out as our first site for Harrison, Mont. at the Medical Center. That went out a couple of months ago, and it's doing quite well. I learned a couple of things in terms of how we structure items, and the results in terms of speed and responsiveness, everybody has seen better than what we had before. So again for us, it proved that we were on the right path.
We are moving the pilot to 40 additional locations. And that's going to take the first part of this year. We're going to Charleston, S.C. and the Tennessee Valley Healthcare System. Those will be in place by the second quarter of 2015. What we will be doing is based upon the success of the pilot, rolling this out over 10 years to replace the existing 1,300 VoIP platforms.
It's a major investment on our part and it is supporting our drive to get to a better and a centric organization such that no matter where they call or when they call, the VA will be there to support them.
Getting Feedback on Scheduling System
[The new patient scheduling system is] a significant change for the VA. We are looking for a commercially available solution of something that is already out there and already in play. The only development we're looking for is the connection between that product and our existing business systems. So we're looking for something that already works, and the piece that we will be asking the team to develop is, "How do we make those connections?"
When we put that request for information out we were asking, "Did we have this structured right? Did we have it phased right?" We got a lot of feedback and we wanted to make sure that that feedback, as well as any insights that came from the Northern Virginia Technology Council was included in that acquisition.
We will have technical folks there just to make sure the connection pieces are also dealt with, but the primary focus is usability, which is why we are asking for something that already exists because we have seen in the marketplace that the tools and the technologies are already there.
Finding a Balance for Better Cybersecurity
The last area deals with information security and the risk balance. One of the things that I focus on with the team is that we're in the business of delivering care and services to veterans, and we try to do that in the most secure way possible. So in any business that is in the mission of delivering something, you do a balancing act between the risks. You do a balancing act of the pluses and the minuses. The goal is to get those services and benefits out to veterans and allow our nation's veterans to take advantage of what they've earned.
When we do the balancing, information security is one of the things that we balance as one of the risks. And when we do that there are a lot of factors that come into play. One of the things that we're doing to bring broader transparency to how we're managing that risk is we've updated how we do our monthly report. We by law are required to report quarterly to congress any incident that resulted in the potential exposure of a veteran information. We actually do it on a more frequent basis. We do it every month.
We heard from the Inspector General and they've notified us that we're going to continue to carry the material weakness [in cybersecurity] in our financial statement audit for another year. [The VA, unlike most other agencies, has a combined report for cybersecurity and financial audits.]
I was disappointed and I know the team was disappointed, given the significant time and effort that we applied this year. But we're going to continue to drive on that. We've doubled our efforts and we're going to continue to push so that we move forward on the rigorous discipline plan that the team has put together. So that when the audit team shows up next year, they will see the constant improvement, that we've gone even beyond that. There is a lot of guidance out there about what is it to meet the security standards, and we take all of that into consideration.
But, I want to try and make sure I emphasis the goal is not to make things completely secure. If you talk to folks in the IT community they will guarantee absolute security by turning systems off. We're here to deliver services and benefits to veterans, and we try to find the best balance we can with the information we have at that time.
Recently, Secretary Robert McDonald talked about wanting to come up with an overarching structure for customer service at the VA and collapsing some of the nine different regional maps into one overall map. Are you having issues with the balkanization of systems under the current organizational layout of the VA?
[Secretary McDonald] has laid down a very bright line for us in terms of individual organizations setting up single lines of engagement that exclude or preclude conversation with a full range of benefits and services that are available to the veterans is unacceptable. A lot of support and a lot of awareness from the team is that we need to change how we've done and what we're doing such that we make it easiest for the veteran and not for ourselves.
So when I talked about the phone system upgrade and modernization, we're putting the basic technologies in place to support that. How do we bring in individuals who are playing in the commercial marketplace today to help us work on our front door? How do we move away from those 14 different websites, the 14 different user names or passwords, and get it down to a single experience? So we are motivated and the team is focused on how do we move from where we are to use your term "balkanize" into a single, unified, veteran-focused experience, which is about what we can do for them and not what they should do to try and figure out what we would give them?
So it's a big change. It motivated the team. Folks are really excited about bringing in the veteran, focusing that veteran service idea, and executing it quickly.
In your estimation, will you be able to consolidate that into one website? Or, do you need to have a couple of different partitions out there?
The piece that we've got to be careful with is that we deal with joint credentials with DoD. So we need to make sure we're respectful and supportive of that. But we also need to figure out how we find that balance between security and usability. You can do both.
I believe it is reasonable for us to come up with a single experience. It may be folks engaged on that experience coming in from different directions, but we should make sure that when they come see us they see one of us. They don't see multiples of us.
We also need to make sure we continue to team with DoD, as well as some of our other federal partners, again to make sure that the veteran experience is as easy and as simple as it can be.
Back to the VoIP system, what is your estimate at total costs over the 10-year period? How does that compare to keeping the legacy systems or replacing them on a one-off basis?
The legacy systems, if I look at the total amount, we have about a $3 billion cost for all of those systems. When you do a PBX [private branch exchange] change-out, since we're talking about PBX's from medical centers, they're running $20-$25 million. If I start looking at some of the other systems, it adds up to a large amount of funds.
When we sat down and we did the cost comparison between, "Do I change the physical plant, or do I move to this Voice in the Cloud solution," it turned out to be notionally cheaper on the Voice in the Cloud. But the one big thing we got out of it was the fact that we were able to put in place a customer or veteran-focused solution. The ability to move calls, transfer calls, and seamlessly route them around using the IBRs [introduction-based routers — a form of network security] on the front setting up menu systems and allowing that to be done easily as a part of how the service is provided, versus having them bolted or added on.
So we were doing the replace, replace, replace, and the team sat down and said, "Wait a minute guys. When you think about what we need to be thinking about, what we need to be doing for veterans in terms of how do we service, this is a path that is building in barriers and obelisks for the future." So that's why that effort started about three years ago, they came in with the conclusion about 18 months ago, and then we moved out on it last year so we can make sure we've got a jump on this because it's a long-term effort. The VA is a nationwide system, and so there's a lot we need to replace and there's a lot that we can do as a result of taking this path.
Can you give more detail on what the VA is obligated to do for the next year to meet the IG's recommendations on cybersecurity?
What they've done is they've actually said, "You've done some great stuff in the past year, but we need you to work harder in four major areas. You need to be working harder in terms of how you manage the configurations of that million-plus system. How do you make sure you do them in a standardized, consistent way?" So the auditor is asking us to do more there. Look at it and make sure the folks out at the sites, the individuals doing the work, are complying with the standards and consistently implementing them.
On the access control, how do you make sure as we do the reviews, as people come in and go out that our HR system is linked up to our IT systems in terms of who has access, why do they have access, and on what frequency do we do reviews on that?
Also on security management in terms of how do we fully implement from the auditors perspective? Are they moved from one point-in-time accreditation to a continuous monitoring of our systems? They pointed out to us a couple of areas where we need to do more, we need to do differently. So we're driving on that.
And the last one is on the contingency planning side, how do we make sure some of the controls in place are there? They've laid out for us broad areas we need to work on. We've got a pretty detailed plan that the team put together in the past year for the past audit season. And we've looked at that plan and have taken the insights that they've given us. We've gone back and we've updated the plan and the team is already moving out. Even though the audit season starts in March and even though we drove pretty hard for the last seven, eight months, we starting driving the day the auditor said, "Nope. You didn't do enough to clear the material weakness."
Can you give some sense of how many of those recommendations are still open, how many of those you've knocked out and how many new ones there are?
This gets to the risk balance conversation. The team identified a number of vulnerabilities. What's important to recognize is that if I'm running on a base of 1.2-1.4 million devices and I'm running multiple services on each one of those devices, you're talking about 70-150 million different things that you're looking for vulnerabilities on. Consider the fact that I scan and monitor 45,000 different applications that we run across or identified and I've got 1,000 systems that we built and deployed that are enterprise systems, we have about 6,000 vulnerabilities and we treat every one of them as important.
But if you look at it on the scale you've got to put some balance on that. We've taken down the number of things that they identified by 21 percent in terms of what the FISCAM [Federal Information System Controls Audit Manual] findings were. We went in and identified a whole bunch of recommendations where we believe we had closed the recommendations. The audit team accepted some of those, but they also came back and said, "We need a little bit more evidence on these over there." So we're constantly negotiating with them and when they publish their report — I believe in the spring — we're just going to keep driving on.
The team keeps evolving. The defenses keep evolving based upon the threats. We take what the auditor has given us very seriously and we just build it in and just keep driving on.
The thing that we struggle with is security is not a one-time deal. The standards change every year. The industry practices change all the time. So one of the reasons we put the Continuous Readiness and Information Security Program [CRISP] in place was to get the organization, not just IT, to recognize it is something we need to work at every day. It is something where we need to be constantly evolving, constantly improving, constantly changing, as a threat change or a defensive posture change, moving from reactive to proactive. And so CRISP is making sure that the VA as an organization, not IT as an organization, recognizes that the practices that are used in day-to-day behavior is critical for how we protect veterans' data.
The times where we have fallen short of our stewardship responsibility has been in the process, paper, and people standpoint. It hasn't been on the cyber side. So that's the place where we keep working with the team. We keep educating, "What does this mean for you?" It's not a tradeoff between services and benefits for veterans and risk. It's a balance between the two.
You described a situation where monitors were blocking off access to sites that appeared to be pornographic, but it turned out to be doctors researching things involving body parts. Can you explain that a little bit?
[That anecdote] was trying to do the balance between mission delivery plus protections you put in place. Most tools where they actually categorize particular sites, it's either done by an automated tool or a person looking. There's an automated tool that scans websites and it looks for pink – flesh-colored pink. And if it crosses a certain boundary, it just automatically scores it as pornographic and it moves on from there.
So we have incidences where clinicians are trying to do their job, get to a site and it's blocked because it's claimed to be a site that you should not go to. There are behaviors and practices that folks need to follow as a part of doing their job. We actually have to make sure we tailor what it is that we need to do based upon mission delivery.
When VA announced in August that the RFP [for a new scheduling system] was going to be released the timeline was September, then it was October, and now we're looking at the end of November. Is it just making sure all the "I's" are dotted and "T's" crossed, or are there any kind of unanticipated hurdles you're running into.
It's making sure we do it right. One of the things that has allowed us to spend a little bit of extra time to dot all the "I's" and cross all of the "T's" was we have already awarded a contract to take the existing scheduling software and change the interface and make it easier to use. That was also drawn up in the spring. The team is making great progress and we're expecting first delivery I believe in the December timeframe to go to the first site to start testing. So the fact that we're bringing relief, we're bringing that capability out to the schedulers to make their lives easier so that they can do the scheduling right. We are paying for somebody to build the interfaces and to put them out in the open so we do it right.
It's also that we made sure we built in an engagement with the VSOs [veteran service organizations]. So we brought the veteran service organizations in and we briefed them on the requirements, on the contract and what the structure is and the process and methodology because we wanted to make sure we weren't missing anything. And then we had [the requirements] reviewed by the Northern Virginia Technology Council. We wanted to make sure of any insights that they had. I would say that's just as important.
We want to make sure that when this thing goes out that we bring in the solution that's going to solve this problem once and for all, and that we've given an appropriate amount of time so that it gets done right this time.
What was the general tenor of the comments that you got from NVTC, the veterans groups, and potential bidders?
The veterans groups have been very, very positive about, "Hey, we love this approach. We love the fact that you've opened it up to us. We love the fact that you've sat down with us." We actually built along the way making sure we got release.
We deployed a scheduling solution for video teleconferencing. So veterans at one location can get support from a different location. So we built that solution. We took it to the VSOs and they said, "Wow, this is great and glad to see you're moving it." And they've also come back and said, "This is unprecedented engagement and we love it." In fact, I have another meeting with the VSOs because I'm maintaining that engagement with myself and the folks on the team. We're keeping them aware, and not just keeping them aware as in, "Hey this is what we're doing," we're asking for insights. We asking for advice, "Are we missing something here? What should we be worrying about? Is this approach going to get there or not?" So there's a lot of engagement there.
The NVTC came in with some insights. The majority of things on the NVTC list are primarily process-driven. They are primarily how the process needs to change, how things are organized, and how they're structured. They gave us some thoughts about where we were on the technology side. We either had things underway or the team is digging into it to understand what it is that we need to do. As an example, "Give all the schedulers two monitors." We had actually already identified and moved out on that a couple of months before.
Is there any timeline for the consolidation of those 14 websites?
The team is actually developing that plan and schedule. We've got all the major players together to talk about it earlier this week. And then next week is the first organizing and, "OK, what are the different capabilities? How do we need to start tracking those and aligning those?"
The complex work for us is how do we take work that's already underway on the multiple websites and relationships that already exist and flag the ones that should continue because it's more back office and add to the ones that are in the front that deal with the interface or the experience? So right now what we're working through is, "What are all the efforts underway that align with what we need to do?"
We need to make sure we lay them down in terms of what are they doing and what are they touching? We need to map them against what do we want that future seamless experience to be? And then make sure we phase and sequence them to get to that output. But that's one that the team will be wrestling through.
And as we lay those things out, we have a discipline at the VA we call PMAS [Project Management Accountability System], which is how we do agile delivery. It's the thing that has allowed us to bring in solutions on about 80 percent to the schedule. And we'll be applying those same disciplines to this effort as we do to all development efforts at the VA.
I would give us probably after the holiday season, because we've got a lot of folks who've been driving hard.
Aaron Boyd is an awarding-winning journalist currently serving as editor of Federal Times — a Washington, D.C. institution covering federal workforce and contracting for more than 50 years — and Fifth Domain — a news and information hub focused on cybersecurity and cyberwar from a civilian, military and international perspective.