SolarWinds: What Really Happened?

Posted on in Presentations

SolarWinds was hit with the most sophisticated supply chain attack in history, rattling the private and public sectors. Many researchers have shared what they know about this attack. For the first time, SolarWinds shares its unique view of the attack’s “what, how, and who”, including key learnings about the novel tradecraft that can help the industry better prevent and protect in the future.

Video Transcript

- Hello, I'm Laura Koetzle from Forrester Research, and it's my very great pleasure to welcome Sudhakar Ramakrishna, who's the CEO of SolarWinds, to RSA Conference 2021. Welcome Sudhakar.

- Thank you, Laura. Thanks for having me and I'm delighted to be here.

- Thank you. So everyone listening is mostly familiar with the broad outlines of the recent breach, but for those who might want a refresher, I'm going to give you a very quick whirlwind tour of the key events, and then Sudhakar and I are going to begin our conversation. So let's go back to the 8th of December of 2020, when FireEye announces that it's been the victim of a nation-state attack, and its Red Team Toolkit has been stolen. On the 9th of December, so the next day, SolarWinds announces that Sudhakar is going to be joining its board and taking over as CEO on the 4th of January of 2021. And then on the night of the 12th of December, Sudhakar learns about the compromise of SolarWinds Orion. At this point, very few people know anything beyond the FireEye breach announcement. The next day on the 13th, FireEye announces that while investigating its breach, it found evidence that attackers used a backdoor in SolarWinds Orion, which they codenamed "Sunburst." Also that day, SolarWinds starts notifying Orion customers, and asks them to upgrade immediately to address a security vulnerability. On the 15th of December of 2020, we learned that the compromise of Orion occurred as early as March of 2020. So it had gone undetected for months. At this point, we find out that the Department of Homeland Security, the Department of Treasury, the National Institutes of Health and the State Department and more, here in the US, are all affected. And remember, at this point, we're still in the middle here in the US of an acrimonious transition between the outgoing and the incoming US administrations, with lots of key roles vacant, or with recently named acting heads. On the 19th of December of 2020, then Secretary of State Pompeo says that the Russians were behind the attack. And that's the first time we the public hear that. On the 31st of December, so Happy New Year, Microsoft says some of it's source code was compromised by the attackers. At this point, everyone's thinking the attack began as far back as October of 2019. On the 4th of January of 2021, as planned, Sudhakar officially took over as CEO of SolarWinds, and he flew to Austin from California. On the 6th of January of 2021, CISA issued guidance requiring agencies with affected versions of Orion to do forensic analysis, and violent insurrectionists overran the US Capitol. On the 29th of January of 2021, SolarWinds issued an advisory for both Sunburst and the malware SUPERNOVA. And on the 26th of February of 2021, Sudhakar testified at a congressional hearing, which is a heck of a second month in the job. So now here we are in May of 2021. So let's go back to the night of the 12th of December, which you may remember from the very beginning of my lovely chronology here. Sudhakar, you were sitting down to your own birthday dinner, right? And you got a call from SolarWind's General Council, and he told you about the backdoor, and that there was malware in the offing. Tell us a little bit about that conversation. How'd that go?

- Laura, it is indeed true that December 12th is my birthday, and I was sitting down for my dinner, birthday dinner. It may have been around 9:00 PM, 9:30 PM Pacific, so about 11:30 PM Central time. Jason Bliss, our chief legal officer, and now chief administrative officer, gave me a call to provide me an update. Turns out that he texted me first, and then I asked him, "If it's urgent, I'll call you back right away," and he said, "No, give me a call in half an hour or so", which I did. He updated me of the incident, and Jason, if you know him, is a no drama guy, he's very plain spoken, and he said, "This is what we found out today. FireEye reported that there was a backdoor into the Orion platform." And we chatted a bit. At that point, Jason did not know a lot of the detail. So when you talk about backdoors, not immediately does a supply chain backdoor come to your mind, because there's a lot of different ways backdoors can be installed. And what I said to him at that point was, it's quite ironic that he talks to me about a security incident that day, because just that morning, as I was preparing for January 4th, I was preparing a list of things to focus on, one of them being the security posture of the company. I did not realize at that time that I would be focused on that as my top priority as I joined the company.

- Okay, so be honest, at this point, do you think about calling the chair of the board of SolarWinds and saying, "You know what? Find a new CEO, I'm out of here."

- Let me be honest on that front, as you asked. I did get a lot of feedback, and a lot of feedback from very well wishes of mine asking me to back out, and telling me that I had nothing to prove, that I am able to find the next best job, and on and on and on. You can say I'm a stubborn optimist, but I decided to persevere with this opportunity, but I did have a call with our chairman, Bill Bock. The topic of that was different than saying, "Go find a new CEO," to your point. The topic instead was, we are in a state of crisis, because the more I learned about it over the next three or four days, once it came into the print media, I felt that continuity and urgency were super important in this situation. And having a new CEO come in and figure out the team, figure out the procedures, understand the issues, could be time-consuming. And so I had offered to Bill if the right decision was to continue on with the previous CEO, that I would be totally fine given the needs of the company and given the needs of the customers. But then it turned out that I was able to come in and I also had continuity and support from Kevin Thompson, the previous CEO, and we were able to navigate it quite quickly.

- So since you brought up customers there, let's talk a little bit about your customers. Because when you started notifying customers about the breach on the 13th of December, you internally I think at the time thought that as many as 18,000 of your customers might've been affected, because they'd downloaded the affected update, right?

- And so--

- Yes.

- What kind of plans did you have in place prior to the 13th, knowing that you're not CEO yet at this point, right? So you're sort of in a little bit of a transitional state. But I know you were talking to customers then. So what were customers asking? What was the hardest to answer? And what kind of plan did you have in place at the time?

- Similar to most mature software companies, SolarWinds also had a product security and incident response team under our VP of security. So that team was used to understanding security incidents, communicating to customers in a structured way, working with our customer success organization. So that machinery, so to speak, was in place even prior to December 13th. Obviously, these events had a escalated significance on that team, in terms of communicating to our customers, so it was quite literally all hands on deck. And the most important questions that customers had at that point was, what does it mean to me, and what do you want us to do? That was really kind of more of a reactionary thing at that point in time, which the team rallied all around and did the very best, I would say, to touch every single customer possible.

- Got it. So here we are in May at RSA Conference, which means we're five and a bit months into dealing with the fallout of this. So how are you supporting your customers now, and kind of how you plan to continue to support them in the future? Because obviously there's a long tail on this with all the forensic investigation and things that customers need to do to change their own procedures and so on.

- Continue to is the operative word, as you mentioned, Laura, because a lot of our software runs on premises as well, so it is not instantaneous that everybody updates at the same point in time. So it is one customer at a time, essentially one day at a time. And in many cases I have told my team one step at a time. So what started off as a reactive measure, we started learning about the incident, we started addressing issues, and one of the foundations of what we've been trying to do is transparency as we enhance the trust that we have with our customers. Specifically, we also worked with our worldwide partners and created a program called the Orion Assistant Program. The idea behind this is we recognize that not all of our customers may have the internal resources to upgrade or rebuild or project into the future. So what we decided to do was work with our partners and extend support to our customers to essentially provide pair of hands in some cases, technology commitments, in other cases, and in many cases, work side by side with them as they completed their upgrades. This we did at our cost. Therefore the customers did not have to incur costs to support this program. And we felt it was our responsibility to help the customers get to a stable and secure environment. And so that was very well received and appreciated, not just by our customers, but also by our partners. Because one of the key things that happened through this process is it brought our partners closer together, and they understood our products better, and they understood how they can serve our customers better as well. So this is going to be an ongoing thing as we move into the future and we deliver different, better and extended solutions to our customers.

- And thankfully this wasn't the first time that you'd been in a position of authority during a breach, because prior you were the CEO of Pulse Secure, and you had a number of vulnerabilities in the products of that company, that you were the CEO of at the time, during your tenure, including one that was on the top exploited vulnerabilities list for 2020. So thankfully this wasn't your first rodeo, as they say. How did those experiences inform what you just described in the way of supporting customers, and sort of what you decided to do and to reveal to the community as the situation unfolded?

- I wish I could use the word thankfully in this context, but to your point, it does help to have experience here. Coming from the software industry, security vulnerabilities unfortunately are commonplace. The important thing to recognize for all of us, no matter how big or small we are, is that we have to be prepared at all points in time, but be humble enough to accept that security vulnerabilities and breaches can happen to anyone, not withstanding what resources we have and how good and great we are. So one thing that my Pulse Secure experience taught me, and even in prior companies, I would say, is that at all points, you've gotta be vigilant, but all points you have to be humble, because you cannot think that this won't happen to you, it might only happen to others. So when you have that mindset of always being vigilant, always being humble, transparency is very, very important, because it's critical to know what's happening, and when something happens, that you are able to communicate it, project it, take ownership and do something about it. And going back to my Pulse Secure experience, similar vulnerabilities to what we had occurred to many different vendors at the same time, whether it was Cisco, Fi, Palo Alto Networks and others. And in the Black Hat Conference, many of the researchers actually gave us, us being Pulse Secure at that time, compliments for how we handled those incidents. And those were rooted in being transparent, being communicative and updating everybody on progress, even at times when you do not have all the details in place and set in stone. So I tried to carry forward a lot of those principles into my experience here at SolarWinds.

- Got it, and I think a lot of us saw that as you guys shared how things were unfolding as they were unfolding, rather than waiting until you knew everything. Because at least in my own observational experience here, you never know everything.

- Absolutely.

- So speaking of never knowing everything, at this point, as far as we, the community, know, the consensus seems to be that attackers first compromised SolarWinds in September of 2019, and they were there and undetected until all of this started to unfold in December of 2020. So is that right? And how were they able to stay undetected for so long?

- Laura, as we have published, the tradecraft that the attackers used was extremely well done and extremely sophisticated, where they did everything possible to hide in plain sight, so to speak. We were looking for all the usual clues. When you go through an investigation, you have a checklist, you have a set of hypothesis, you try to map things. And in this particular case, given the amount of time they spent, and given the deliberateness that they had in their efforts, they were able to cover their fingerprints, cover their tracks at every step of the way. Given the resources of a nation-state, it was very difficult for one company such as ourselves, or as you have seen, other companies that are coming out with their own reports of other nation-states, as well as the same nation-states breaching their assets, it was a very difficult thing to uncover. But as we've been looking back into our history and we have stumbled upon, as is the case with many investigations, some old configuration of code where we were able to track down what exactly the attackers did. And this was, to give you a perspective, we were assessing hundreds of terabytes of data and thousands of build systems, virtual build systems across the environment, just to give you a perspective. And what we have found more recently is that the attackers may have been in our environment as early as January, 2019. We published, obviously, that it was in the September, October timeframe, but as we look back, they were doing very early recon activities in January of 2019, which explains, I would say, what they were able to do in September, October of 2019 as well.

- Got it, okay, so even earlier than we had previously thought up till now.

- Absolutely.

- So switching gears to what was going on inside of SolarWinds, and has gone on for the moment, you got lots of friendly advice about what you should do about your new CEO gig, given the release. And I know that you also got some friendly advice about what you should do about the head of security who was in post at SolarWinds at the time. So do you still have the same CISO as before? What did people tell you? The reason I ask is because, of course, a lot of what we in the community worry about is, hey, something goes wrong, you know, the CISO is the sacrificial lamb, as it were. So tell me what happened, and what kind of advice you got, and what you did about it.

- You're spot on. I got all of that advice and more . And I'm pleased to report that our CISO Tim Brown was there before I joined and is still a CISO. This is a very important part of how I like to lead. I do not like to flog failures, so to speak. And it is not even clear that this failure is one person's fault. When a nation-state attacks you, it is impossible for one person to be able to thwart that entire attack or take full responsibility for it. I also felt, as I got to know Tim Brown, that he's a highly competent and highly committed individual. He is a 30-year veteran of the security industry, was a former Dell fellow. And if I were to go out and hire somebody, I might have hired somebody with that profile. So while I acknowledge and accept that if you want to be an action-oriented CEO, so to speak, in quotes, that you come and fire a bunch of people. And I do not think that is doing justice to either the person or your job, because you are really paid to get the most and the best out of the people that you have. And yes, accountability matters, but just like CEOs get a lot of credit when things go well, and unduly so, I would say, being a CEO, I do think that some CISOs get undue discredit, and I felt that I should not be doing what is the norm or what is stereotypical in these situations, and went about my own way.

- Well I think everybody's probably relieved to hear that, that you decided to discard that lovely advice. So let's talk a little bit about other SolarWinds personnel. So you had the joy of testifying before Congress in your second month on the job, as I mentioned. And there was a lot of focus about an intern who posted a password that had nothing to do with this particular incident. But the reason I wanna ask you about this is not to sort of shove splinters into things, but to say, thinking about the folks who are early in their careers, who are in university now, who might wanna be interns at security product companies, they might be a little worried about starting a career in SolarWinds, because they think, "Oh God, if I make a mistake, I'm gonna get thrown under the bus in front of Congress." So how are you gonna reassure those folks that that talent should join your company? Because obviously scarcity of talent is something we all talk about all the time.

- Absolutely. Laura, first of all, I have long held a belief system and an attitude that you never flog failures. You want your employees, including interns, to make mistakes and learn from those mistakes and together we become better. Obviously you don't wanna make the same mistake over and over again. You want to improve. So what happened at the congressional hearings, where we attributed it to an intern, was not appropriate and was not what we are about, or is not what we are about. And so we have learned from that, and I want to reset it here by saying that we are a very safe environment and we want to attract and retain the best talent. And since you mentioned universities, I have been a big fan of university programs and internship programs. And you will actually see us amplifying those. And hopefully you will hear more and more interns saying how great an experience it is to work at SolarWinds, as opposed to one incident that may have happened at a hearing.

- Got it. So I'm gonna close with this question, because I know we're going to get ourselves kicked off the virtual stage very shortly, which is looking back over the last fiveish months, if there were one thing that you could go back and do differently, after you took over in January, obviously, because no points for saying your predecessor should've done things differently, what would it be?

- Having a stronger media response is the way I would describe it. Let me explain that. SolarWinds historically has kept to itself, focusing on customers, focusing on itself internally. And it was never trying to grab attention. In this particular case, the attention was thrust upon us. And if I thought about one area where we were not fully prepared, unlike some companies that have armies of PR people just managing the message, and, in many cases, neutralizing it, we were not prepared. And to your question about reflecting back, I wish we had more resources, more proactive outreach. And we have learned from that, and we continue to grow our communications team. We have an excellent communications team. It's now a matter of giving it a bit more support, so we can be more prepared for this and others, although I do not wish something like this to happen to anybody in the industry.

- Well, thank you so very much for being so candid with us today and sharing the new things that you've learned in the interim. So everyone please join me in thanking Sudhakar for being willing to share so much with all of us, and thank you all as well for listening, and enjoy the rest of RSA Conference 2021.

Laura Koetzle

Vice President, Group Director, Forrester

Sudhakar Ramakrishna

President and Chief Executive Officer, SolarWinds

C-Suite View Hackers & Threats

endpoint security hackers & threats ransomware endpoint detection visibility & response artificial intelligence & machine learning



Share With Your Community