Building Effective Cybersecurity Defenses with eSentire's CTO Dustin Hillard

Jon Prial: Welcome to the final podcast in our series on cybersecurity. If you liked it, please rate or review us on your podcast app and talk to us. Find us on LinkedIn, on Twitter, our handle's @ georgian_io. So let's just get right to it.

Dustin Hillard: My name's Dustin Hillard. I am the CTO at eSentire and I came to eSentire almost three years ago now through the acquisition of a startup that I helped found called Versive, and so that was eight or nine years ago now. We had a fun journey there. We started like any good ambitious startup with the moonshot of trying to be the machine learning platform for the enterprise and solve all problems. So, we made it a good portion of the way down that path and had some of the largest banks as our customers and investors, but found that actually that's still a pretty hard problem to solve and that no one's quite cracked that yet. So, the last couple of years there, we focused in on a particular application and we were doing advanced adversary hunting for security. So down that path is how we came to see a good fit with eSentire, eSentire had had rapid customer growth really was fitting a major need in the security market. At Versive, we had built a lot of the core, underlying AI technology that could enable these types of use cases, but we weren't quite achieving the customer scale. So, this was a really nice opportunity to come into a place that had a large customer base and for any data science, machine learning, AI person, a a lot of data and actually even better here is because they're security analysts, constantly labeling this data to stop customer problems. It's this massive, large labeled data set that was sitting there, ready for people with the right technology and machine learning chops to take a cut at it and it's a pretty exciting opportunity because I think the outcomes that you can get there are really pretty valuable. There's not the justification of working on cybersecurity today, it's not really needed anymore. It's more, what can you accomplish? So, that's a pretty exciting place to be.

Jon Prial: I like the fact that you came in with the ML skills and you saw this awesome database label database that," Oh, we could attack this and do all kinds of good things," that's just awesome.

Dustin Hillard: Yep, absolutely.

Jon Prial: So you're actually the final step in this podcast series, in helping companies better understand how they're going to build up their cybersecurity defenses. But of course, no matter how good a defense is built, something is going to happen. It's been a while since I spoke to eSentire on a podcast and the company has clearly grown and evolved over the years, but one of the messages early on was a message that was quite simple. It said, you will be hacked. Well, we know it was true then, it's definitely true now. But from your perspective, have things changed? Have the odds shifted one way or the other in terms of companies versus adversaries?

Dustin Hillard: I guess you'd have to say probably the adversaries are still gaining the advantage. I think if you take a simple metric, like how often you hear about it in the news, you'd have to say that just probably more today than it was five or 10 years ago. I think we've been pretty fortunate with our customer base that we haven't had any of those major breaches ourselves. I think incidents are a given these days though. A big part of how we think about it is that inevitability of a breach is, I think every cybersecurity person that's got experience, will tell you that's going to happen. So, it really is starting to come down to how quickly can you identify that something malicious has landed and gotten access or started to act and how quickly can you then contain and stop the business impact of that? So, it's really about minimizing the business losses and whether that's ransomware that expands and takes over a bunch of machines and so business continuity is becoming probably... that's something that wasn't as prevalent, I think probably five years ago, but today a ransomware attack can literally shut down your business. But to do that, it lands in one machine typically, whether that's through phishing or some open vulnerability or whatnot, but it's how quickly you can identify and contain that is starting to become, I think really a big part of the success metric for what cybersecurity can mean.

Jon Prial: This really maybe is an AI/ ML type question, are you always playing react and respond? I know this section, this whole podcast, is about responding, but I'm at the more higher level thought, that an antivirus... this goes back a million years ago, the antivirus, somebody creates a virus, they figure it out and they put it on a published list of viruses, everybody else to detect and they update their software. Are they always ahead? Or can you put your white hats on and figure out, what's coming next and prevent that as well? I know we've talked about prevent an earlier podcast, but I'm curious your perspective in terms of can things be prevented before someone on the bad side, figures it out? Can you get ahead of them or you always have to respond?

Dustin Hillard: This is one of my favorite things of thinking about how machine learning can be used here, because I think the traditional answer would be security's super tough to be on the defense because the adversary only has to find one hole in your wall and being that we're all human, there's inevitably going to be a hole, whether that was a misconfiguration of somebody in the IT or security team that's likely to happen at some point, the important part of all our companies are the humans that do our work and it's very difficult to always avoid every phishing email. Even I've clicked on them before, knowing that I never should. So, I think the inevitability of that gap appearing is I think the traditional answer for why it's hard to be on the defense because you basically can't have a perfect defense. I think the way to think about how you can have a more proactive defense, and so what are the advantages you can have as the defender? I think that's knowing your own terrain and your own environment. So, especially for identifying unknown attacks, this is an area where you can have a much better assessment of what should be happening in your environment because it's your environment. So as an attacker, assuming that that attacker is going to get in, how quickly can you identify them and or how hard can you make it for them to hide basically. So if you do a really good job of knowing your environment, such that anything unusual happens and it pops out and you get identified, for your attacker to hide from that, they basically have to know your environment as well as you do. So, I think that's a way to turn the tables and think about how can you as a defender actually have the advantage? That's knowing your environment extremely well, having great telemetry and understanding, such that you can identify unusual behavior and react quickly to it. That's something that, for an adversary to attain that same level of understanding of your environment, the level of activity they'd have to have on your network to actually do all that discovery and reconnaissance and understanding of your network would inherently... they would stick out. So this is a way to turn the tables and think about... make it as for your adversary to attack, they have to know it as well as you do.

Jon Prial: So, now I got this vision of some movie and the safe crackers are going to come in. The first step is somebody's got the map and now they can figure how to break into the bank. So, I got the map, they don't have the map, I can do a lot... That's a great way to think about it. I love that.

Dustin Hillard: And for them to obtain that map, that's the hard thing. So, you got to make sure that your detailed knowledge doesn't leak so that-

Jon Prial: Exactly. So I want to get into, obviously both the technology and the people side and responding to threats, which we will get to. But before we get there, I just want to spend a little bit with you on how you view detection of potential issues. Take us through what you see as signals. I mean, one of my favorite terms is the term a sniffer a sniffer's just in a network. It's just a great word, but there are obviously more than that. So what do you view when you think of the type of signals that are out that might be detected?

Dustin Hillard: Yeah, and for me, I think of two major classes here, and one is what you might call the more traditional ones that are around signatures. So, the most basic of these are antivirus and known bad files and hashes and so forth, and then further network signatures. There can be more complex versions of these as well, but they are basically, this type of malicious activity has been seen in the past, identified by the security community. It gets put into a common set of detectors and these are the things that every good environment should have these in place because they are the commodity tools that any attacker could obtain and start to use. A lot of attackers are successful, I think that's one set. A separate set is these more behavioral ones and they're the more advanced set where if you think about an attacker that's using... There's a lot of great thinking around the MITRE ATT&CK framework and that's, I think, an important way for the security community to be aligning around what kinds of attacks there are. But if I were an adversary with unlimited resources, I'd look at that and say," Great, here's the things I'm not going to do, because those are the things people are looking for. And I'm going to go try to use a technique that doesn't sit this attack framework because there's a much lower likelihood. People are looking for those kind of things." So, I think those sit in that more behavioral realm. There are behavioral techniques on ATT&CK framework as well. But when you get outside of the signatures and thinking about what advanced adversaries are doing, they use the known attacks as the things to avoid and they're developing novel techniques, and those are expensive to develop and deploy. But I think that's the other area to be thinking about, especially in what's evolving and the recent SolarWinds exchange are good examples of that, where they were previously unknown vulnerabilities and attack methodologies.

Jon Prial: You talk about learning and keeping databases of everything that's been done. So what percentage of this part of the business, of just detecting of things, is automated?

Dustin Hillard: At eSentire we have a mix of that and I think a lot of places are doing it this way, where putting the best in class tools in place means that a lot of these baseline detections that are catching all of the known threats are relatively automated in nature. So that detection framework comes along with putting those best in class tools and that foundation in place. The second layer that comes after that, that's still a challenge in security today, is the noise that's generated. There's an inherent level of false positives that also come out of those tools. So, a big part of what we're doing is the volume. We get millions to billions of events coming out of these standard best in class tools still, but the automated filtering and reducing of the noise to get it down to things that are worth a human looking at is a big part of what we focus on. There's a primary detection that comes out, but then what kind of orchestration and contextualization can we use to put around that event and understand whether that's expected for some reason, maybe it was a security scanning tool that did that. So it's expected, that's a simple example, but there's lots of other business reasons why that may have been valid. So, understanding the context around an event and being able to filter out the noise of basically false positives that nevertheless triggered a signature, I think is a lot of automation work that we've put into that. Some rules- based and some machine learning- based, and so there's different flavors of automation there.

Jon Prial: Interesting. So, let's do one last intriguing example for me, that's always fine. You actually said it, that you've clicked on a phishing expedition and I have also admit that I've clicked and then went,"What did I just do?" Let's do the human side, the social engineering, the HR side of things. If we have a, I don't know, disgruntled employee, are there detectable behaviors that you can figure out on bad behavior inside a company or bad behavior in terms of clicking on a phish that might be more detectable? I'm just curious what's your sense of on that, how you break through the human challenges that exist? Because you said we're all human, there's going to be glitches along the way.

Dustin Hillard: Yeah, I think those are two separate classes. There's the accidental ones. So I think for those, they may be a slightly easier class of things because it might be when a human accidentally clicks on a phishing email, they go to a site or trigger some behavior that the good standard security tools can identify that and flag and same thing even for IT or developer mistakes, or misconfigurations. A lot of the cloud monitoring tools for instance, will pick up on those kind of things.

Jon Prial: Human error, I got it, and I'm that I'm not alone. I'm actually glad there things that be done where we have those clicks that should not have happened. Now, and?

Dustin Hillard: The flip side of that are those insider type of attacks where someone is disgruntled or maybe they've been coerced even, in some of the worst cases into... because depending on how motivated an external actor is, that's another methodology, is to coerce an employee into doing something, whether knowingly or not

Jon Prial: Have to say, I didn't think about those three different elements, but it makes a lot of sense. Once you understand them, it makes it that much easier to solve the problem too.

Dustin Hillard: I think that gets more into that behavioral realm again, of understanding what are the behaviors indicative of an attack and what do people need to achieve that goal? So, we think about this in terms of some of those core behaviors, of one step that needs to happen is this reconnaissance and discovery. So, that's that, like we were talking about, the mapping of the network, understanding where the resources are, where the data or targets are, or where the weaknesses are, if you're implementing a destructive attack. So that's that mapping phase. Another is this collection or lateral movement, so expansion and movement through the network, because oftentimes the place you're able to land or that you have natural access as an insider, you need to pivot from there and move to other locations to get at your targets. Then the last stage of exfiltration or destruction, where you're getting the intellectual property out of the environment or the information that's of value to you, or executing a destructive attack. So we think about, you can see those behaviors from the telemetry and environment. So identifying those type of malicious insider or... Advanced adversaries look very much like insiders, they've obtained access somehow and then they have to go through these same stages. So, we think about looking for those advanced behavioral attack elements and building a case out of putting those together. Each individual behavior can be pretty hard to weed out, but if you can aggregate those behaviors together and see multiple malicious behaviors, emanating from the same set of small hosts, that that's a good seed for a threat hunt.

Jon Prial: Threat hunt, I like it. It's just so much more pointed. Not necessarily more accurate than detection, but it's a great way to end this section. Now, let's move on to the next element of, we can call fun stuff. Let's do response. So, take us through the role of a security operation center.

Dustin Hillard: Yeah, we've talked a lot about that detection element. So, the way that we do this is we have a lot of telemetry and signals feeding into a platform, it's aggregating all these signals and trying to filter them down and put just the most interesting ones in front of our humans. So we've gotten to a pretty good place there, in the realm of one in five investigations that they do are going to lead to a true positive.

Jon Prial: Wow.

Dustin Hillard: So, they're going to be talking to a customer about what happened. That doesn't mean it's a breach event, it's something got past your firewall and what can you do to improve that, because you'd rather not that get into your network. The number of cases where malware actually lands and there's a larger risk is obviously much smaller and then that, but that's what the core flow looks like for us, is these massive amounts of signals coming in, weeding those down, aggregating them into investigations. So, what it looks like for an analyst is they're seeing all these different signals, investigating, trying to understand whether this is a true positive and then what actions are going to be required for a containment. So, once you've validated that this is a true positive event and it's actually the malicious activity that was potentially identified, then we started thinking about the remediation and containment steps. So, depending on what tools are in place, there's a couple of different major ways to contain a threat. One is stopping the network communication. So if it's an external actor that has a command and control interface, or if there's data that's leaving the network, then that's one of the important elements, is can you break that connection and stop the ability to control that host? So you can do that from the network is one element. Host isolation from EDR tools is another major one. So, doing containment to that host, and that essentially locks down all network connections.

Jon Prial: You mentioned one out of five events that a person sitting inside your SOC sees is actually relevant. To me, that's a function of how good the data you have is, and the labeling is and how you can identify what risks are. How have you evolved over the years from that? I mean, that to me, that's pretty amazing that maybe better is five out of five, but then again, whoa, maybe then you're missing something, there's always... So talk to me what your thoughts are on this one in five.

Dustin Hillard: Yeah and that's one of the core things, a way that I get measured. I mean, to your point, the customer's metrics, they don't necessarily see every investigation that we do, They just see the outcomes. So from their perspective, they want to make sure we never miss internally and to be effective and scalable and profitable, we also need to make sure we're doing that efficiently. If we're having to look at 100 things for every one time we're talking to a customer, then that's not working out too well for us in terms of the level of human effort that we have to put in. Recently, we were in the 10 to one range and further back it was 40 to one. So, we have had this march of improvement in driving that down and that's a big part of how we continue to scale. The MDR businesses, I think have to constantly track this, because to obtain the high margins that investors like and be closer to SaaS and product- like businesses, we really have to focus on this and be able to get the human effort focused in on the most efficient way possible to serve the customer outcomes without a huge number of humans required that makes the business untractable.

Jon Prial: We do talk about SOCs a lot, but I'm really glad you used the term MDR. A security operations center is what you provide in this market, that's called MDR, managed detection and response. I'm glad you were talking more about your business numbers. It's important to talk about the balance of humans and I'm really glad to hear you say, you're going to scale on the human side, but you only want the scale as you get more and more customers.

Dustin Hillard: Yep, absolutely. I think there is a network effect element in this all as well. So, as we obtain more customers, when we see a new threat at one customer, a big part of what we're doing with our platform is feeding that back immediately. So we're at around a thousand customers today, so we can immediately protect all 1,000 customers once we've identified and operationalized the response to that threat. So as we grow, there are some of these network effects that can help us, the automation and on the machine learning side as well. The larger your data grows and the more you can learn from it, sometimes I make the analogy to Google they ran away with the search market because they had all the users and so they were seeing what people were clicking on. So, the more efficient you get at seeing what people are doing, understanding it, learning from it and feeding that back to your customer base, you actually start to win with scale because if you're appropriately and effectively learning from your data.

Jon Prial: Well, the joys of big data. To me, when I think about what you're describing and obviously I'm at the value of you supporting 1,000 customers, but to me, I would view... If I'm a CISO I'd view eSentire as part of my company, not really an outsourced agency, per se. So, what's your relationships with CISOs? How do you port and tell them what you're working on?

Dustin Hillard: Different customers, this varies for sure. And so many of our early customers were hedge funds where they may only have a hundred employees, even though they're managing hundreds of millions or billions of dollars in assets, but their security teams might be one or two people that are responsible for IT and security. So in those cases, many of them do have a CISO now as well, but they don't have an operations team. So, they need bank level like security, but they have a very small team and so we are their full operational team. So, we're reporting to them as the CISO, but pretty much all the operational aspects of their security are being handled by us. That varies as we get into some larger enterprises that have hundreds or thousands of employees. In those cases, we're much more of a partner and so we're taking on... Our SOC is able to amortize the work of all this operational triage, and so their people can focus on what's special about their business and strategic for their security program while we're constantly making sure that all the data and signals coming in are being handled appropriately. So that partnership model, as we get into the larger companies, is definitely what it looks like, where our security operations center is able to take that load of the constant onslaught away from their team and help them focus on their strategic initiatives while we are at the castle walls, fighting off everything coming at the gates.

Jon Prial: So, in the security space, we always talk about the balance of false positives versus false negatives and we touched on that a little bit, but that's very transactional. Let me take that same thought and ask you a bigger question. Is there a similar type of balance on the response? If there's a DDOS attack, maybe the first thing you have to do is shut down a system, but maybe that doesn't have to be the case. Maybe there's different ways to keep a system operational. So, how do you work that balancing act?

Dustin Hillard: Yeah, this is a big part of what we're thinking about over the next year is this kind of proactive risk assessment. So, it's great if an event comes in and we're able to contain it within minutes so that it doesn't spread to the environment, that's a pretty good outcome for a business and that's certainly what we're doing today. What we want to get to is understanding the attributes of that attack and what enabled it or caused it to happen, so that we can stop it before it even occurs. With this growing customer base, a big part of what we can start to do is see when attacks are coming in, what are the attributes of the asset? Whether that's the user or the device that that attack occurred on, what are the elements that are common across at these attacks, and then start to proactively show customers that risk on their own assets. So, a kitchy example might be that if you have Flash on a machine at this point, it maybe it hasn't been updated in a while, or it's going to be more vulnerable to attacks. So, by looking at the attributes of the host in your environment and saying," These are the six machines that are still running Flash," if you go and patch those machines, or get that software off that machine, that's going to... We've seen across our customer base of a thousand customers that machines that have this software running are much more likely to be attacked and have a true positive security incident. There's much more complex examples of that. It might be around user behavior or how much network exposure that device has, and that these elements of risk that we can learn across a large number of customers are a big part of what I'm excited about to proactively remediate and avoid the threat ever landing in the environment.

Jon Prial: So it's interesting, even getting the response right, still comes back to the protection, and you mentioned the castle walls, and we've talked about the castle walls on these prior podcasts as well. It's still figuring out those sensitivities, those pain points of where something bad can happen, but at times if somebody needs to shut down the network, will you be the ones to say," We got to shut it down."

Dustin Hillard: Yeah, and that's a big difference of what an MDR provider can do compared to what a lot of traditional MSSP type of interactions are.

Jon Prial: MSSP, managed security service provider. Now, that term goes back as far as the'90s and ISPs, and there's a number of different piece parts, but part of that broad market is what eSentire does as an MDR, and I think it's an important distinction that I want to make for our audience.

Dustin Hillard: And I think that's something we've worked very hard as, to build that trust, because you have to have that trust and that process. When you're the CISO and you've chosen to outsource a part of your security to another team, the trust and process has to be there. So we do have that with most of our customers, where they allow us to kill a network of connection, or to isolate a host, even without asking them because we've built up that level of trust that we are going to correctly identify a threat and that they see the value in us containing that threat as fast as possible without having to go through a bunch of approvals on their side. So, most of our customers are in that mode with us and that allows us to very effectively contain. We build that up operationally over time and understand maybe on their core exchange server, they're not going to allow us to shut down their email without a conversation with them, but on all their user laptops. Company by company, that varies and getting the operational details of where it's okay to contain and remediate versus where we need to collaborate with them because of the potential business impact is a big part of the customer relationship and process that we build up.

Jon Prial: I love the relationship with the CISO, I think that makes an awful lot of sense. I like the fact that you'll build that sense of trust and then that puts more control in your hands, allows you again, to be that partnership with your customers. But there are so many piece parts to this business. The market map is insane. Now, it's not totally off the whacko world, insane, like say market automation, but the cybersecurity market map is still pretty complex with different piece parts. So, when you begin to think about working with a CISO and how they make these decisions, how do they make purchase decisions? I guess the underlying piece of this question is are there platforms or point solutions? They're always both, this is IT, pendulum swings back and forth, but how do you think about communicating what you are to your CISOs? Answer this, if you don't mind across the whole spectrum of this series of podcasts all the way from inaudible all the way through to responding.

Dustin Hillard: This is something we think about a lot because it's important to us to justify our value and we sit very far down the value chain or towards the end an outcome portion. So, that's a good place for us to be, but we have to show that value. We do think about this best and breed point solution as the traditional thinking versus a kind of all in one platform that's easy to deploy and gets you good enough across all of that. I think what we're thinking right now is we'd like to say that's a false choice, that you have to do one or the other of it. So, because we're sitting at that outcome stage, the way we're thinking about it is that a good platform is open and agnostic. So, a lot of the large platforms will try and get you to converge to use all of their tools, but that a good platform and the one that's governing your outcomes, you want that to be able to ingest and take in the tools that you have is one part, because a lot of CISOs have made long- term investments. Most of these tools, you don't want to have them around for less than three years in your environment. So when you work with a customer, you don't want to come in and say," You need this, you have to throw out what you've already spent a lot of money and time tuning to your environment." So I think that's one element, is working with the tools that are in place, is an important thing.

Jon Prial: I guess for me to wrap this up, one of the big messages that I'll take away hearing this and talking about point solutions and platforms, or best of breed. Good enough in the world of cybersecurity is definitely not good enough.

Dustin Hillard: Definitely.

Jon Prial: Dustin, thank you so much for taking the time to be with us today. This is just fantastic. I really enjoyed our conversation, thank you.

Dustin Hillard: Thank you, Jon.

Jon Prial: This concludes our cybersecurity podcast series. We hope you enjoyed it and learned some things. It's really been fun, pulling it all together. I'd really like to thank all of our guests. We hope you subscribe to the podcast and head over to our website @ georgian. io. Check out all our great content, subscribe to our newsletter. Thanks for being part of all that we do for Georgian's Impact Podcast. I'm Jon Prial.