Transcript

hello hello hello everybody uh Happy Friday uh it is what is today today’s the 23rd of February wow the month’s already gone um really excited to have uh a good friend and colleague uh Robert here with us today with woohoo team Tim um I know Jesse’s out there in the cold weather these days so he’s still got his hoodie on so he’s gonna be our Packer um I’m Tim golden founder of uh both team Tim with team with Tim and Jesse and compliant scorecard um why don’t you take a second Jesse say hello and then we’ll pass the mic over to Tim and then have Robert yeah hey everyone uh happy to be here another Friday come and gone uh looked up and it was one o’clock so that’s crazy to me but I’m Jesse Miller uh with our PSA Consulting we help msps build Security Programs at scale and we help them do it profitably so with that I’m gonna kick it up to Tim schner diagonal something diagonal uh Tim schner uh in the channel focus on uh cyber security uh shifting left uh a lot of experience in intellectual property and Insider threat and uh you know they keep inviting me back every week I guess because we’re calling it Tim Tim team Tim right Tim somebody’s gonna have to somebody else gonna be besides may but uh yeah no I’m uh I’m doing some so work but also looking for new opportunities in the channel and uh excited for this week uh Robert Shafi who we’ve I think most at least the three of us have certainly seen him in action uh at conferences he’s got a huge role coming up at this RAB boom conference coming up in Vegas in two weeks I believe it is two less actually uh but Robert yeah introduce yourself talk about bit about your background and uh I’ll kick off after you intro uh with some with some questions to you okay sure I appreciate the invitation and for all the uh listeners and viewers tuning in uh no time is precious so thanks for Lending your time to us today uh my name is Robert chafi I’m the CTO and co-founder of progressive Computing we’re an MSP based in yonr New York just north of New York City if you don’t know where yonr is and today today today today is a special day because it is our 31st anniversary 31 years ago uh I started this business co-founded it with my uh High School College buddy Ugo kulie uh so very proud to I don’t know if proud proud is the right word um or scratch in my head wondering what the hell I’m doing in this business for so long but um but I love this business that’s why it’s been 31 years and um I feel like uh the the community has given much to us and I’m trying to give back in every way that I can so that’s my quick intro I’ll pass it back to you guys awesome awesome so like I said I think the three of us have been at conferences we’ we’ve all heard about your you speak about your terrible horrible no good day in 2021 I think it was July 2021 2021 uh maybe give I don’t know if there’s a short version uh maybe just a little bit of a summary for our audience on kind of your history and and uh what you usually speak about at conferences uh yeah I mean even in an hour and a half I really can’t tell the full story um so I’ll give you the hopefully two-minute version here so July 2nd 2021 it was a Friday uh just before independence day here in the United States in case uh anybody is tuning in from outside of the US and uh you know beautiful day here in New York uh we were using CA vssa rmm to manage about 2500 end points of which included about 250 servers uh looking forward to the three-day weekend you know it’s beautiful weather here anybody from the northeast or the Midwest knows what Winters are like they kind of suck you know the the ground is brown and the sky is gray all the time and we’re just so looking forward to those beautiful summer days the forcast was amazing and you know we just um you know we experienced that worse nightmare scenario uh around 1230 we had realized that 100% of those end points that we manage were encrypted with reval sod no kibi uh threat actors had discovered a um zero day exploit to walk right past um all of our security didn’t need administrator passwords walked right past MFA uh logged on uploaded um reval Stone KB um and pushed that out to all the end points that we manage so that was about 80 clients across 200 physical locations in four time zones and like I said it was um out of the 2500 mend points about 250 servers and the balance of which were were end user desktops and laptops so that’s a quick version of it and we can go deeper if you want yeah I think that’s great um it’s not great it’s not great it’s a great two minutes summary of uh like I said that terrible horrible no good day um but uh you know Jesse Tim I have some experience in breaches nothing like everything’s encrypted end of the world right um on the corporate side factions of the business may have problems or incidents um I don’t want to say breach right that’s a bad word Brad gross told me that yeah and um but you know it’s different right when it’s personal and it’s not like a corporation where you know people are just doing their jobs and they have they have roles and to you know in an incident response plan but Jesse Tim maybe you guys can talk about your history um right AB boom yeah um the the first one we had dealt with was a pretty significant um client incident and it was I’d say a little bit different than than Robert story because uh it was we had prepared and it wasn’t us it was a client so uh you know starting starting a a security focused MSP program you know pretty early in the industry in like 2014 2015 we were going to our clients giving risk assessments um this happened to be our largest client at the time uh they said no we no thank you we don’t want to need your Security Services we can do it ourself that’s what we had done has gone and delivered risk assessments to every client that year well it’s about six months later they were completely encrypted um with Ki ransomware right and Ki was very prolific actor there for a while um and so I mean it did throw us into it did throw our it did throw the organization into chaos because we it was our largest client we’re trying to get them resolved but it was a different kind of thing it was more like the big game right like this is it’s go time here’s where the rubber meets the road you know we made and we I mean we did an excellent job I think for for the first one you know put our put our inate response plan into play executed on it and uh eventually got the customer back up and running they didn’t pay they ended up not paying the ransomware or paying the ransom they didn’t have backups so we were built we were we were pitching them a cloud migration at the time so we said they said okay we’re signing off on the cloud migration here you go so we just built the cloud environment net new and migrated them over and um you know got him back up and running was extremely painful and I’ll tell you what you you think you know but you learned so much in the throws of that incident um that you know it really in influences the way you look at an incident kind of I think colors the way the things you think are important are not as important during an incident so I have some other questions but I want to hear from you Tim your story about an incident so is this where I share the one where I uh encrypted and Ransom we myself sure so uh TLD drr uh uh I was uh triing a new Tool uh one of the features of that tool around data Discovery and and monitoring one of the features was like you know protect the data not knowing enough about it didn’t really do my homework on as well as I should have way back in the day I was like of course I want to protect my data check find no problem you know we evaluated the tool for a couple of months I decided I didn’t want to use it so I went to their portal and did the uninstall through their portal everything’s all well and good all of a sudden you know one file server in like you know California hey I can’t get to this file another file server a couple days later in Florida hey I can’t open this file and then we kept seeing this repeating pattern like random files not being able to get opened okay kind of sounds like a ransom but I’m not really sure random things here and there and so I started like opening up documents looking through them there was you know a signature across the top of every single file and it was the exact same signature but it was just random files across you know the entire organization well come to find out I got you know you know our great friend Wes Spencer I got Kyle over at Huntress I brought in like the people I know to bring in like what is this Randomness long story short was when I uninstalled the the software it didn’t decrypt all the files before it removed the agent so the files were there and we could open some of them but not all of them because not all the files got de encrypted and the tool I’m not going to call it out or whatever but I I essentially ransomware like seven locations and hundreds and hundreds of random files across multiple different file servers that I then had to go find and thank God we had backups but it took me and people way smarter than me like the people from Huntress and and Wes who was at perch at the time like we were all baffled and had I not looked at add and remove programs and saw like there was a signature still left there and I was like oh my God I wonder if it was that and it was so I ran some weird myself and that was bad pretty embarrassing story but I’ll own it because I didn’t know any better at the time good job there Tim good job yeah it’s a little harder to get mad at yourself right than some nefarious Young little boy from Eastern Europe uh you know in in Robert’s in Robert’s case or I guess there was another actor as well right who was still at large there are multiple actors one is still at large at least one other is still at large yeah yeah it’s interesting that we’re talking today and and and recounting this because you know I was talking about as we learn from these incidents and I think if you’ve been through one you have invaluable experience right but it’s interesting we’re talking today in the wake of this screen connect zero day that’s been out there right and that has been causing it’s very similar to the ca problem that you experienced right Robert maybe not an exploit exactly but the type the level and ease of bypass is pretty pretty straightforward right well I think we’re still early in it right we think that we’re already past those early stages I think we’re still in those early stages because I I’ve been thinking about all of the companies that have been victimized by this but we just don’t know it yet well right right and so I can see this as being something that uh is uh you know we there may be some residual of this where we hear in the upcoming weeks or months or maybe even a year later that you know someone kind of raises their hand and say yeah I was one of those guys you know we had a bunch of data stolen exfiltrated or you know they used it to push ransomware or whatever they did right so I’m I’m still holding my breath waiting for the shoe to drop on that yeah yeah and I I think though like what I think we’re all pretty aligned on at least well maybe I’d like to get your feedback on it but the fact that connectwise did take some proactive measures and shut down the licensing to prevent prevent further exploit had that uh had that incident occurred with CA or not occurred with CA and we didn’t have precedent for this maybe they wouldn’t have done that but they’ve seen and learned right so I guess in your experience what have you learned the most and what are you applying the most as you help educate the industry on incident response that maybe they’re not thinking about msps that are not thinking about it haven’t gone through it oh boy there’s like so much to unpack in that to give one piece of advice is really hard I mean it’s the thing if I had to really say one thing it’s that we all already subscribe to and know and that cyber security is a journey it’s not a destination you’re not showing up to a conference and walking away with all the secret magic you know it doesn’t work that way uh I have a philosophy of and it’s you know it’s a little bit sort of taken from a page from Atomic habits that sort of 1% Improvement every day I keep telling my team yeah if we could do one thing a little better today right and just keep stacking those every day uh we’ll just continue to increase our cyber secur posture our ability to uh not only defend against attacks but to be resilient in the event uh that one will happen right it’s not a matter the if it’s the when so um you know I wish there was something that I can point to to say that hey if you do this that will you know really that’ll take you from 50 to 70 or 50 to 90% secure yeah no this is you know it’s a marathon it’s not a Sprint it’s certainly not a one day race me I can’t believe it’s still 2024 and we’re still having to press the issue on two- Factor multiactor like it it just bothers me and and right and how long have we been you know how long has that been a very prevalent part of the conversation I’m just going to throw a number out I don’t know if I’m right or wrong three years like you know that’s long time and we’re still having this conversation yeah it’s yeah I was gonna say go ahead Jesse no I was just gonna say I think you’re right and I think you know uh the fact of I wanted to hear from you about and I just want to get this resource out there because it’s there for msps is what you’re doing with CompTIA in terms of the the 911 MSP rescue um can you talk about that a little bit um sure so I’ll try to make it brief because I’ve got a lot of stories um so it was so remember the attack took place July 2nd 2021 um such tremendous help from the community both our peers as well as solution Partners uh the axian the huntress the conect wises the Pates of the world people showed up to help us in Mass uh in lots of different ways in fact I even had a stranger show up at our office from Iowa uh Jim Allen if you know uh Jim from Aces Iowa um he uh you know he came here he didn’t even know us he heard through a friend but anyway so there I was it was in August uh you know still heavily licking our wounds and I was um I was reflecting on The Good Fortune we had of being so well vested uh in the industry and had built so many good Community connections which you know came to our Aid not just that day but for the month right not you know it was significant investment on their part as well and I thought you know when when news broke here internally also that franticness that I felt that sort of helplessness yeah the without the head sort of syndrome and I wondered is there a way that msps can hit a giant red Mercy I need help button or you know break glass in case of emergency that sort of feeling and then I thought well of course a website so I went and I looked up MSP 911.org it was available I registered it and then I said what the hell am I going to do with this thing right I I I have a vision in my head that it would be a for msps by msps completely volunteer Le and run I’m not you know I had monetization of this idea was the last thing on my mind uh because it was uh the only way that I as a victim could effectively fight back against our attackers right I don’t have the technical chops to go up against cyber criminals one alone a gang of them right I I don’t I I’m not that smart or knowledgeable when it comes to that sort of stuff uh I’m I don’t also don’t have the resources but what I considered or at least I would would like to think that I have this much of is community influence and and again making a long story short I ended up meeting MJ Shore over at CompTIA later in that year where he was speaking during a presentation sort of about this same idea and I approached him long story short he put resources behind it and we rebranded it as the CompTIA Emergency Response Team so if you go to MSP 911.org and I know that you had it up there before as I was speaking it will redirect to a a landing page where if you’re a victim you fill out a form hit submit and a bunch of us a bunch of volunteers are going to get a um a call myself cell phone will ring I’ll get text messages alert apps uh we’ve got a completely back-ended automated system to manage cases uh and to provide um um here’s a word that you may not be thinking about in your cyber security stack Comfort right a friendly voice coaching mentoring right just some I wish I had somebody that day uh or I know that there are other msps out there that may go through an event and feel like they’re alone and I did not ever want anyone to go through an event like us and feel like they couldn’t get the help of at least at minimum somebody to say I’ve been there I’ve gone through what you’ve gone through I know know what you’re feeling here are some things to consider so there’s more to it than that in terms of what we can provide uh but that’s the basis of the idea two two things Robert um when someone submits is it Anonymous like do you keep it confidential or like is there people worried about that like am I how how can it be anonymous like we need to know who you are if you want to have a conversation right but I mean the the volunteers that are helping right like you you know you’ll you’ll keep this under wraps right so what happens is it gets assigned to everon call and then we pull in whatever specific resources are needed and there’s a whole NDA process that kicks off from there okay we sign n just to be part of the organization but then uh if I want to know about a specific case I have to be a part of that NDA so I’m actually working a case right now it’s kind of it’s you know an attack took place uh over the Christmas New Year Break um somebody’s you know uh business was completely Ransom word very similar to what happened with us uh and I’ve been providing just some I’ve I’ve been providing a shoulder to cry on is essentially it because you know they were all by the time they contacted us they were kind of already through the worst initial part of it yeah but now it was just like come on keep your spirits up I know that you’re feeling whatever you’re feeling I know that this bad stuff is happening but you got to push through you gotta energize your team you gotta reach out to those customers that are pissed off and I don’t know send them a cake or something show up with donuts something so that you don’t lose your business and your livelihood that’s a really great it’s a really great spot into our next question about you know learning from those emotional scars right and I I love the fact that you use the word Comfort right because and Jesse I’m sorry I didn’t mean to interrupt but I you know kind of talking a little bit around like Lessons Learned like we always forget or we tend to forget I hate to use absolutes we tend to forget about like there are real people behind this you know there are humans on the other end that forget to eat forget to drink forget to like and and like that emotional piece behind it is just I can’t of fathom it I mean I know when I did it myself and I was a wreck for weeks trying to figure this out and yeah you maybe talk a little bit about that uh well for sure we had people here who were working a hour days uh constantly I remember it was the Second Sunday in our event so I think we were probably I want to say maybe 10 days into the whole thing where we just ordered the entire team to stop and I had people resisting like no I want to keep going I’m like no you need to go home and rest so most people um um complied but there were just some who couldn’t bring themselves to doing it so you know I think Tim you’re hitting it on the head here we we think about technology we think about the MSP business we think about response and recovery and Frameworks and all and tool sets and we think about it just being bits and bites and Technology right it’s not there are real people here whose lives can be affected in a very negative way yeah not only as they’re going through this but also in the aftermath and we cannot lose sight of uh of the human impact the psychological impact uh if I may share for a moment that I don’t have a diagnosis of PTSD but I’m telling you from what I have read it sure in hell feels like I have it I might be joking and smiling and you guys might see me presenting on stage and every once in a while you’ll see that I’ll crack it’s because it catches up to me sometimes yeah the feeling does not go away um so for those that would lob and I’m not s suggesting that this is prevalent these days but in the past for those that might lob criticisms towards another MSP or a company how could they let this happen what would they assleep at the wheel like you know what what gross incompetence hold on a second you know why don’t you dial that back and kind of figure out like there’s probably way more to the story than you know that’s number one and number two like instead of shooting your mouth off why don’t you pick up the phone and ask how you can help like that might be actually useful yeah exactly I I thought I thought I saw it multiple times every time he speaks I think the biggest lesson um is really the impact of how quickly your multi-million dollar 31y old MSP to today went from that multi-million dollar value to potentially negative as you said right I I can’t think of another industry where that is possible in that short a time span without you know doing like the CEO just doing something crazy gross grossly negligent right like even then like there’s always there’s always insurance and protection and the business continues like um other Industries like like public publicly traded companies right like our they’re they’re basically set up to built to manage risk to like basically not ever lose that kind of value and it’s it is like a little bit of the Wild Wild West like as you said the no safy net now do we we’re not we’re not public companies that have you know even the US government uh will step in and help with resources and I don’t mean to it seem like the government won’t help small businesses um it’s just that they’re not even aware of how to help a small business um so you know you touched on a very important topic here Tim that I’ve been trying to figure out ways to unpack and you know again Sony gets hit uh Colonial pipeline gets hit uh uh Target gets hit Home Depot you know Capital One we can go on and on and on Dro bucket all look what happened at AT&T this week even though apparently it was not a Cyber attack but everybody like sisa the you the White House like everybody was like hey do we need to help right doesn’t happen to the average MSP no no and it’s yeah it’s just it’s about like I said and every day like you you got to be a little uneasy right like we’re dealing with sophisticated systems that have lots of code and we have adversaries in a in a grow like digitalization grows what like 20% every year like of your business is going online and it’s like worldwide access you’ve got attackers in every nation and it’s it’s a little scary I mean and as you said like no one should ever be like oh that guy got hit like what was he doing wrong like the the reaction should be oh man another one got hit like you know it’s just a lot of it’s bad luck to some degree um certainly in your case I mean it was like a had nothing to do with you know it was basically supply chain uh for you I mean I’m not I’m not trying to be defensive here but uh just mainly stating facts we were uh we were on the very latest security patch the only release that we were behind on was a feature upgrade uh but the last security patch that was released by CA was I believe it was at the end of May of 2021 so this is early July I mean we were fully patched we were implementing MFA we were following the prescription of what you know they recommended you should do to protect and and again and and again I don’t mean to make it about me or that specific instance but I think you you land on um the important fact here is that everything is very interconnected here and supply chain attacks are only going to get worse and worse and worse uh we were just talk I forget who I was talking about this with the other day about how um uh threat actors are compromising um and this is nothing new but uh threat actors are compromising uh companies but instead of attacking them what they’re doing is they’re attacking who they’re connected to by impersonating right um so now you think you’re dealing with the CEO of that company that you always deal with and yet you’re not so it’s it’s there’s a lot of lot of things to consider here yeah yeah I think you know that’s a that’s a huge piece and I’ve people don’t always agree with me on this but I’ve always been a fan having worked in SEC regulated Industries for a long time I think the SEC actually does have some pretty good pragmatic approaches to ensuring that that they’re their regulated companies adhere to cyber security standards that actually make a difference right and so to your point that’s why you see in the new cyber security rules this year they require you to disclose on your 10K statement what you’re doing for third party risk management like that’s now a requirement yeah if you’re a publicly traded company you have to say I I’m either doing it or I’m not doing it because that’s material right yeah so and it’s funny that that doesn’t apply to msps yeah well we’re the on ke to the kingdom right yeah if you want to open up a giant can of worms not yet right until government decides to regulate us so not to throw a hand grenade on the table here but I think we need to police our own before we get policed well I I think you’re right oh go ahead that brings us back I was gonna say that brings us back to CompTIA right in the Trustmark program right having having work with Chris Johnson and the team over there to build out a framework to help msps become an SRO a self-regulated organization you know there’s plenty of models look at like law firms look at like the the board right for legal right there’s plenty of organizations and you know compa is on the Forefront of that right CompTIA is you know built this framework built this trust Mark piece so that we can essentially self regulate before we get regulated from people that probably have no business regulating us because they don’t know know who we are and what we do right um at the end of the day it all it always boils down to like you know the the MSP 911 like us working together right yes we’re competitors as vendors across platforms across msps Even in our own Hometown but you know who says it all the time better together or Rising tides and ships right being able to work together especially during an incident like this recent one with screen connect so we have a bunch of people listening right now uh if you could maybe drop a comment in yes no maybe around have you been affected by screen connect or what kinds of things have been coming up in your conversations with your clients about the screen connect right uh we’d love to hear from the audience as as we’re kind of you know Meandering through the rest of this conversation Robert I was going to ask as well have you gotten a 911 you know org uh anything on that yet or too early uh oh about this no yeah we haven’t we we anticipated uh you know my uh Jason slagel who’s on the leadership committee uh you know he was texting me um I don’t mean to laugh about it but he was really concerned that we were going to get flooded with calls uh thankfully we have not received any yet about it however like I alluded to earlier I it may be too early yeah you know if I if I haven’t patched on day one or day three it’s unlikely I I mean this is just my speculation and theory that if I didn’t wake up to that news by then the likelihood is that I’m never GNA patch because my head’s in the sand yeah yeah it’s and we all it’s it goes back to we’re all here on you know in social media and kind of in this MSP sphere commenting and getting these notifications and we’re plugged in for lack of a better word there’s plenty of nms out there that aren’t as plugged in as some of the people that are staying in tune with the industry right and they could be ones that don’t even know about it or they hear about it a week later like oh did you hear about that screen connecting yeah we should probably patch that right so there are ones out there like that I think and remember if you’ve patched even though connectwise has taken some very bold action which by the way I mean kudos to them for for doing exactly what they’ve doing what what they’ve done I I don’t you know I I I just I think it’s admirable that we’re you know that finally somebody has a much more aggressive stance with this you know be damned with the ramifications this is what’s right yeah so again kudos to them but uh if I wake up a week later and go oh crap like I got a patch I might have already been compromised initial access Brokers might have already sold off uh access to my systems and that patch doesn’t mean anything anymore yeah article yeah that article came out with Kyle hansan h and he’s like I’m really worried about this there’s a lot there’s a lot of open end points that haven’t been patched and this was like two days ago three days ago and it was like there’s a lot of people that don’t know connectwise right and certainly in foreign attacking regions and now they’re like oh did you read this article on TechCrunch front page about this connectwise thing I’m like let’s go check it out right so it’s kind of like this balance of you have to disclose and help but at the same time like you’re you’re saying you know you’re kind of exposing to the world that there might be an opportunity here I think the cat was out of the bag earlier than that though I mean there were a couple of proof of Concepts that were released and so you know and I got to imagine if you’re in that criminal underworld uh that news you know news Cycles there just as it Cycles with us right right you know it’s interesting because it’s screen connect and because automate uh those types of systems like I said back when we were doing this in 201 14 and 2015 building our practice it was kind of this Awakening for our MSP to be like holy crap we’re really vulnerable in a lot of ways and so we poured so many manh hours into building our own custom detections just for our automate database because there was nothing around at the time I mean now we have that but it’s still kind of dodgy to to be able to get really good detections in automate right so um I think that like you said we may see some Fallout from this because people think on Pat now I’m fine well if you’re not doing threat hunting and you’re not staying plugged into those ioc’s and then plugging in those IPS and ic’s back into threat hunts throughout your Fleet and your environment you’re not rinsing and repeating that you’re you might still be in danger and you don’t know it so I just would caution people that are watching this to please make sure this take this as a chance to up your threat intelligence game up your threat investigation game um do some threat hunting in your environment and build some queries and start running that stuff against your Fleet on a consistent basis and updating it and enriching it with the data that’s coming out as we learn more about this every day right yeah so um don’t quote me on the number but I think it was some somewhere around 6,000 uh end points that still haven’t been patched as of a day or so ago right so that’s you know that’s that’s a potential for you know 6,000 like we’re we’re talking about it we’re like cat’s out of the bag so okay um but it could be possible of lateral movement right it could be like yeah some so Matt Lang just popped that over there in chat like thanks thanks Tim for pulling that up right there are you know still lots of machines that are that are unpatched and so my recommendation is we keep spreading the word right we keep helping those msps that probably aren’t even aware that these things are are still vulnerable right and you know uh Robert you were saying like the the MSP 911 the CompTIA uh Emergency Response Team geared up in preparation for this but you haven’t started to see that but we might not know the extent of lateral movement yet right we might not know what that would look like it is quite possible that your phone might start ringing tomorrow or tonight right that was my point Tim was that I I I don’t think the worst of this has come out yet I mean it feels like oh the news broke everybody’s talking about it everybody you know everybody was fearful and nothing happened so ah all right whatever let’s move on and I think that we’re still actually I’m agreeing with Kyle from Huntress that I think the worst of this is yet to come uh I’m gonna I’m gonna way I really really hope I’m wrong but I mean share this real quick yeah go let me just sh share this real quick because this is scary 9,066 machines show up on showen with screen now whether they’re all vulnerable or not I don’t know but it’s literally there’s 9,000 machines with screens connect like that’s a big attack surface especially as an MSP because I might have 20 clients so let’s say the average MSP has 10 clients well that’s 9,000 times 10 right the lateral movements across and so on like that’s scary if we think about it right let’s say that half of 1% right this is all math right half of 1% of that 9,000 is actually vulnerable and being attacked that’s a pretty low percentage but let’s go with it that’s 45 did I do the math right PR well the I guess the the minor and Mathematics finally paid off that’s 45 times how many on average let’s say a low average of 25 so I mean you’re talking about over a thousand uh customers right that could be and then how many endpoints and behind those that could 10 that could be 20,000 yeah right these are admin privileges basically right like I mean you’re theth is startling yeah yeah with that said I I wanted it’s hoping it’s a good segue to get into what I wanted to talk about today is just some pragmatic advice for getting an incident response plan built that’s going to help make you more resilient right and so I know Robert you’re probably Consulting and talking with msps and helping them build those out and I you know speaking to the the the person or the the effect the personal effect of this stuff I think people drastically underestimate how how confident they’re going to be in the face of this kind of Devastation because I know I was I underestimated it right and so uh just talk about that a little bit and what are you seeing work well for msps to get the ball rolling on a good incident response plan so you’ve gota I mean I think it just starts with the basic education I mean if you’re if you don’t even know if you don’t know how to spell IRP like you know start with just start attending some sessions uh and there’s there’s plenty of smart people out there doing uh some really cool stuff uh PX eight Matt Lee does a lot of this stuff Chris Lair does a lot of this stuff uh just sit through some table topex or Huntress actually has gamified it a bit uh Ethan tan credy um Black Hills yeah Black Hills yes of course how can I forget Black Hills uh back doors and breaches just start plugging yourself into that stuff um I I mean I’m not here to promote any specific company or tool or whatever but you know there’s good tools out there I just feel like you know because exence was so uh gracious to lend their tool to us at the emergency response team we use an incident response platform in order to be able to do the intakes and uh uh uh manage an incident online because if you’ve got that paper based system right if your incident response plan is word um what happens to that document when you’re ransomware right right is that IRP separated from the rest of the systems yeah so again use use something make sure that you have your plan elsewhere I mean a cloud-based service would probably be a smart idea but you can’t just subscribe to that and think that you’re done you’ve got to run the exercises now I am not a military man um I am very uh patriotic uh but I have never served in the military that’s why I always look to my friends who have and I’ve asked them questions especially those that have actually seen action for real real action I’ve said you know did basic training help you and their answer is no right but but it also invaluable for them to at least just get some of the basics uh done so that’s why tabletop exercises and R and role playing I mean you know it’s by the way I’ve mentioned this before in the past and I’m always surprised when I ask how many D and D players out there or tolken fans out there and like mine is the only hand that goes up like what we it professionals isn’t the requirement the basic requirement to being an MSP is that you have you know you’ve played a lot of dungeons dragons um role playing that’s the point role playing you need to roleplay this stuff out to to um to to practice to understand like what are the vulnerabilities and how would I behave in certain scenarios or if you’re going to proverbially tie my hands behind my back how do I act right yeah you just said that on the military and I think you said how do you gain comfort for like the worst day possible right so are you gonna feel more comfortable being an 18-year-old kid that goes to you know Marine Corps basic training or are you gonna feel a lot more comfortable going to war and you’re a Navy SEAL at a Coronado right like drills drills drills drill right like how many rounds those guys fire and how you know how many times they practice right so I think the the more you practice and the more experience you get you’re going to feel a lot more comfortable for that worst day you know with facing that adversary so yeah I think this is like where where High School football players asked to somehow compete in the Super Bowl during game you know right an incident and we’re just not prepared like we have we understand the basics and the concepts but we’re just not ready exactly je you were going to say something you know it’s interesting because yeah no it’s I think this is all great advice and I think it leads into that you know I think and I know for myself when I procrastinate it’s because I feel INE INE equipped or unequipped to do something right so when I procrastinate it’s because I don’t know how to do it really well and I want to I’m someone who’s a perfectionist and I want to be really good when I do something so that’s me that’s me personally but I would venture to say that probably a lot of people in this industry are similar right so people look at this and say well I don’t know exactly how to do it and I need to do some more research and really need to figure out my Approach for a tabletop and all that just sit down with your leadership team and start talking it through like okay if we got Ransom word what would we do first who would we have to notify first would we call our ins Insurance have we talked to Legal about what we should and shouldn’t say right who are the people that we should notify in the order that we should notify them I think just getting a list of that do we have customers that we have to notify do we have vendors that we have to notify just building your list of entities that you have to contact is going to open up a real Vista of where you need to start going so that’s something you could sit down and simply do day one to get the process started and get the juices flowing that’s why we you know we put out a Huda call checklist right at rmsp was something that I just standardized when I brought in a new customer I literally went through the checklist like who’s the legal who’s the CFO who has buying power who’s this and yeah put together a who to call checklist and then don’t store it in that ecosystem yeah it’s a great Point how about an export of your customer list yeah and Def not only not only did we Define like who to call not store it within any of the standard ecosystems either ours or our clients but some third party and backup communication plans this is when Shadow it is your friend yeah I don’t know I wonder how many msps actually ask their customer like who’s your legal team can I get the contact info who’s your insurance carrier can I get the contact info yeah right that’s a hard conversation to ask but yeah how about keeping was compromised that day so we couldn’t get to our customer list like we had no well I mean we had ways to contact them but only through memory or or digging through email yeah I heard I heard an instant response uh Insurance call uh webinar recently and they’re talking about paying the ransom and the and the attacker enters the zoom call it was like you better pay so I mean you think about how like everything is compromised like how far how far can it go so like how do you air Gap that shadow it or whatever you want to call it so yeah you have to think that through autoband Communications is a huge problem that you should solve for ahead of time you know that is one that at least I’ve made mistakes and instant response don’t get me wrong I’m not some perfect done it all right I’ve done it all wrong so I know how to do it better I guess that’s probably what I would say about myself but in terms of outut ofand communications there’s some really simple steps you can take is get your leadership team and your incident response team on signal chats ahead of time right just do some or you know whatever you want to use whatever your encrypted Chann autoband channel is and just get that in place so that you can start uh responding with those people you know I think things like getting an IR retainer on in place with with a a reputable firm that’s outside of your insurance right BEC and then working to get somebody that you like on the insurance list that you can use but you want to be able to bring them in for example this would be a perfect example is the screen is a screen connect issue we don’t know if we’re compromised we think we are well insurance is not going to pay for that but I can go tap into my IR retainer I have with u Mandy and her secure works or whoever else is out there right Artic wolf whatever whatever it may be and say hey can you guys come do a sanity check on my environment yeah it’s going to cost me five 10 grand right but at the end of the day I can be really comfortable that I don’t have to worry anymore right so and I I can you know stay vigilant but that’s money well spent my in my opinion so Tim I wanted to ask you and you don’t usually you know we’re pretty vendor agnostic here but I know you’ve got some great resources on compliance scorecard is it really just an incident response plan is it kind of like an escalation tree like what are some of the things that you know you’re yeah again we don’t like to talk about ourselves but since you brought it up I I’ll be brief um you know we worked with fifth wall right so fifth Wall Solutions right cyber liability insurance company we all know who fifth wall West and and and you know Dustin and Reed and that we all know who they are and so I took my IRP that I had been using in our federal environment for years and I I sat with them and I was like what do you see here what’s broken you know what what can we how can we improve this so we work together collaboratively to put together you know a solid um incident response Plan and there’s one big section right here in the middle that has great big bold red text and says stop here and call your insurance company period so that yeah that’s available but you know the the the who to who to call checklist is available too I think that is that is probably the first place if I don’t if I’m an MSP and I don’t have an IRP an incident response I’m afraid I don’t know like ah it’s scary all the things that we’ve been talking about the last 45 minutes probably the first thing you should do it is February it is the beginning of the year you’re about to start your quarterly business reviews if you haven’t already grab the Huda call checklist start building that out and store it someplace separately and as Jesse said have a backup communication plan a signal group a WhatsApp a you know carrier pigeon have a separate section or a separate line of communication so I mean I could go on and on but thanks Tim uh you know having IRP too afraid to build out an IRP start writing down you know pen and paper if you have to who’s their insurance carrier who is their point of contact and it doesn’t even have to be around an incident right around a breach I hate to use that word server goes down and you need to go buy something do you have the CFO and their their credit card right or you picking up that cost right even just like that right if you think about it that way right C call checklist would be a great place to start as building this out it’s great advice yeah so Robert and like top three things that you wish you had that that day customer list I mean first yeah an incident response plan I wish uh that would have contained things like um an up-to-date exported customer list because our Communications went to zero because connect wives was encrypted right you know we ate our own dog food so to speak actually nobody eats dog food well maybe some people eat dog food but dog eat dog food we we we ate our own tacos I should say and put you know our C agents were managing our own systems and you know we were encrypted um so uh for sure getting uh a incident response plan in place uh is number one uh you know I I’m really big on CIS controls I don’t want to make it s like you should be following that thing to the letter um because I think there’s various stages of maturity that you can achieve with CIS controls but at least start with just understanding what’s in it and do a gap analysis on what you have like what your practices are versus what their uh prescription is I’m not here to say that CIS controls is the be all end all but it’s a good place to start and I wish wish we had done that prior to the event so that’s at least for you Tim yeah yeah compliance doesn’t equal security but CIS control is pretty darn good and there’s a really smart people that put it together so um Tim what do you think last uh wrap it up here final thoughts yeah and you know and I gotta bounce because I’m realizing I’m like I have another Zoom with five people waiting for me over there so uh that’s what happens when you triple book um so my key takeaway for us and the MSP Community you’ve heard me say it a dozen times start someplace do something I’m gonna pivot that a little bit and I’m gonna say write down who the key contacts are right even if you’re too afraid to think about an IRP today at least be able to determine who your key contacts are at your customer and your own MSP and as Robert said do an export one a month of your customer list and keep it outside of your existing systems in some secure place to do that right because when the crap hits the fan and your data gets lost stolen missing and griped or whatever you’re gonna want to know who to call and know how to call them and have some communication method to do that so who to call checklist probably a Client List and some kind of you know outof band commun communication methods just in case thank you Jesse for that one sorry I stole you thunder all right Jesse you want to go yeah yeah I you know I think to me it’s been and this is top of mind for me because of a few organizations I’m working with right now but I think having a a an out of band we’ll talk out of band again having an outof band incident response retainer with a with a reputable firm to just be able to do some sanity checks on your environment in this type of scenario I think for service providers I wouldn’t recommend that at all my customers but I think for service providers it’s it’s something it’s a good investment to make in a way that you can show that you’re really eating your own tacos Robert um to to your clients right um so I think that’s a good takeaway and I guess one more I would say is just um you know write down your three probably most probable things that could happen to your business from a Cyber attack and then just start gaming those out and make it a make it a goal an okay are this year to get through that by 2025 I think that’s an achievable goal that most people can stomach I think you do those two things and you’re going to be well far ahead of the attackers should an incident happen right yeah so I’ll go um Roberts like I said his track at conferences is really impactful um you know and you’re a great speaker obviously you’re you’re MCN in two weeks here so I think you know there’s a little bit of fate in that the fact that you are able to basically convey the pain and how you know how what happened and how you know a lot of things are not in your control and how do you you know how do you train to be that Navy Seal for next time right so hopefully there is never is a next time but at some degree like there’s going to be incidences there’s going to be software build materials like you we really just can’t dig into the code that deep right like there’s always going to be you know supply chain attacks here so um like I said if you haven’t heard Robert speak it’s really impactful like and the fact that you can basically go from a highly successful MSP that you took you know decades to build and you know something like this can happen so um there’s no one there to back us up there’s no we’re not a bank the FDIC is not going to bail us out like in 2008 and and insurance which I’ve never heard this story but I actually want to dig in with Robert on that as well on how that went down but you know they’re not there they’re not going to cover they’re not going to cover those kinds of losses right they’re not going to replace a you know multi-million dollar business overnight um if if they’re going to replace it at all so um it’s you’re on your own and uh the the more resilient you can be the more you can prepare the better so be careful about what’s in your insurance policy make sure you understand and read that clearly um because anyone who’s been in an auto accident which is probably a more common thing that we experience even like little fender benders you don’t realize what’s in your policy until you actually need to activate it yeah and I’ve seen these carve outs right fifth wall talks all the time on how you know even if you had an invoicing attack and like the the max limitations are 250k or you know if you really read the fine print you dig in and then you also look at all the things you repped and warrantied saying you did this you did that you know um it’s not there right like cyber Insurance they’re not there to lose money and people are losing a lot of money here so um you know it’s not it’s it’s no surprise final Robert yeah go sorry I’m sorry final thoughts yeah we’re just um I don’t have any final thoughts my thoughts go on forever so that’s final thought list true if you’re gonna be I can’t shut up um if you’re gonna be at right of boom um uh please um you know I’m going to be super busy but uh I would love to be able to meet anybody who’s listening if we have not met I’m happy to continue the conversation uh privately publicly what whatever it needs to whatever I need to do to just keep making sure this community keeps moving in the right direction uh and forward awesome awesome Tim wrap it up I think I don’t I’ll check what we got next week but I am hold on one second I’m getting us over there were’re ending four minutes early unless somebody has something else to say if not I got my zoom all right go ahead Jessie I thought it was really interesting what you said about read your policy and know what’s in it I got a farm story about that recently we had a big hail storm in Minnesota this winter and uh golf ball siiz hail just shred not shredded but just destroyed the exterior of the barn and a shed and so when I went to file my filing claim guess what I found out look my farm policy it cosmetic damage so because there was no holes in the side of the barn I get nothing so I thought there were holes I know for there were I could see at least shotgun holes in there at some point they uh that’s that’s that’s the pigeon’s fault let’s put it that way okay okay okay all right so uh thank you everybody uh Robert thank you so much for coming on and sharing your story um just remember at the end of the day there’s people behind all of these things There’s real humans with real emotion and real psyche that we need to make sure that we’re taking into consideration yes we have to take the boxes do the ones do the zeros but just remember buy them a pizza get them a Coffee Kick them out of the room give them that reprieve and maybe bring in somebody that’s not in your business to talk to them right there are real humans behind all of these horrible stories don’t forget that and give your team the support that they need thanks everybody Tim golden uh one of the founders here and of um Team Tim and we will see you all next week oh I suppose I should do the like the thing right that’ll pop up right now ready we’re doing identity go ahead identity access for msps right multi-tenant identity access so I mean this is a complicated one that’s really really important so excited for that all right everybody now do the like and the Subscribe and the share and the tell your friends subscribe now