Tales in Sysmon

Sysmon is an excellent free tool from Microsoft care of the talented Mark Russinovich. Essentially it takes an XML file that configures logging for lots of different events on windows, from network traffic through to the more important process monitoring. Essentially it will fingerprint every process, and map out the parent processes. It's great for incident response, if you don't have it, at least on key systems you're missing out.

This post is likely to be updated as I find issues.
I built up a process to do it at work, and on most systems it worked fine. On some systems it didn't. I had very little logs or events to troubleshoot it with. I was getting on the host a blank Sysmon log. The only linking factor was they were all windows servers.

Nothing Logged in Sysmon logs.

Doing some standard troubleshooting, uninstall, reboot, reinstall, reboot. No go.
Then I noticed the directory we'd deployed it to was were the sysmon service was pointing. Seems our install script set it to run from there.
I uninstalled and moved the exe to a different directory and it worked. Sysmon extracts a .sys driver file, that was unfortunately being overwritten by our deployment software, and not loading the driver, hence no logs. Hope this helps someone else.

Don't log your logger 

You want to reduce your noise into your SIEM, so make sure to not log processes started by your logger.

Curioser and Curioser

So I haven't had experience with home based anti-virus in a number of years. Yes all our computers at home run av, but that will be whatever free one I can find to mitigate the risks of malware... av really doesn't do much now-a-days with malware outnumbering legitimate software.

I had a friend tag me on Facebook for some help. It was an interesting problem, which of course piqued my interest. Some sites were not working, specifically two airlines, from her PC. The error looked very much like she was being actively blocked. She assured me it worked on her phone and her phone was connected to the same network. I was on my way home from work, so all of the troubleshooting was done via FB messenger.

The error as you can see below looks interesting, add to her saying that both SingaporeAir and Emirates didn't work, but Qantas and Etihad did. I was instantly thinking Web-application firewall (WAF). WAF is interesting as most IT people don't even know what these are, so how is a smart but non-it person supposed to approach this. Also this error page is terrible, better off giving the user some guidance on how they can rectify the situation...
This looked like her IP or Browser fingerprint (I've seen a chrome plugin cause a WAF to block a user) had been blacklisted somewhere and then replicated out through the threat feeds that these WAF's get.

I noticed in the unedited screenshot that she sent me that there were other browsers. Going with my theory that the above is a WAF error, and it could be something as simple as a plugin causing the block. I asked her to try a different browser. Same error.
Now to verify the phone and computer are on the same network. Do a google for what is my IP address I said. She told me they were different. Ah hah... Give me those addresses, I say. The one she was getting on her computer looked to be owned by avast (remember I was on my way home so I did this all via mobile);

It is also was blacklisted...

I verified with my friend that she used AVG, but a quick search shows AVG VPN uses the same back-end as Avast. I explained to her why you need a VPN; for privacy, but that most of the time it is not needed, eg it is not on on her phone. Told her to turn it off and SingaporeAir and Emirates worked again.
I've since advised AVG of the issue via facebook... the only easy way I could see to contact support...

So advise to companies running WAF's. Your customers are going to get blocked, it isn't going to only be bad guys. Make your pages as helpful as possible, maybe even a link to an unblocked contact us page, or an email address they can send their long obfuscated reference number too.

Advice to endpoint security companies... Since when did you jump on the VPN bandwagon? Ok, cool, do that, sure it is a revenue stream. But please monitor your IP's for being blacklisted as they are then going to be blocked from a large number of sites. Maybe filter what your users can do on these VPN's so you don't block the vast majority of users of your service from using the internet.

Edit: I notified AVG before this post went live, published this a few days later, and checked just now.28/03/19 @ 6pm.. it is still blacklisted