I've been sitting on this blog post for a long while. I have a history of working with (so called) security researchers that I would describe as poor. I don't want to besmirch the profession of security research. I enjoy reading security research write ups, I follow a lot of the security best practices, I subscribe to the security mailing lists of the OSes I use and overall have high regard for the professionals in the field.
On the other hand, over my career I have many different interactions with security companies and researchers working in those companies that have all been bad. I worked at a cyber security company along side security researchers from the IDF's 8200 unit. I received notices on security vulnerabilities on sites I or the companies I work at run (especially after publishing a security.txt policy). But best of all is my experience with the Israeli National Cyber Directorate. Let me get started.
Security audits and certifications
A few of the companies I worked at went through security audits to get a certification (SOC2 or HIPAA). As the person responsible for the infrastructure and our CI/CD pipelines I was a part of the audit from beginning to end and when the audit report was delivered, I addressed some of the findings. From the few audits I took part in, I can say that the worst was a company that ran a few automated scanners in the vein of SSL Test and the better ones ran something akin to Semgrep and maybe checking the OWASP top 10.
All of the audits I've been part of had not produced any worthwhile results. No actual vulnerabilities were ever found and most the time a few publicly available security scanners were used (the screenshot from the SSL Test is still vivid in my mind).
Working with security researchers
I worked at a cyber security company with an actual cyber security product. There we had a security research team with people from the IDF 8200 unit. From my dealings with them, they have poor knowledge of things you would expect (on the level of not knowing the difference between symmetrics and asymmetric encryption) and their research can boiled down to running Nmap and Metasploit.
When one of them learned that I run my own mail server, he claimed to be able to break in to my server. I said go for it, hoping to learn something new and fix whatever vulnerability my server may have. Looking over his shoulder, I saw that he was running Metasploit with a preset for mail servers. Having found nothing (not because I'm that good, I just install security updates and have sane settings) he turned quiet.
The Israeli National Cyber Directorate
I saved the best for last, the reason I felt the urge to write this post. Over the last 3 or 4 years I was contacted 3 times by the INCD to let me know of vulnerabilities they found in my personal sites and services.
The first time I was contacted by phone. I was a little surprised and took the matter seriously. I was told that my mail server had an RCE. Asking for details, I was told the CVE and the person on the other end explained to me that I need to update my mail server. I quickly checked the CVE and I found that Debian had backported the patch but the server version stayed the same (or maybe some suffix was added, I don't remember). I tried to explain that I had a patched server but it fell on deaf ears and they were adamant that the version I was using vulnerable and I had to update ASAP. I thanked them for letting me know and promised to look in to it.
The second time I was again contacted by phone. This time I was less surprised. I was told that my GitLab instance was misconfigured, although it required logging in, repositories were exposed through the /explore URL. I explained that it was deliberate, that I develop opensource software and that is were I store it and make it available for others (if you take a look, all of the repositories have an opensource license and my blog even links to them). Again, it didn't convince the person on the other side. I thanked them for letting me know and promised to look in to it.
The third time I was again contacted by phone. This time I was not a bit surprised. I was told that my SSH server is vulnerable and I have to update it. I explained that I am running OpenSSH on an OpenBSD machine and that the vulnerability in question only happens on Linux machines. The person on the other end didn't know what OpenBSD is (I tried explaining that the developers of OpenBSD also develop OpenSSH, they didn't seem to get it). Showing my age, I complained that this is a waste of the taxes I pay. The person on the other end didn't appreciate it and ended the call.
Closing thoughts
When I was growing up and the internet was becoming accessible to everyone a new phenomenon named script kiddies started. People scanning ports, open Windows shares and guessing SSH usernames and passwords. Then somebody got the bright idea of making a career out of it by selling people some scary stories and exaggerating their own capabilities and calling it security research. While true that there are unpatched and vulnerable machines on the internet, this is not security research and because I have sensible security practices I only encountered false positives due to rudimentary scanners flagging my servers as vulnerable without checking if they are indeed vulnerable.
I don't remember which company it was, but I remember one such company had an realtime map of the internet showing realtime attacks. Looking closely, each ping and each new connection to port 22 was an attack. The field is now filled with charlatans that instead of trying to break in to your servers now try to bill you for running Nmap or verifying your DMARC record. They've turned this in a very successful industry and the Israeli government seems to have fallen to this trap as well (as I'm pretty sure other goverments have as well).