(no subject)

[kradeelav]

> wakes up
> reads about crowdstrike (shitty security vendor) pushing an update that took out half the US
 > dismayingly reads the tl;dr by several sysadmins
> jaw dropping very slowly at how many garbage-ass practices led to this clusterfuck. (like seriously, basic shit like not having forced updates plus not slowly rolling out a major update)

seriously if i as a shallow-end-of-the-kiddie-pool-linux-hobbyist-beginner knew not to do two thirds of this shit. just... what. the fuck.

reason no 3255322 my ass ain't ever working in tech

Comments

[tei]

this post... so true https://jorts.horse/@beanclock/112812161378603443

[kradeelav]

I was having such a shitty day so far and your link made me genuinely cackle out loud ty ty <3

[helicoprion]

I cannot log into my work computer because of this. Other coworkers are earnestly rebooting over and over and trying to log in every 30 minutes and sending me panicked emails asking me to tell our boss that they're having trouble logging in ~for some reason~ (our boss also can't log in, if I get him a message he won't see it) and it's like... have you considered that this is just a surprise holiday? In commemoration of bad IT practices?

[kradeelav]

you know what? that really, really should be a for real holiday, lol.

[queenlua]

i saw the news & immediately texted the friend i know at crowdstrike with a wall of troll emojis to dunk on them, like the good baptist girl i was raised to be, lollollol

my annoying "well actually" take which can be SAFELY IGNORED I PROMISE is (1) it'll be sort of hard to understand the full story of What Went Wrong until we get a full postmortem; it's entirely possible crowdstrike does ordinarily have reasonable best practices but some godawful confluence of multiple things failing all at once caused a really-difficult-to-see-in-advance disaster, a la point #3 in how systems fail https://how.complexsystems.fail/ , so i'm suspending judgment til we get a clearer picture of what happened, but also (2) the real solution to this problem is "don't run closed-source third-party ring-0 code" because the potential failure modes are so egregiously catastrophic, but eh that's kind of a moot point because for various Mostly-Bad-But-Sometimes-Okay-ish Reasons (e.g. compliance) people seem to still think Antivirus Software Is The Answer and it's effectively *impossible* to write antivirus software that isn't ring-0 code, so we're stuck with it, but anyway (3) at the end of the day the incentives in this space all kind of point toward... this sort of thing happening? "no one ever got fired for buying IBM" and now all our infrastructure uses the same 3 clouds lmao FUCK

but also:

[kradeelav]

lua, i hope you know that that image and you sending troll emojis to the friendo got a very needed belly laugh from me last week XD oh my god. you little shit.

and bless you also for having some sound reasoning as well! it was pretty fascinating learning about ring0 code vs the rest -- if a virus is so nimble to get root/kernel access to everything (hell reading about BIOS level viruses was crazy), then true, the defense has to be that embedded. i do not envy you cybersecurity folks lol

[neotula]

yeah, it was insane to read about that this morning: seems like a wild mistake/series of mistakes to make that's gonna cause so many problems.

[kradeelav]

soooo many problems yeah... it was wild checking into a hotel 4-5 days later and they still had issues with crowdstrike very visible to the guests. not many bugs these days that are that level.