So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.

The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.

I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.

  • LainTrain@lemmy.dbzer0.com
    link
    fedilink
    arrow-up
    23
    ·
    3 days ago

    Cron job that evals some base64 encoded string which is actually downloading a script from a personal GitHub repo of an IT guy who left…