Hi all, we detected 500s on the inbox API endpoint this morning and had to take the app down for 4m 35s to resolve an underlying issue with the database. We've been monitoring ever since and it appears to be resolved.

A certificate issue impacted site availability on @openfn v2 ( from 11:38 to 12:19 UTC today. We have resolved the issue and normal availability has been restored. Apologies!.

Hi all, we've had DNS problems today and this alert didn't work as planned! Apologies. We've been tracking and updating the incident here: . All looks good now, but don't hesitate to reply if you're having trouble with

Hi all. We will be performing database maintenance tonight at 11pm UTC and expect about 60 seconds of disruption on inbox endpoints. You can track updates here:

This afternoon we caused ☹️ and fixed a DNS-related issue impacting systems sending data to openfn inboxes using the naked domain rather than . The issue lasted about an hour an you can read about it here: .

We had a brief outage at 11:31 UTC today. It lasted ~2 minutes. Details here:

502s on @openfn 's swiss servers detected. Investigating now. Please stay tuned for updates.

For updates from @googlecloud directly, please follow

@openfn is back online, the GCP incident seems to be resolved for all @googlecloud customers.

@googlecloud Quick update here: looks like this is a @googlecloud failure impacting a large number of clients. Waiting for an update from GCP at the moment.

Detecting 404s from @googlecloud on right now. Investigating.

All fixed and rolled out across US and Swiss servers:

While runs themselves have not been impacted, logs containing certain special characters are being truncated on the platform. We've replicated the issue are are working on a fix right now:

We've detected problems with @Google and @googlecloud but so far, service at @openfn has not been interrupted. We are monitoring the problem and will let you know here if there is any service degradation.

Still hanging out over here. Not much to report. @kubernetesio is awesome. Also, really loving @sorentwo 's work on Oban. Someone come say hi sometime.

Hi. It's been a while. Just wanted to report that all systems are still healthy over at (And lemme tell you, we're really enjoying this auto-scaled, load balanced deployment of @elixirphoenix on @GCPcloud and #Kubernetes .) #ict4d.

Our US servers had severely degraded performance on the inboxes (receiving requests) and the job runners from 01:54 to 02:12 UTC and again from 02:19 to 03:56 UTC. Everything is back to normal.

GitHub has officially marked this as resolved: [GitHub status] Resolved: Incident on 2020-04-02 20:20 UTC

Github has rolled out a fix to this issue and is currently monitoring the situation. We have tested the OpenFn-GitHub integration and it is working again. Follow the issue on GitHub's status page here:

Notice for enterprise users who version control their OpenFn projects in GitHub: GitHub is currently investigating an issue with their webhooks service. Not all changes made in GitHub may deploy correctly to OpenFn until this is resolved: