Postmortem -
Read details
Apr 16, 17:04 UTC
Resolved -
This incident has been resolved. We continue to work on a post-mortem, and will share it when it's available. -IF
Apr 10, 13:54 UTC
Update -
We are no longer seeing lags with processing of submissions and queries in our queue (doi.crossref.org). That said, we continue to monitor the performance of the queue, while we work on that post-mortem. -IF
Apr 9, 19:55 UTC
Monitoring -
We believe a fix has been found and implemented; everything is flowing far faster now. We will monitor the results for a bit longer and share a post-mortem soon.
-GH
Apr 9, 11:29 UTC
Identified -
We believe we have identified the underlying issue that has been causing submission processing lags. We're working on a fix. -IF
Apr 8, 20:30 UTC
Update -
We are continuing to investigate this issue.
Apr 7, 21:45 UTC
Update -
We're still focused on speeding up slow deposit and query processing in the submission queue. Some progress was made over the weekend so things are looking better; more details soon.
-GH
Apr 6, 10:14 UTC
Update -
We are continuing to investigate this issue.
Apr 4, 19:08 UTC
Update -
Our technical team continues to investigate the cause of these performance lags. -IF
Apr 3, 21:31 UTC
Update -
We are continuing to investigate this issue.
Apr 2, 20:43 UTC
Investigating -
We reverted additional code earlier today during our weekly release in an attempt to pinpoint the origin of the performance lags in the submission and query queues at doi.crossref.org. Unfortunately, this did not result in a return to the processing speeds that we saw prior to 18 March. We have enabled additional processing threads for submissions and queries and continue to investigate the source of the lags. -IF
Apr 1, 19:02 UTC
Update -
We are continuing to monitor for any further issues.
Mar 31, 17:07 UTC
Update -
The emergency maintenance has now been completed. We have dedicated additional resources to drive down the backlog of pending submissions and queries in the queue at doi.crossref.org. -IF
Our changes have improved performance. We continue to monitor the results.
Mar 29, 14:37 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
Mar 29, 13:58 UTC
Investigating -
Unfortunately, today's special release has not resolved the issue. We're continuing to investigate the cause of the performance lags in the submission queue - doi.crossref.org. Resolving these lags is our technical team's highest priority. -IF
Mar 27, 17:42 UTC
Update -
We will be performing special maintenance - release v0.231.0 - tomorrow, 27 March, to try to further improve submission processing performance. -IF
Mar 26, 19:56 UTC
Update -
Our change yesterday improved performance in the submission queue - doi.crossref.org - for several hours, but we're seeing lags again today. We're investigating other potential causes of this issue. -IF
Mar 26, 13:20 UTC
Update -
The deployment earlier today did not fix the performance lags in our submission queue - doi.crossref.org. Our technical team has reverted a change that they believe is responsible for the lags. We're monitoring the results. -IF
Mar 25, 17:38 UTC
Monitoring -
We have observed slower processing times in our submission queue - doi.crossref.org - during the last few days. We've exceeded 50,000+ pending submissions at the moment. We're still processing your submissions. We're hopeful that today's deployment will fix the performance lags. We're monitoring the issue. -IF
Mar 25, 14:13 UTC