Slower processing in the admin tool (doi.crossref.org) today
Incident Report for Crossref
Resolved
The issue has been resolved and the queue looks back to normal once more.
Posted Oct 21, 2021 - 08:05 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 20, 2021 - 19:34 UTC
Investigating
We are still investigating the root cause of these submission processing delays.
Posted Oct 20, 2021 - 17:43 UTC
Monitoring
A fix has been implemented, and submission processing has resumed at doi.crossref.org. We continue to monitor.
Posted Oct 18, 2021 - 17:44 UTC
Update
We have suspended the queue and are restarting the database and all processing threads. We continue to accept submissions during this time, but processing will be delayed.
Posted Oct 18, 2021 - 15:36 UTC
Update
We're suspending the queue shortly to address the issue of stalling threads. We'll continue to accept your submissions during this time, but there will be a processing lag as a result.
Posted Oct 18, 2021 - 14:51 UTC
Investigating
Over the course of the last week, we've had a few instances of submission processing threads stalling. We have another example of this today. We're investigating the underlying issue. We continue to accept and process submissions during this time, but this recurring issue continues to result in slower processing times.
Posted Oct 18, 2021 - 14:30 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 15, 2021 - 17:09 UTC
Investigating
We look to have stalled processing threads in the admin tool (doi.crossref.org) causing additional delays at the moment. We're investigating.
Posted Oct 15, 2021 - 14:20 UTC
Update
We have re-enabled the queue. Submissions are now processing. Yay! We continue to monitor the queue and your submissions.
Posted Oct 13, 2021 - 17:03 UTC
Update
We will be pausing the submission queue (doi.crossref.org) shortly to remove ~10,000 submissions that one of our members flagged as containing metadata errors. All submissions sent to us during this time will be stored and processed as soon as the erroneous submissions have been removed and the queue reactivated.
Posted Oct 13, 2021 - 16:32 UTC
Monitoring
A fixed has been implemented. Submissions are now processing in our admin tool (doi.crossref.org). We continue to monitor the results.
Posted Oct 12, 2021 - 20:43 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 12, 2021 - 19:49 UTC
Investigating
We look to have stalled processing threads in the admin tool (doi.crossref.org) causing additional delays. We're investigating.
Posted Oct 12, 2021 - 19:16 UTC
Monitoring
Submission processing in the queue is in much better shape today than last week. We have ~27,000 pending submissions in the queue - as of this writing - compared to 75,000+ at several points last week. The current issue is that several users have very large file sizes for book registrations and reference list updates working their way through the queue (these larger submissions require more time for us to match references against and process). Given the logic in submission processing in the queue, those larger submissions are largely affecting other larger submissions. Instead of a general backlog, like we had last week, we currently have a backlog in submissions of very large file sizes. Thankfully, for members submitting smaller file sizes, you will likely not notice any significant lags in submission processing.
Posted Oct 11, 2021 - 19:35 UTC
Identified
Due to yesterday's outage - https://0-www-crossref-org.library.alliant.edu/blog/outage-of-october-6-2021/ - we will need to reprocess all submissions that were processed between 1400 and 1900 UTC yesterday, 6 October. This will result in additional submissions into the queue, which is already delayed. We will be begin reprocessing these submissions imminently.
Posted Oct 07, 2021 - 17:14 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 04, 2021 - 18:16 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 04, 2021 - 14:49 UTC
Investigating
Processing of submissions in the admin tool (doi.crossref.org) is slower than we would like today. We are storing all submissions sent to us and are processing them as fast as we can, but there is a noticeable backlog at the moment. We're investigating.
Posted Oct 04, 2021 - 14:29 UTC
This incident affected: Content Registration (Admin tool, Test admin tool) and Beta (Metadata Manager).