I wanted to add a plugin to Discourse, so I
followed the usual steps for doing so, which
involves adding the plugin to the app configuration
and then running
./launcher rebuild app.
However, I accidentally forgot that
./launcher rebuild app will also update
Discourse -- but the update failed. So, now I'm
stuck trying to figure out why the update failed.
Looking at the logs, I see:
I, [2025-04-02T08:51:34.524786 #1] INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
`/root` is not writable.
Bundler will use `/tmp/bundler20250402-1714-yuyf6i1714' as your home directory temporarily.
rake aborted!
Aws::S3::Errors::InvalidArgument: Unsupported header 'x-amz-checksum-crc32' received for this API call. (Aws::S3::Errors::InvalidArgument)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/client.rb:17315:in `put_object'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2994:in `block in put'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in `metric'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2993:in `put'
/var/www/discourse/lib/s3_helper.rb:82:in `upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `open'
/var/www/discourse/lib/tasks/s3.rake:41:in `upload'
/var/www/discourse/lib/tasks/s3.rake:197:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/s3.rake:197:in `each'
/var/www/discourse/lib/tasks/s3.rake:197:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
I, [2025-04-02T08:51:48.501828 #1] INFO -- : Installing CORS rules...
skipping
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.br.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.gz.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js.map
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.br.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.gz.js
Uploading: assets/scripts/discourse-test-listen-boot-c65930f97c9935680e942f8e32df616cc91ab7c9371b86db6e5ddf9ad868ae22.js
I, [2025-04-02T08:51:48.507455 #1] INFO -- : Terminating async processes
I, [2025-04-02T08:51:48.508153 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main pid: 41
2025-04-02 08:51:48.508 UTC [41] LOG: received fast shutdown request
I, [2025-04-02T08:51:48.508981 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 108
108:signal-handler (1743583908) Received SIGTERM scheduling shutdown...
108:M 02 Apr 2025 08:51:48.513 # User requested shutdown...
108:M 02 Apr 2025 08:51:48.514 * Saving the final RDB snapshot before exiting.
2025-04-02 08:51:48.518 UTC [41] LOG: aborting any active transactions
2025-04-02 08:51:48.533 UTC [41] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
2025-04-02 08:51:48.539 UTC [50] LOG: shutting down
2025-04-02 08:51:48.542 UTC [50] LOG: checkpoint starting: shutdown immediate
2025-04-02 08:51:48.569 UTC [50] LOG: checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.013 s, sync=0.003 s, total=0.030 s; sync files=2, longest=0.002 s, average=0.002 s; distance=1 kB, estimate=23 kB
2025-04-02 08:51:48.588 UTC [41] LOG: database system is shut down
108:M 02 Apr 2025 08:51:48.620 * DB saved on disk
108:M 02 Apr 2025 08:51:48.621 # Redis is now ready to exit, bye bye...
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 1712 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
6edbf13ce0bdc52df819524321e5e6b66bca7dd7ce2c5997262ba8008c6de371Running discourse-doctor does not
successfully diagnose the problem.
First, I looked up the "/root is not
writable" error message and found this Discourse
support thread. However, there was no resolution
to this issue in the thread and the original poster
appears to have a working Discourse instance, so I
doubt this is the issue.
Looking up the error message about the unsupported header online, I find a Discourse support thread about the same issue dating back to Feb 24th. I use Backblaze for S3 storage and apparently, the AWS SDK developers introduced a breaking change with no regard to third party implementations and won't fix it:
Hi - I’m Pat Patterson, Chief Technical Evangelist at Backblaze; I arrived on this thread because I have a self-hosted proof-of-concept Discourse forum, and I happened to bump into this exact issue today while configuring my forum to use Backblaze B2 for backups and uploads.
[...]
The problem is that a checksum (either the Content-MD5 header or one of the new checksum headers) is required (rather than just supported) for these operations, and this causes the current AWS SDKs to provide the new checksum header. As far as I know, there is no way to override this and have the SDK provide Content-MD5 as it used to.
It also appears that the AWS SDK will not be fixed, as the issue was raised but closed as "not planned":
Yeah, and AWS SDK maintainers are only giving us the cold shoulder on this.
https://github.com/aws/aws-sdk-js-v3/issues/6819#issuecomment-2625426822
The fix appears to be to remove the
expire_missing_assets step:
It’s reported that the AWS SDK is not respecting those ENVs for DELETE operations, so you need to remove the
- sudo -E -u discourse bundle exec rake s3:expire_missing_assetsline for now.
However, this also fails with the same error. It
appears that the s3:upload_assets step
fails before it gets to expiring the missing
assets.
Reading earlier in the thread, I see that a different suggestion was made first:
Try adding to your ENV
AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED
Judging by the context in the thread, I'm
guessing that this is also required as part
of the workaround. So, I try again with the ENV
change and also the
expire_missing_assets step commented
out:
env:
AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED
## elided
## elided
hooks:
## elided
after_assets_precompile:
- exec:
cd: $home
cmd:
- sudo -E -u discourse bundle exec rake s3:upload_assets
## - sudo -E -u discourse bundle exec rake s3:expire_missing_assetsIt works!
This does mean that assets won't be deleted from the bucket anymore...
Yes, assets won’t be removed from the bucket anymore.
...but I think that's an okay workaround for now.