Skip to content

[warosu] Skip non-archived 4chan CDN links in warosu extractor #7698

Description

@NecRaul
Contributor

Sometimes, warosu.org doesn't archive certain files, and those files' URLs end up being what they were on 4chan's CDN. These <a> attributes aren't grabbed by the code below (something else gets picked up, but I couldn't find enough examples to pinpoint exactly what):

url = extr('<a href="', '">')
if url:
if url[0] == "/":
data["image"] = self.root + url
else:
data["image"] = url

Regardless, in my experience, these posts are deleted from 4chan's CDN before they can be archived by warosu.org, so attempting to download them will result in an error anyway.

In this example, this post was deleted before it could be archived. Trying to download the thread with gallery-dl results in the following error:

[download][error] '#p18461063" class="backlink" onclick="replyhighlight('p18461063'):' URLs are not supported/enabled
[download][error] Failed to download #p18461063" class="backlink" onclick="replyhighlight('p18461063')

In this case, the URL (or rather, the <a> attribute) being passed to gallery-dl is actually from the post itself - an onclick event referencing another post in the same thread.

I suggest adding a check to ensure that the URL in post["image"] belongs to warosu.org before attempting to process it. This avoids grabbing irrelevant or malformed URLs.

Activity

added a commit that references this issue on Jun 21, 2025
added a commit that references this issue on Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @mikf@NecRaul

      Issue actions