Sometimes, warosu.org doesn't archive certain files, and those files' URLs end up being what they were on 4chan's CDN. These <a> attributes aren't grabbed by the code below (something else gets picked up, but I couldn't find enough examples to pinpoint exactly what):
|
url = extr('<a href="', '">') |
|
if url: |
|
if url[0] == "/": |
|
data["image"] = self.root + url |
|
else: |
|
data["image"] = url |
Regardless, in my experience, these posts are deleted from 4chan's CDN before they can be archived by warosu.org, so attempting to download them will result in an error anyway.
In this example, this post was deleted before it could be archived. Trying to download the thread with gallery-dl results in the following error:
[download][error] '#p18461063" class="backlink" onclick="replyhighlight('p18461063'):' URLs are not supported/enabled
[download][error] Failed to download #p18461063" class="backlink" onclick="replyhighlight('p18461063')
In this case, the URL (or rather, the <a> attribute) being passed to gallery-dl is actually from the post itself - an onclick event referencing another post in the same thread.
I suggest adding a check to ensure that the URL in post["image"] belongs to warosu.org before attempting to process it. This avoids grabbing irrelevant or malformed URLs.
Sometimes,
warosu.orgdoesn't archive certain files, and those files' URLs end up being what they were on 4chan's CDN. These<a>attributes aren't grabbed by the code below (something else gets picked up, but I couldn't find enough examples to pinpoint exactly what):gallery-dl/gallery_dl/extractor/warosu.py
Lines 97 to 102 in d9432ee
Regardless, in my experience, these posts are deleted from 4chan's CDN before they can be archived by
warosu.org, so attempting to download them will result in an error anyway.In this example, this post was deleted before it could be archived. Trying to download the thread with
gallery-dlresults in the following error:In this case, the URL (or rather, the
<a>attribute) being passed togallery-dlis actually from the post itself - anonclickevent referencing another post in the same thread.I suggest adding a check to ensure that the URL in
post["image"]belongs towarosu.orgbefore attempting to process it. This avoids grabbing irrelevant or malformed URLs.