Skip to main content

View Post [edit]

Poster: saurik Date: Mar 17, 2017 2:24am
Forum: general Subject: archive doesn't work for my website :(

Hello. I have a website, https://www.jayfreeman.com/, which shows four captures during 2016, but if you try to click on any of them it just skips back to a capture in 2010; if I ask the site to capture now, I get "This url is not available on the live web or can not be archived.". For a while, robots.txt was returning a 403 error... could that have caused this problem? I fixed that a week ago (now that returns a 200 for an empty text file), but I'm still getting this error message :(. I would definitely appreciate any help understanding what I'm doing wrong.

Reply [edit]

Poster: MeditateOrDie Date: Apr 21, 2017 10:54pm
Forum: general Subject: Re: archive doesn't work for my website :(

The problem might be something to do with http://cloudfront.com/ cdn cache.
eg: https://web.archive.org/web/20170422055304/http://52.85.89.221:80/

This is happening a lot, to many sites recently.

Perhaps try changing your http://cloudfront.com/ AKA https://aws.amazon.com/cloudfront/ settings if they are available to you, otherwise contact your hosting company for a solution if they are controlling the cdn settings. Be firm if they resist - they work for you.

Reply [edit]

Poster: saurik Date: Apr 22, 2017 12:26am
Forum: general Subject: Re: archive doesn't work for my website :(

Do you have any idea what could be wrong, though? I see your link to the archive for that IP address, but that is exactly the response one would expect from any name-based virtual host: I'd go so far as not only to say that the vast majority of websites return an error message if you try to connect to them with an IP address for the Host header, but that a website which responds with anything more solid than a 301 redirect to such a request is incorrectly configured (but would personally much prefer to see a 400, which is what CloudFront happens to return there). I am more than happy to modify my hosting stack, but I just don't see what could be wrong currently.

Reply [edit]

Poster: MeditateOrDie Date: Apr 23, 2017 10:34pm
Forum: general Subject: Re: archive doesn't work for my website :(

I know,
I pinged your sites domain to get an IP then used web.archive.org/save/http://52.85.89.221:80/ to see what response I got from the attempt, which was amazons cloudfront cdn.

I tried a number of tricks which work for most sites behind overly restrictive cdn networks, which all failed. There are some more tricks I could try, though none of them will solve your problem.

It looks like cloudfront is configured to ignore most connections from web.archive,org which would explain why archive.org says that the site is not available on the web when you try to save it. Responses from a browser or fiddler4 https://fiddler2.com/r/?GetFiddler4 will be different to what web.archive.org sees (or doesn't see) when saving.

... If not cloudfront then maybe something in your server config, which seems doubtful if you have not changed anything.

https://archive.is/? can access your page, though it's not a complete copy ie: no scripts and other files 'as-is':
https://archive.is/tylyz

webcite also:
http://www.webcitation.org/query?url=https%3A%2F%2Fwww.jayfreeman.com%2F&date=2017-04-22

~~~
I also made an .MHT archive which should contain all of the pages files, then backed it up to archive.org just in case it's ever needed:
Saves:
https://web.archive.org/web/*/http://aspnetmht.com/getmht.aspx?mht=www.jayfreeman.com_201704220231.zip
DL:
https://web.archive.org/web/http://aspnetmht.com/getmht.aspx?mht=www.jayfreeman.com_201704220231.zip

There are tools available to extract all files from an .MHT archive into a folder from which the page can easily be re-constructed in future if need be.

The .mht did not display the image in my IE11/Win10 for some reason, though it's probably just a minor issue and if so then the .MHT could be patched to make it fully functional.

Good luck!
This post was modified by MeditateOrDie on 2017-04-24 05:34:24

Reply [edit]

Poster: MeditateOrDie Date: Apr 23, 2017 11:25pm
Forum: general Subject: Re: archive doesn't work for my website :(

A possible work-around: Some quick experimentation showed that most sites behind amazons cloudfront.com are able to be saved via web.archive.org/save/http://... , not https://, so if you were able to temporarily remove the 301 redirect to https, then you should be able to save your page with:
http://web.archive.org/save/http://www.jayfreeman.com/
Note: You may have to clear persistent cookies in you browser/(s) if it/they still redirect (301) to https://
This post was modified by MeditateOrDie on 2017-04-24 06:25:34