I have been trying, unsuccessfully, for two days to download the latest iPhone SDK and XCode IDE from the Apple Developer Center with absolutely no luck.

I’ve recently moved in to a new house and the DSL connection we have, slow at the best of times, flakes out several dozen times per day. Connectivity problems aside, there had to be a way of reliably downloading large files from Apple’s Developer Center without using Firefox or Internet Explorer because neither of them would resume a failed download. Unfortunately, after having also messed about with a few different download managers, none them would continue a broken download either.

What to try next?

How about good old fashioned WGET?

WGET, originally for UNIX, and available as part of the GNUWin32 project for Windows, is an absolutely fantastic command line tool. WGET lets you retrieve web pages and files from both web servers and ftp servers anywhere on the internet. It’s robust, flexible, versatile, reliable and is a really handy tool to have in your tool box.

Back to the problem of reliably downloading XCode and the iPhone SDK, a two gigabyte disc image which means plenty of chances for crapping out before the download completes.

Apple’s Developer Center requires that you login, which means session state, and that is going to be in the form of a cookie. That’s easy then, we just need to get access to that cookie. One twist though, and I found this out after a little messing about is that when you download the iPhone SDK, or in fact any kind of secured download, it gives you another cookie that is valid for that download.

The job is to get those cookies out of Firefox, or whatever other web browser you use, and into a format suitable for WGET.

I will detail how to export your cookies in a while, but the first thing you have to do is get the cookies from Apple.

Load up Firefox or Internet Explorer , login to the Apple Developer Center website at http://developer.apple.com, then navigate to the file you want to download, in this case, XCode 3.2.3 with the iPhone 4 SDK.

The URL for that particular file is https://developer.apple.com/mac/scripts/downloader.php?path=/iphone/iphone_sdk_4__final/xcode_3.2.3_and_iphone_sdk_4__final.dmg.

Make sure you click to download the file via your web browser. That step is very important. After a few seconds of file transfer you can terminate the download.

The reason for this is that you need the download authorization cookie as well as the website user login cookie that are supplied. If you do not do this download and terminate step, you won’t get the proper cookies.

Instructions to Export Cookies from Firefox

The version of Firefox that I am running stores the cookies in a SQLite database, not exactly human readable.

Firefox add-ons to the rescue!

I am currently running Firefox 3.6.4 so I used the Cookie Exporter 1.5 add-on to export my cookies in to a cookies.txt file that WGET can make use of.

Assuming you have Firefox open and are logged in to the Apple Developer Center.

Step #1

Install Cookie Exporter 1.5 or similar add-on if you do not already have a cookie exporter add-on available.

Step #2

Click Export Cookies from the Tools menu

Step #3

Browse to the location where you want to export your cookies and click Save. I put mine in C:\temp.

Instructions to Export Cookies from Internet Explorer

Assuming you have Internet Explorer open and are logged in to the Apple Developer Center.

Step #1

Click Import and Export…

Step #2

Click “Export to a file” and then click “Next”

Step #3

Click “Cookies” and then click “Next”

Step #4

Browse to the location where you want to export your cookies to and click next.

If you are on Windows, and use Firefox, it may be easier, if you don’t want the hassle of installing an additional add-on to login in to the Apple Developer Center via Internet Explorer. Click the download link you want, and then export your cookies directly from Internet Explorer. It achieves the same result.

Once the cookies are exported in to a cookies.txt file, right click on the link and select “Copy Link Location.”

Right click “Copy Link Location” in Firefox to copy the URL location.

That puts the URL to the file on the clipboard and can be pasted into a command window at the appropriate time.

Okay, I’ve got the cookies, I’ve got the URL of the file. Time to open a command window and navigate to where the location where the cookies.txt file was saved.

Once that’s done, issue the following WGET command: WGET –server-response –continue –no-check-certificate –load-cookies=cookies.txt https://developer.apple.com/mac/scripts/downloader.php?path=/iphone/iphone_sdk_4__final/xcode_3.2.3_and_iphone_sdk_4__final.dmg

The full URL of the file to download is on the clipboard by now, if you have been following along, and so you can just paste it the URL in to the command prompt window.

Paste the download link in to the command prompt window.

The WGET Options Explained

Continue: Should you need to terminate the download for any reason, such as a reboot, or closing the command prompt, you can continue downloading where you left by using the –continue option.

No-Check-Certificate: Apple appears to use a self-signed certificate on their Developer website. This means that when WGET goes to check the certificate authority, it throws out an error because WGET cannot verify authenticity. The certificate in this case is not used to prove that Apple is Apple, but to enable a secure connection between your computer and the Apple Developer server. A self-signed certificate is nothing to concern yourself with in this case, your data transfer, including user-name and password are all encrypted between your computer and Apple’s servers.

Load-Cookies: The cookies we saved from our browser session need to be loaded up by WGET, this option allows that to happen. Some older WGET clients will require that the equals sign between –load-cookies and the filename be omitted, be aware of that.

Server-Response: The reason to use the “–server-response” option is to debug the transfer when something goes wrong. With this option you can figure out what the server is trying to tell you. Perhaps a 403 Forbidden error, in which case, make sure you logged in and clicked at least once on the download you want. Perhaps you are getting a 404 Page Not Found, in which case, check to make sure you entered the URL of the file correctly.

The last parameter that begins https://developer.apple.com is the actual URL of the file to download. Again, you can paste that in by copying the download link from your browser.

What You Should See If Everything Went Right

If everything is working correctly, you should see something like the following:

In that screenshot of the command prompt window, I issued the WGET command and relevant options, and WGET has begun the download of the latest version of XCode and the iPhone 4 SDK. I also threw in the option to limit the download rate to just five kilobytes a second so that it does not affect anybody else trying to use the internet connection.

What To Do When Things Go Wrong

When you first attempt the download, you may get a failure, just a few kilobytes of downloaded of binary data, and nothing more. If that’s the case, don’t worry, it’s one of two things. In my case it was an old version of WGET.

Firstly, simplest thing to check, you do not have the necessary cookies, in which case, start from the top. You can verify you have the correct cookies by looking in the cookies.txt file for the string ADCDownloadAuth. This is the cookie that Apple Developer Center sends you when you request a download from them.

You can easily search the cookies file by issuing the following command at the command prompt: find “ADCDownloadAuth” cookies.txt

Find the relevant cookie

Second thing to check: You might be like me and have a WGET client that just cannot handle a two gigabyte download. On my workstation I confusingly have two WGET clients, both by GNU. I have version 1.8.2 and version 1.11.4. I have to keep the older version around for a particular application that I am developing for a website that requires a specific version of the WGET client. Unfortunately in my case version 1.8.2 and version 1.11.4 were both on the path, but it found the older version first and decided to execute that.

You can find out which version of WGET you have installed, if any, by issuing the following command at the command prompt:

WGET –version

Make sure you are running an up to date version of WGET otherwise you may not be able to download properly.

These instructions work equally well if you are on a Mac or Linux, you just need to export the cookies from your browser by whatever means you use.

The great thing about using WGET for a big download like this is that the utility will just keep plugging away in the background, until the operation has completed, no matter how many times you have to cancel and continue the download.

If you have both CURL and WGET installed, you can also do the login and download directly from the command line without having to use your browser at all. Just three CURL commands followed by WGET. I’ll leave that solution as an exercise for the reader.

Subscribe to the RSS feed or follow me on Twitter to stay up to date!