Downloading large files in Python

Vote count:

0

In python 2.7.3, I try to create a script to download a file over the Internet. I use the urllib2 module.

Here, what I have done :

import urllib2

HTTP_client = urllib2.build_opener()
#### Here I can modify HTTP_client headers
URL = 'http://www.google.com'
data = HTTP_client.open(URL)
with open ('file.txt','wb') as f:
        f.write(data.read())

OK. That's work perfectly.

The problem is when I want to save big files (hundreds of MB). I think that when I call the 'open' method, it downloads the file in memory. But, what about large files ? It will not save 1 GB of data in memory !! What happen if i lost connection, all the downloaded part is lost.

How to download large files in Python like wget does ? In wget, it downloads the file 'directly' in hard disk. We can see the file growning up in size.

I'm surprised there is no method 'retrieve' for doing stuff like

HTTP_client.retrieve(URL, 'filetosave.ext')

asked 57 secs ago

Boris

6

Downloading large files in Python