Question:
I am attempting to install boto3 on an ec2 instance running RHEL7.6. I am behind a proxy, and the proxy information is listed in /etc/environment. I am certain the proxy is working. I am able to curl the https://pypi.org and use our proxy regularly. I am running Python 2.7.5 and Pip 8.1.2.
I run the following command to produce the 502 Bad Gateway:
pip install --proxy http://<redacted-ip>:<redacted-port> -v --log pipinstall1.log boto3
I’ve included the –proxy flag because I encountered the same errors without it. I wanted to be sure that the proxy is used by pip, but to no avail.
I apologize for pasting the log directly, but I am unable to use hosting service to attach the log file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
Collecting boto3 1 location(s) to search for versions of boto3: * https://pypi.python.org/simple/boto3/ Getting page https://pypi.python.org/simple/boto3/ Looking up "https://pypi.python.org/simple/boto3/" in the cache Returning cached "301 Moved Permanently" response (ignoring date and etag information) Looking up "https://pypi.org/simple/boto3/" in the cache No cache entry available Starting new HTTPS connection (1): pypi.org Incremented Retry for (url='/simple/boto3/'): Retry(total=4, connect=None, read=None, redirect=None) Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Starting new HTTPS connection (2): pypi.org Incremented Retry for (url='/simple/boto3/'): Retry(total=3, connect=None, read=None, redirect=None) Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Starting new HTTPS connection (3): pypi.org Incremented Retry for (url='/simple/boto3/'): Retry(total=2, connect=None, read=None, redirect=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Starting new HTTPS connection (4): pypi.org Incremented Retry for (url='/simple/boto3/'): Retry(total=1, connect=None, read=None, redirect=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Starting new HTTPS connection (5): pypi.org Incremented Retry for (url='/simple/boto3/'): Retry(total=0, connect=None, read=None, redirect=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Starting new HTTPS connection (6): pypi.org Could not fetch URL https://pypi.python.org/simple/boto3/: connection error: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/boto3/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))) - skipping Could not find a version that satisfies the requirement boto3 (from versions: ) Cleaning up... No matching distribution found for boto3 Exception information: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 307, in run requirement_set.prepare_files(finder) File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 370, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 522, in _prepare_file finder, self.upgrade, require_hashes) File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 268, in populate_link self.link = finder.find_requirement(self, upgrade) File "/usr/lib/python2.7/site-packages/pip/index.py", line 491, in find_requirement 'No matching distribution found for %s' % req DistributionNotFound: No matching distribution found for boto3 Looking up "https://pypi.python.org/pypi/pip/json" in the cache Returning cached "301 Moved Permanently" response (ignoring date and etag information) Looking up "https://pypi.org/pypi/pip/json" in the cache No cache entry available Starting new HTTPS connection (1): pypi.org There was an error checking the latest version of pip Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pip/utils/outdated.py", line 126, in pip_version_check headers={"Accept": "application/json"}, File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 487, in get return self.request('GET', url, **kwargs) File "/usr/lib/python2.7/site-packages/pip/download.py", line 378, in request return super(PipSession, self).request(method, url, *args, **kwargs) File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 606, in send history = [resp for resp in gen] if allow_redirects else [] File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 179, in resolve_redirects **adapter_kwargs File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 585, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send resp = super(CacheControlAdapter, self).send(request, **kw) File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 465, in send raise ProxyError(e, request=request) ProxyError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/pip/json (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 502 Bad Gateway',))) |
Update
To eliminate the potential that Python 2.7.5 is too old to download from PyPi, I stood up a new ec2 instance running RHEL 8. This instance is running Python 3.6.8 and pip 9.0.3. I verified the proxy is working via curl https://pypi.org/simple/boto3/ and it returned successfully.
I am still receiving the 502 Bad Gateway when running:
pip3 --log pip3_log.txt --proxy http://<redacted-ip>:<redacted-port> --trusted-host pypi.org install boto3
The output of pip3_log.txt is below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
Collecting boto3 1 location(s) to search for versions of boto3: * https://pypi.python.org/simple/boto3/ Getting page https://pypi.python.org/simple/boto3/ Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))': /simple/boto3/ Could not fetch URL https://pypi.python.org/simple/boto3/: connection error: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/boto3/ (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))) - skipping Could not find a version that satisfies the requirement boto3 (from versions: ) Cleaning up... No matching distribution found for boto3 Exception information: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 346, in run requirement_set.prepare_files(finder) File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 381, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 557, in _prepare_file require_hashes File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link self.link = finder.find_requirement(self, upgrade) File "/usr/lib/python3.6/site-packages/pip/index.py", line 514, in find_requirement 'No matching distribution found for %s' % req pip.exceptions.DistributionNotFound: No matching distribution found for boto3 There was an error checking the latest version of pip Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 595, in urlopen self._prepare_proxy(conn) File "/usr/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 816, in _prepare_proxy conn.connect() File "/usr/lib/python3.6/site-packages/pip/_vendor/urllib3/connection.py", line 294, in connect self._tunnel() File "/usr/lib64/python3.6/http/client.py", line 919, in _tunnel message.strip())) OSError: Tunnel connection failed: 502 Bad Gateway During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3.6/site-packages/pip/_vendor/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/pip/json (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pip/utils/outdated.py", line 141, in pip_version_check headers={"Accept": "application/json"}, File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 521, in get return self.request('GET', url, **kwargs) File "/usr/lib/python3.6/site-packages/pip/download.py", line 386, in request return super(PipSession, self).request(method, url, *args, **kwargs) File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 640, in send history = [resp for resp in gen] if allow_redirects else [] File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 640, in history = [resp for resp in gen] if allow_redirects else [] File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 218, in resolve_redirects **adapter_kwargs File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.6/site-packages/pip/_vendor/requests/adapters.py", line 502, in send raise ProxyError(e, request=request) pip._vendor.requests.exceptions.ProxyError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/pip/json (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 Bad Gateway',))) |
Answer:
I ran into the same issue a few days ago, because I was also using a proxy which had only 2 domains whitelisted: pypi.org and files.pythonhosted.org
The problem is that pip is running on an older version. In my case pip was running on version 9.0.1. This version of pip tries to look for packages at an old url pypi.python.org which now is actually redirected to pypi.org
|
1 2 3 4 5 6 7 8 9 |
1 location(s) to search for versions of pip: * https://pypi.python.org/simple/pip/ Getting page https://pypi.python.org/simple/pip/ Looking up "https://pypi.python.org/simple/pip/" in the cache No cache entry available Starting new HTTPS connection (1): pypi.python.org Incremented Retry for (url='/simple/pip/'): Retry(total=4, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',))': /simple/pip/ |
You can upgrade pip offline by transferring the whl file. You can download it from here https://files.pythonhosted.org/packages/5a/4a/39400ff9b36e719bdf8f31c99fe1fa7842a42fa77432e584f707a5080063/pip-20.2.2-py2.py3-none-any.whl
Upgrade pip by running:
$ python3 -m pip install pip-20.2.2-py2.py3-none-any.whl
Verify pip is upgraded by running:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
$ python3 -m pip show pip Name: pip Version: 20.2.2 Summary: The PyPA recommended tool for installing Python packages. Home-page: https://pip.pypa.io/ Author: The pip developers Author-email: distutils-sig@python.org License: MIT Location: /home/ubuntu/.local/lib/python3.6/site-packages Requires: Required-by: |
The newer pip looks for packages at pypi.org directly and downloads all files required from files.pythonhosted.org.
Once pip was upgraded, I was able to install pip packages.
Important note :
Please ensure you have whitelisted pypi.org and files.pythonhosted.org. Without access to these domains pip wont be able to install any package using internet.