Pip trusted_host问题记录

问题定位

一日我在Pipenv上收到一个issue: 用户说Pipenv执行的pip命令中--trusted-host缺少了port部分。然后我去扒源码,结果发现有两处同样的函数:[1][2]逻辑不一致。顿时感觉事情没那么简单。于是我本地搞了一个pypi server, 并用自签名支持了https,然后用pip测试1

Bash
$ pip install -i https://localtest.me:5001 urllib3 --trusted-host localtest.me:5001
Successful

$ pip install -i https://localtest.me:5001 urllib3 --trusted-host localtest.me
Looking in indexes: https://localtest.me:5001
Collecting urllib3
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)'))': /urllib3/
  ...
  Failed

$ pip install -i http://localtest.me:5000 urllib3 --trusted-host localtest.me:5000
Looking in indexes: http://localtest.me:5000
Collecting urllib3
  The repository located at localtest.me is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host localtest.me'.
  Could not find a version that satisfies the requirement urllib3 (from versions: )
No matching distribution found for urllib3

$ pip install -i http://localtest.me:5000 urllib3 --trusted-host localtest.me
Successful

惊呆,HTTPSHTTP针对trusted-host带不带port的处理方式不一样:HTTPS希望你带port,而HTTP不需要带port。这显然是不合理的,于是我去看pip的源码关于这块的处理逻辑。以下基于pip 19.2.3的源码。 src/pip/_internal/download.py

Python
...
    insecure_adapter = InsecureHTTPAdapter(max_retries=retries)
    # Save this for later use in add_insecure_host().
    self._insecure_adapter = insecure_adapter

    self.mount("https://", secure_adapter)
    self.mount("http://", insecure_adapter)

    # Enable file:// urls
    self.mount("file://", LocalFSAdapter())

    # We want to use a non-validating adapter for any requests which are
    # deemed insecure.
    for host in insecure_hosts:
        self.add_insecure_host(host)

def add_insecure_host(self, host):
    # type: (str) -> None
    self.mount('https://{}/'.format(host), self._insecure_adapter)

insecure_adapter是不检查证书的,secure_adapter是检查证书的,可以看到在add_insecure_host()这个函数中,是把传进来的host加上末尾的/拼成一个URL来新增一个adapter端点的,而在requests中,多个adapter端点是依靠startswith来识别是否匹配的。所以如果trusted-hostexample.org,则只有https://example.org/会被识别为信任的站点而https://example.org:8080/不会。

以上是仅针对HTTPS而言,HTTP是无需证书检查的,它相关的逻辑在

src/pip/_internal/index.py

Python
def _validate_secure_origin(self, logger, location):
    # type: (Logger, Link) -> bool
    # Determine if this url used a secure transport mechanism
    parsed = urllib_parse.urlparse(str(location))
    origin = (parsed.scheme, parsed.hostname, parsed.port)

    # The protocol to use to see if the protocol matches.
    # Don't count the repository type as part of the protocol: in
    # cases such as "git+ssh", only use "ssh". (I.e., Only verify against
    # the last scheme.)
    protocol = origin[0].rsplit('+', 1)[-1]

    # Determine if our origin is a secure origin by looking through our
    # hardcoded list of secure origins, as well as any additional ones
    # configured on this PackageFinder instance.
    for secure_origin in self.iter_secure_origins():
        if protocol != secure_origin[0] and secure_origin[0] != "*":
            continue

        try:
            # We need to do this decode dance to ensure that we have a
            # unicode object, even on Python 2.x.
            addr = ipaddress.ip_address(
                origin[1]
                if (
                    isinstance(origin[1], six.text_type) or
                    origin[1] is None
                )
                else origin[1].decode("utf8")
            )
            network = ipaddress.ip_network(
                secure_origin[1]
                if isinstance(secure_origin[1], six.text_type)
                # setting secure_origin[1] to proper Union[bytes, str]
                # creates problems in other places
                else secure_origin[1].decode("utf8")  # type: ignore
            )
        except ValueError:
            # We don't have both a valid address or a valid network, so
            # we'll check this origin against hostnames.
            if (origin[1] and
                    origin[1].lower() != secure_origin[1].lower() and
                    secure_origin[1] != "*"):
                continue
        else:
            # We have a valid address and network, so see if the address
            # is contained within the network.
            if addr not in network:
                continue

        # Check to see if the port patches
        if (origin[2] != secure_origin[2] and
                secure_origin[2] != "*" and
                secure_origin[2] is not None):
            continue

        # If we've gotten here, then this origin matches the current
        # secure origin and we should return True
        return True

    # If we've gotten to this point, then the origin isn't secure and we
    # will not accept it as a valid location to search. We will however
    # log a warning that we are ignoring it.
    logger.warning(
        "The repository located at %s is not a trusted or secure host and "
        "is being ignored. If this repository is available via HTTPS we "
        "recommend you use HTTPS instead, otherwise you may silence "
        "this warning and allow it anyway with '--trusted-host %s'.",
        parsed.hostname,
        parsed.hostname,
    )

    return False

看上去是分别匹配scheme, hostnameport,没什么问题。问题在于self.iter_secure_origins()这里产生的值,在同一个文件中:

Python
def iter_secure_origins(self):
    # type: () -> Iterator[SecureOrigin]
    for secure_origin in SECURE_ORIGINS:
        yield secure_origin
    for host in self.trusted_hosts:
        yield ('*', host, '*')

这里没做任何处理,就把trusted-host当做hostname丢出来了,看来这里压根没考虑过trusted-hostport的需求。

问题修复

找到了问题所在,总结一下:

  • HTTPS需要带port是因为requests.Sessionmount是依靠前缀匹配来获取对应的适配器(adapter)的,并且末尾会加上一个/
  • HTTP需要不带port是因为检查是否安全URL的时候,是拿目标URLhostname(不带port)去匹配trusted-host的值。

所以对应的修复方法就是:

  • 添加信任的端点时,如果trusted-host不带port,则需要把https://hostname:也添加为无需安全检查的端点(利用前缀匹配的特性)。
  • 生成secure_origin时解析传入的trusted-host值,分成hostnameport部分分别匹配。

具体代码可以看我提交的PR,这个PR已经被merge,预计可以在下个版本中发布。

  1. 这里我用了一个trick,使用了localtest.me转发localhost的请求,因为localhost是永远被信任的地址,trusted-host不起作用。

Last edited
Post Link: http://frostming.com/2019/09-03/pip-trusted-host
This work is licensed under a CC-BY-SA License.
TABLE OF CONTENTS