大虾居

谈笑有鸿儒,往来无白丁。

0%

前言

Nginx反向代理速度快,可以自定义各种转发规则,可以说是站长的好朋友。

acme.sh支持多个域名服务商的证书生成,让站长免费获取到ssl带来的安全性,也是不可或缺的好伙伴。

本文虾哥将本着DRY原则,让站长不需要记住部署nginx和acme的若干命令,快速搭建起安全实用的ssl反向代理服务器。

1. 准备

使用docker部署程序省去安装各种语言运行环境的麻烦,但docker本身还是需要准备一下。

下面介绍在centos7系统下安装docker 和 docker-compose的步骤。其他系统可以自行查找相关文档。

##1.1 安装docker

sudo yum install docker -y

安装完成后启动docker

sudo systemctl start docker
sudo systemctl enable docker	

##1.2安装docker-compose

docker-compose是python包,通过docker-compose,docker运行镜像的各种配置可以直接写入文件,重启和修改都非常容易。

# 通过yum安装pip官方没有直接提供rpm包,可以通过epel-release库安装
sudo yum install epel-release
sudo yum isntall python2-pip

sudo pip isntall docker-compose

这样就安装完成了,运行docker-compose命令行

docker-compose --help

1.3 准备程序运行目录

在决定运行docker-compose的任意路径创建好目录,以后我们程序运行相关的文件都保存在该路径下,如/app

mkdir /app
cd /app

在运行目录下创建nginx

2. acme.sh 申请证书并定期更新

2.1 acme.sh 申请证书

在程序运行路径下创建docker-compose.yml,通过docker-compose启动docker镜像的所有参数信息都会写入到该文件中

# /app/docker-compose.yml
version: '3'
services:
    acme.sh:
        image: neilpang/acme.sh
        volumes:
            - "./acme.sh:/acme.sh:z"
        environment:
            - CF_Key="xxx"
            - CF_Email="yyy"
        command: daemon

本文以cloudflare申请证书为例,其他域名服务商可以根据要求将相关的身份信息写入environment环境变量

运行acme生成证书

sudo docker-compose run acme.sh --issue --dns dns_cf -d xxx.example.com

域名和dns服务商参数根据实际情况自行修改。要使用cloudflare创建证书,需要把域名的ns指向到cloudflare并在cloudflare网站上申请管理员的key。运行完成后证书会保存在.acme.sh/xxx.example.com/文件夹下。

2.2 自动更新证书

这样免费申请的证书 只有3个月的有效期,通过command: daemon参数运行后,该进程会定期自动去域名服务商更新证书,再也不用为证书过期苦恼了。

docker-compose up -d

2. Nginx

Nginx提供了官方镜像
直接使用官方镜像就可以启动nginx服务。但要想根据我们的要求运行反向代理,还得编写一个配置文件。

创建/app/nginx/conf文件夹

mkdir -p /app/nginx/conf

在该文件夹下创建proxy.conf文件

# /app/nginx/conf/proxy.conf

server {
    listen 80;
    server_name   xxx.example.com;

    return 301 https://$host;
}

server {
    listen 443 ssl;
    server_name             xxx.example.com;
    ssl_certificate         /etc/nginx/certs/$server_name/fullchain.cer;
    ssl_certificate_key     /etc/nginx/certs/$server_name/$server_name.key;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    location / {
        proxy_pass http://realserver;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_http_version      1.1;
        proxy_cache_bypass      $http_upgrade;

        proxy_set_header Upgrade                $http_upgrade;
        proxy_set_header Connection             "upgrade";
        proxy_set_header Host                   $host;
        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto      $scheme;
        proxy_set_header X-Forwarded-Host       $host;
        proxy_set_header X-Forwarded-Port       $server_port;

    }
}

这个配置启动了HSTS,强制使用https,增强了web服务的安全性。

把xxx.example.com替换成实际的域名。 realserver是反代对应的真实服务器地址。

要禁用直接用ip地址访问可以增加一个server节:

server {
    listen 80 default_server;
    return 444;
}

nginx在docker-compose中配置如下:

nginx:
    image: nginx
    ports:
        - "80:80"
        - "443:443"
    volumes:
        - "./acme.sh:/etc/nginx/certs"
        - "./nginx/conf:/etc/nginx/conf.d:z"
        - "./nginx/log/:/var/log/nginx/:z"
    environment:
        - ENV=production

通过docker-compose启动一组容器

docker-compose up -d

大功告成。

最后完整版的docker-compose文件如下:

# /app/docker-compose.yml
version: '3'
services:
    acme.sh:
        image: neilpang/acme.sh
        volumes:
            - "./acme.sh:/acme.sh:z"
        environment:
            - CF_Key="xxx"
            - CF_Email="yyy"
        command: daemon
    nginx:
        image: nginx
        network_mode: host
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - "./acme.sh:/etc/nginx/certs"
            - "./nginx/conf:/etc/nginx/conf.d:z"
            - "./nginx/log/:/var/log/nginx/:z"
        environment:
            - ENV=production

Hostloc是先进最火热的VPS和站长论坛,大家在上面分享建站经验和建站资源,是非常棒的站长交流平台。

但是很多人发现网站需要邀请码才能注册,那么怎么获得邀请码呢?

可以直接在某宝购买,在某宝上搜索hostloc邀请码,1-2元一枚,在线购买直接发货就可以注册了

hostloc发帖,回帖,访问其他人的主页(仅限PC版面)均可以获得积分,越活跃升级越快。

在开发python包时经常需要用到python setup.py develop将当前路径动态注册到python,
以后每次修改都不需要重新安装就可以在全局引用包。

但是最近发现在运行python setup.py develop时报错,显示

Traceback (most recent call last):
  File "setup.py", line 38, in <module>
    'attrs>=17.4.0'
  File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/site-packages/setuptools/command/develop.py", line 27, in run
    self.install_for_development()
  File "/usr/lib/python2.7/site-packages/setuptools/command/develop.py", line 129, in install_for_development
    self.process_distribution(None, self.dist, not self.no_deps)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 701, in process_distribution
    distreq.project_name, distreq.specs, requirement.extras
TypeError: __init__() takes exactly 2 arguments (4 given)

原因是python默认安装了旧版本的setuptools,pip安装时不会删除默认版本,导致site-packages下面有多个版本

 ll /lib/python2.7/site-packages | grep setuptools
drwxr-xr-x.  6 root root   4096 Jul  8 11:43 setuptools
drwxr-xr-x.  2 root root    179 Jul  8 11:43 setuptools-0.9.8-py2.7.egg-info
drwxr-xr-x.  2 root root    170 Jul  8 11:33 setuptools-41.0.1.dist-info
-rw-r--r--.  1 root root     33 Jul  8 11:33 setuptools.pth

解决方法是多次运行pip uninstall删除所有版本,再重新安装

pip uninstall setuptools
pip uninstall setuptools
pip install setuptools

scrapy项目依赖twisted,但是在centos7上直接用pip install twisted可能会安装失败。本文介绍如何在Centos7上顺利安装Twisted

step1 安装epel-release包

Centos7官方repository中没有包含python相关的包,幸运的是epel-release包含了python2相关的包。

yum install -y epel-release

step2 安装twisted依赖包

Twisted有C扩展,默认需要python-devel和GCC,可以通过yum安装

yum install -y python-pip python-devel gcc

step3 安装twisted

pip install twisted

1.安装ius源

ius是一个专门提供经过验证RPM包的第三方社区,很多官方RPM库没有提供的新功能可以在该网站上找到。

sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm

2.更新YUM

sudo yum update

3.安装python3包

sudo yum install -y python36u python36u-libs python36u-devel python36u-pip

然后就可以尽情使用python3了

python3.6 -V

# Python 3.6.8

pip3 -V

# pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)

添加软链接

ln -s /usr/bin/python3.6 /usr/bin/python3
ln -s /usr/bin/pip3.6 /usr/bin/pip3

现在可以直接使用python3和pip3命令了

Twisted库是比较流行的一个异步操作库,实现了大量的网络协议,scrapy就是基于Twisted实现异步操作和网络通讯。文末有彩蛋。

方法1.源代码安装

Twisted默认源代码安装需要在本地编译c代码,需要安装VC

pip install twisted

方法2.第三方预编译包

https://www.lfd.uci.edu网站上提供了大非官方提供的windows平台二进制python包,很多无法直接安装的库可以在这里找到平台专用包。

打开该页面,Ctrl+F打开搜索框,输入要查找的包名称即可快速找到安装包。

twisted的版本如下。

https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

截至发稿最新版本Twisted19.2.1的安装包连接如下

python27 Win64

pip install https://download.lfd.uci.edu/pythonlibs/t4jqbe6o/Twisted-19.2.1-cp27-cp27m-win_amd64.whl

python27 Win32

pip install https://download.lfd.uci.edu/pythonlibs/t4jqbe6o/Twisted-19.2.1-cp27-cp27m-win32.whl

python37 Win64

pip install https://download.lfd.uci.edu/pythonlibs/t4jqbe6o/Twisted-19.2.1-cp37-cp37m-win_amd64.whl

python37 Win32

pip install https://download.lfd.uci.edu/pythonlibs/t4jqbe6o/Twisted-19.2.1-cp37-cp37m-win32.whl

其他版本可以自行查找下载。

方法3(推荐)

官方最新提供了支持pip安装方式,如下

pip install Twisted[windows_platform]

一键安装,从此无烦恼。

Microsoft Visual C++ 14.0 is required

如果在安装过程中提示错误:

error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/

可以在这里下载Visual C++ 2015(14.0) Build Tools.

see:

小彩蛋

通过twisted可以快速建立各种简易的网络服务器,比如启动一个本地测试用的FTP服务器代码如下:

ftpserver.py:

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

"""
An example FTP server with minimal user authentication.
"""

from twisted.protocols.ftp import FTPFactory, FTPRealm, BaseFTPRealm
from twisted.cred.portal import Portal
from twisted.cred.checkers import AllowAnonymousAccess, FilePasswordDB
from twisted.internet import reactor
from twisted.python import log, failure, filepath

"""
An example FTP server with minimal user authentication.
"""

#
# First, set up a portal (twisted.cred.portal.Portal). This will be used
# to authenticate user logins, including anonymous logins.
#
# Part of this will be to establish the "realm" of the server - the most
# important task in this case is to establish where anonymous users will
# have default access to. In a real world scenario this would typically
# point to something like '/pub' but for this example it is pointed at the
# current working directory.
#
# The other important part of the portal setup is to point it to a list of
# credential checkers. In this case, the first of these is used to grant
# access to anonymous users and is relatively simple; the second is a very
# primitive password checker.  This example uses a plain text password file
# that has one username:password pair per line. This checker *does* provide
# a hashing interface, and one would normally want to use it instead of
# plain text storage for anything remotely resembling a 'live' network. In
# this case, the file "pass.dat" is used, and stored in the same directory
# as the server. BAD.
#
# Create a pass.dat file which looks like this:
#
# =====================
#   jeff:bozo
#   grimmtooth:bozo2
# =====================
class LocalFTPRealm(BaseFTPRealm):
    def __init__(self):
        self.userHome = filepath.FilePath('./HOME')
           
    def getHomeDirectory(self, avatarId):
        return self.userHome
p = Portal(LocalFTPRealm(),
           [AllowAnonymousAccess(), FilePasswordDB('./auth', cache=True)])
#
# Once the portal is set up, start up the FTPFactory and pass the portal to
# it on startup. FTPFactory will start up a twisted.protocols.ftp.FTP()
# handler for each incoming OPEN request. Business as usual in Twisted land.
#
f = FTPFactory(p)

#
# You know this part. Point the reactor to port 21 coupled with the above factory,
# and start the event loop.
#
reactor.listenTCP(21, f)
reactor.run()

ftp同一文件夹下创建一个auth文件,每行一个用户,用户名和密码用冒号:分割。

auth:

abc:123

FTP根目录对应当前文件夹下的HOME文件夹,Enjoy.

出错情况

以前一直用的一个vagrant环境,突然启动出现问题。
vagrant 运行在windows上,vagrant 启动centos7虚拟机,vagrant up时提示错误,无法将host上的文件夹映射到虚拟机里。

错误信息如下:

Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel module.
Please verify that these guest additions are properly installed in the
guest. This is not a bug in Vagrant and is usually caused by a faulty
Vagrant box. For context, the command attempted was:

mount -t vboxsf -o uid=1000,gid=1000 vagrant /vagrant

The error output from the command was:

/sbin/mount.vboxsf: mounting failed with the error: No such device

处理方式

先ssh连进guest

vagrant ssh

然后

cd /opt/VBoxGuestAdditions-*/init  
sudo ./vboxadd setup

提示信息

VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
grep: Unmatched ) or \)
This system is currently not set up to build kernel modules.
Please install the Linux kernel "header" files matching the current kernel
for adding new hardware support to the system.
The distribution packages containing the headers are probably:
    kernel-devel kernel-devel-3.10.0-957.21.3.el7.x86_64
VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
grep: Unmatched ) or \)
This system is currently not set up to build kernel modules.
Please install the Linux kernel "header" files matching the current kernel
for adding new hardware support to the system.
The distribution packages containing the headers are probably:
    kernel-devel kernel-devel-3.10.0-957.21.3.el7.x86_64
modprobe vboxguest failed
The log file /var/log/vboxadd-setup.log may contain further information.

安装kernel

sudo yum install kernel-devel kernel-devel-3.10.0-957.21.3.el7.x86_64

重新运行 ./vboxadd setup 提示错误

VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
grep: Unmatched ) or \)
VirtualBox Guest Additions: Look at /var/log/vboxadd-setup.log to find out what went wrong
VirtualBox Guest Additions: Starting.

/var/log/vboxadd-setup.log:

Building the main Guest Additions module.
Building the shared folder support module.
Building the graphics driver module.
Error building the module:
Could not find the X.Org or XFree86 Window System, skipping.

在虚拟机中安装xorg

yum -y install xorg-x11-drivers xorg-x11-utils

在host上安装vbguest插件

vagrant plugin install vagrant-vbguest

重新启动虚拟机,问题解决

vagrant reload

Redis TTL

Redis 只在key一级上有TTL (Time to live) 机制,https://redis.io/commands/expire,有的时候我们需要把item按某种顺序排序,如排行榜或移动时间窗口,这个时候使用Sorted Set(Zset)就非常有用了。

Redis Zset 按时间过期机制

_1. zadd 添加item时设置score为当前unixtime
__1.1. 如果只以第一次插入时间戳做删除则指定nx=true, https://redis.io/commands/zadd
__1.2. 如果需要更新已经存在item时间戳为当前时间TTL则指定默认选项,score设置为当前unittime
__1.3. 如果每次需要延长固定时间,检查item存在时使用ZINCRBY命令
_2. 定时做清理任务,每次计算删除某个时间点之前的所有item zremrangebyscore

Redis Zset 命令时间复杂度

Command Complexities of Zset and Set

| | Zset | Set |
|———————- |————- |—— |
| Add | O(log(N)) | O(1) |
| Delete (1 items) | O(log(N)) | O(1) |
| Iterate items by key | O(log(N)+M) | O(N) |
| Scan by score | O(log(N)+M) | N/A |

可以看出zset单个元素操作一般为Log(N)其中N为当前集合元素个数。

Redis Zset 底层结构及算法实现

与set无顺序存储不同,Zset按score顺序进行存储,这也是为什么基本操作都是O(log(N))复杂度。

Redis使用两种结构存储zset,在数据个数较少时使用ziplist,数量超出阈值时使用skiplist,阈值通过zset-max-ziplist-entries and zset-max-ziplist-value设置。

ziplist

ziplist使用连续空间存储双向链表,相比基于堆空间指针的链表前后向移动速度更快。

reids list也是使用ziplist存储。

skiplist

skiplist 跳跃列表:

/* ZSETs use a specialized version of Skiplists */
typedef struct zskiplistNode {
    sds ele;
    double score;
    struct zskiplistNode *backward;
    struct zskiplistLevel {
        struct zskiplistNode *forward;
        unsigned long span;
    } level[];
} zskiplistNode;

typedef struct zskiplist {
    struct zskiplistNode *header, *tail;
    unsigned long length;
    int level;
} zskiplist;

typedef struct zset {
    dict *dict;
    zskiplist *zsl;
} zset;

跳跃列表保存一个有序排列的链表,通过采用多层存储且保持每一层链表是其上一层链表的自己,从最稀疏的层开始搜索,从而达到比链表O(N)更优的查找和插入性能O(log(N))。

具体说明参见wiki